Crawl errors: how to find them in 20 minutes.

You can find your crawl errors in under 20 minutes using Google Search Console and one free crawler — no agency, no budget, no technical background required.

Crawl errors are the silent killers of search visibility. Google sends a bot to read your site. If that bot hits a dead end — a broken URL, a blocked page, a redirect that goes nowhere — it stops indexing. Your page disappears from search results. Clients looking for exactly what you offer find someone else instead. The fix starts with knowing what is broken.

What a crawl error actually is

A crawl error happens when Google's bot requests a URL on your site and gets a response that prevents it from reading the page. There are several types, and each one tells you something different.

A 404 error means the page does not exist. Someone linked to it, or it used to exist and was deleted without a redirect. Google tried to visit it, got nothing, and marked it as dead.

A redirect error means the page tries to send Google somewhere, but the destination is broken, missing, or loops back on itself. Redirect chains — where page A sends to page B sends to page C — are slower and weaker than a direct path. Redirect loops, where the bot circles back to the start, are a full stop.

A server error (5xx) means your server failed to respond. This is less common for small sites, but it happens after migrations, plugin updates, or hosting outages. If Google hits a 5xx three times in a row, it stops trying.

A blocked resource means your robots.txt file or a noindex tag is telling Google to stay out. Sometimes that is intentional. Often it is not — someone ticked the wrong box in a WordPress setting and the whole site went dark.

Knowing the type tells you where to look and how to fix it.

Step one: open Google Search Console

Google Search Console is free and it is the single most useful technical SEO tool available to small business owners. If you have not verified your site yet, that is the first 10 minutes of your session.

Once you are in, go to Indexing → Pages in the left sidebar. This section used to be called Coverage. It shows every URL Google has attempted to crawl, grouped by status.

You will see four buckets: Error, Valid with warning, Valid, and Excluded. Click Error first.

The errors list will show you the exact URLs that Google could not crawl, the error type, and the last time Google tried to reach each one. Export this list. You want it in a spreadsheet so you can sort by error type and prioritise.

Spend five minutes here. Look at the total number. For a small business site with under 100 pages, anything over 10-15 errors warrants same-week attention. For a larger site, look at the ratio of errors to valid pages — if more than 5% of your indexed URLs have errors, that is a structural problem, not a one-off.

Next, click Excluded. This bucket does not always mean trouble, but it contains a category called "Crawled — currently not indexed." That means Google visited the page, decided it was not worth indexing, and moved on. That is worth a separate audit — those pages may be thin, duplicated, or missing signals — but it is not a crawl error in the strict sense.

Step two: run a free crawler

Google Search Console shows you what Google found. A crawler shows you what is actually on your site — including pages Google has not visited yet.

Screaming Frog SEO Spider has a free version that crawls up to 500 URLs. For most small business sites, that covers everything. Download it, enter your homepage URL, and run the crawl. It takes two to five minutes depending on your site size.

When it finishes, go to the Response Codes tab at the top. Filter by Client Error (4xx) first. You will see every broken internal link — every time one page on your site points to another page that does not exist. These are fixable in under an hour for most sites.

Then filter by Redirection (3xx). Look at the chain column. Any chain longer than one hop is worth simplifying. A URL that redirects twice before reaching its destination loses some of the link value passed through it. More importantly, it slows the bot down.

Finally, go to the Response Codes tab and look for Server Error (5xx). These are usually either hosting issues or plugin conflicts. If you see them, your hosting provider or developer needs to know.

At this point — roughly 15-20 minutes in — you have two lists. One from Search Console showing what Google already found broken. One from Screaming Frog showing what is broken on the site itself. Cross-reference them. Fix what overlaps first.

Step three: check your robots.txt and noindex tags

This step takes two minutes and catches the mistake that causes the most damage: accidentally blocking Google from your own site.

In your browser, go to yourdomain.com/robots.txt. You should see a plain text file. The most dangerous line in that file is:

Disallow: /

If that appears under User-agent: *, Google is blocked from your entire site. This happens more often than it should — usually after a developer sets up a staging environment and copies those settings to the live site without checking.

Next, go to Google Search Console and use the URL Inspection tool on your most important pages — your homepage, your main service page, your contact page. Type each URL into the bar at the top. Look for the line that says "Page is available to Google." If it says "URL is not on Google" and the reason is a noindex tag or a robots.txt block, that is the problem.

For WordPress users: go to Settings → Reading and check whether "Discourage search engines from indexing this site" is ticked. It is a single checkbox. It has taken down entire websites.

What to fix first and in what order

Not every crawl error deserves the same urgency. Here is how to prioritise.

Fix blocked important pages first. If your homepage, service pages, or contact page cannot be crawled, nothing else matters. This is a same-day fix.

Fix 404 errors on linked pages second. If other sites or your own content links to a broken URL, you are losing the ranking value of that link. Add a 301 redirect pointing to the most relevant live page.

Fix redirect chains third. These are lower urgency but worth cleaning up during your next maintenance window. Go direct wherever you can.

Fix soft 404s and thin pages last. These are the "Crawled — currently not indexed" pages. They do not break the crawler, but they dilute your site's overall quality signal. Consolidate, improve, or remove them.

The whole diagnostic process — Search Console, Screaming Frog, robots.txt check — takes 20 minutes. The fixes take longer. But you cannot fix what you cannot see, and 20 minutes gives you the full picture.

What this doesn't fix

Finding crawl errors is the start, not the finish. A clean crawl report means Google can reach your pages. It does not mean Google will rank them.

If your pages load slowly, have weak content, or lack the signals that match what a searcher is looking for, you will still be invisible. Core Web Vitals: the three numbers that decide if Google bothers covers the performance layer that sits on top of a clean crawl. And if you want to understand why page speed is a revenue problem, not just a technical one, why your slow site is a sales problem, not an IT problem connects the dots.

Crawl health is one pillar. It is the floor, not the ceiling.

For a law firm client we worked with — McShanes Solicitors — fixing crawl errors was one of the first actions we took before anything else. The site had 23 broken internal links and three redirect chains that had accumulated over years of content updates. None of it was intentional. All of it was invisible until we looked. Cleaning it up did not produce overnight rankings, but it meant Google could finally read the site properly — which is the prerequisite for everything else.

If you want the full technical picture for your site — not just the crawl layer but the whole foundation — Search Foundations is where we start every engagement. Discover. Audit. Position. That sequence exists because you cannot build on ground you have not checked.

A repeatable 20-minute audit routine

Do this once a month. Set a calendar reminder. It does not require a specialist.

1. Open Google Search Console → Indexing → Pages → Error. Export the list.
2. Run Screaming Frog on your domain. Filter by 4xx, then 3xx.
3. Check yourdomain.com/robots.txt for accidental blocks.
4. Use URL Inspection on your five most important pages.
5. Compare this month's error count to last month's. If it is growing, something changed — a plugin update, a URL restructure, a deleted post — and it needs attention.

That is it. Twenty minutes. A spreadsheet. A clear list of what is broken and in what order to fix it.

Search visibility does not require constant intervention. It requires consistent attention to the things that quietly break — and the discipline to check them before they compound.