robots.txt Blocking Crawl

What This Means

The site's robots.txt contains Disallow rules that are blocking crawlers from pages that should be indexable. This may be an over-broad Disallow directive (Disallow: / blocking the entire site), or rules that cover important content directories.

Why It Matters for SEO

A robots.txt Disallow is absolute — a disallowed URL cannot be indexed, regardless of how many links point to it. The page can still appear in search results (as a URL with no description) if external links force indexation, but it cannot be crawled to verify its content. An overly-broad Disallow is one of the most severe technical SEO issues — it can cause an entire site to disappear from search results.

This issue is particularly common after site migrations where a staging robots.txt (which typically blocks everything) is accidentally pushed to the production server.

What the Platform Checks

The platform fetches and parses robots.txt, then checks which of the crawled pages are covered by active Disallow rules for the common search engine user-agents (Googlebot, Bingbot). Fires when key pages or a significant portion of the site is disallowed.

How to Fix It

• If the entire site is blocked (Disallow: /): this is almost always a staging robots.txt deployed to production. Change to Disallow: (empty = allow all) or Allow: /.
• If specific important directories are blocked: Remove or narrow the Disallow rules to exclude only the paths you intend to restrict.
• Test before deploying: Use Google Search Console's robots.txt tester to verify your intended rules before pushing changes to production.
• After fixing: Request a crawl refresh via Search Console and monitor coverage to see blocked pages becoming indexable.