Robots Directives
Robots directives control what search engines are allowed to crawl and index on your site. A missing or misconfigured robots.txt, a noindex on a page that should rank, or a deprecated directive that confuses crawlers can all have significant indexation consequences. These thirteen checks cover every robots directive the platform detects — from site-level robots.txt issues to page-level meta robots tags and individual directive types.
No robots.txt file at the domain root — crawlers have no instructions.
robots.txt is blocking crawlers from key pages or the entire site.
Page has a noindex directive — intentional or accidental exclusion from the index.
A nofollow on the entire page — crawlers will not follow any outlinks from it.
The "none" directive is both noindex and nofollow — pages are excluded from indexing and crawl.
A meta robots tag found outside <head> — likely ignored by search engines.
Page uses noimageindex — images on this page are excluded from Google Image Search.
nosnippet prevents search engines from showing a text or video snippet in results.
notranslate suppresses Google's automatic translation of the page.
unavailable_after sets an expiry date for indexation — the page will be removed from the index.
noodp prevented use of DMOZ descriptions — DMOZ closed in 2017, making this directive obsolete.
noydir prevented use of Yahoo Directory descriptions — the directory closed in 2014.
The XML sitemap lists URLs that have noindex directives — a contradictory signal to search engines.
Audit every robots directiveacross your site
A full audit surfaces every page with robots directives that may be preventing indexation or causing crawl issues.