Missing robots.txt

What This Means

No robots.txt file was found at https://yourdomain.com/robots.txt. The domain has not published any robot crawl instructions.

Why It Matters for SEO

A missing robots.txt is not a catastrophic error — without one, all pages are implicitly crawlable. But its absence creates two practical problems: there is no mechanism to prevent crawlers from accessing development areas, admin paths, duplicate content URLs, or other pages you do not want indexed; and there is no sitemap directive pointing crawlers to your XML sitemap.

Most well-configured sites have a robots.txt that at minimum declares the sitemap location and excludes admin or system paths.

What the Platform Checks

The platform makes an HTTP request to /robots.txt on the root domain. Fires when the response is a 404, or when the server returns a non-200 status with no valid robots.txt content.

How to Fix It

Create a valid robots.txt at your domain root. At minimum:

User-agent: *
Allow: /
Sitemap: https://yourdomain.com/sitemap.xml

Add specific Disallow rules for paths you want to exclude from crawling (admin panels, login pages, duplicate parameter URLs, internal search results). Place the file at https://yourdomain.com/robots.txt — no subdirectory.