robots.txt tester

Why this is worth checking before you launch

Two failures account for almost every "why is my site invisible" question from a pre-launch founder, and both live in this one file:

• A leftover Disallow: / from a staging template. It blocks everything, and it is invisible unless you look.
• A robots.txt that returns a server error. Google treats a 5xx here as "do not crawl this site at all" — not as "no rules". A flaky server on this one path quietly suppresses the whole domain.

A 404 is fine. You only need a robots.txt if you want to restrict something or point at a sitemap.

Search, AI answers, and AI training are three different decisions

Blocking a training crawler does not remove you from AI answers, and blocking an answer crawler does not remove you from search. They are separate tokens with separate consequences, which is why this tool groups them rather than giving you one number.

The crawlers we check

Registry last verified 2026-09-01. Every row links to the operator's own documentation. Download as JSON (CC BY 4.0).

Search index

AI answers

AI training

AI agent

Link preview

SEO tool

Ads

Archive

Other