Robots.txt checker and tester

What robots.txt does

Robots.txt is a plain text file at the root of a site (https://example.com/robots.txt) that tells crawlers which paths they may request. Search engines read it before they crawl, and so do most AI crawlers. Since 2022 it has been an internet standard, RFC 9309, which describes the format that Google, Bing and others follow.

It controls crawling, not indexing. Google says plainly that robots.txt is not a mechanism for keeping a page out of Google: a blocked URL can still appear in results, without a description, if other pages link to it. To keep a page out of search, let Google crawl it and add a noindex robots meta tag, or protect it with a password.

It is also a request, not a lock. Googlebot and other well-behaved crawlers obey it, but nothing forces a crawler to, and the file is public, so never list private folders in it hoping to hide them.

How to read the rules

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

User-agent: GPTBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

• User-agent starts a group and names the crawler it applies to. * means every crawler that has no group of its own. Names are not case sensitive.
• Disallow and Allow are path prefixes, matched from the start of the path. Paths are case sensitive: /Blog/ and /blog/ are different.
• Wildcards: * matches any run of characters and $ marks the end of the URL, so Disallow: /*.pdf$ blocks every PDF.
• Sitemap lines can go anywhere in the file and apply to every crawler.

Which rule wins

Two rules decide every result, and this checker applies both the way Google does. First, a crawler obeys only one group: the one with the most specific user agent that matches it. If there is a User-agent: GPTBot group, GPTBot ignores the * group completely, so rules you meant for everyone have to be repeated there. Second, inside that group the longest matching path wins, and when an Allow and a Disallow match with the same length, Google uses the least restrictive one, the Allow.

In the example above, /wp-admin/admin-ajax.php is allowed for Googlebot, because the Allow rule is longer than /wp-admin/. The same URL is blocked for GPTBot, whose own group disallows everything.

How to fix "Blocked by robots.txt" in Search Console

Search Console reports "Blocked by robots.txt" when Googlebot wanted a URL but a rule stopped it. Paste the URL above with Googlebot selected: the verdict shows the exact rule and its line. If the page should be in Google, remove or narrow that rule, then test the URL again with your edited file before you publish it. If the page should stay out of search, the better tool is noindex, which only works if Google can crawl the page and see it.

"Indexed, though blocked by robots.txt" is the opposite case: Google found links to a URL it may not crawl and indexed it anyway. If you want it gone, allow crawling and add noindex; blocking harder makes it worse, because Google can never see the noindex.

Google caches robots.txt for up to 24 hours, so changes take a while to apply. Google retired the old robots.txt Tester in Search Console in 2023; the robots.txt report in Search Console now shows which files Google fetched and when.

Don't block CSS and JavaScript

Google renders pages much like a browser to understand them. If robots.txt blocks the stylesheets or scripts a page needs, Google sees a broken or empty layout, which can hurt how it judges the page, for example on mobile. Google's advice is to block resources only when the page does not depend on them. Old WordPress setups often block /wp-includes/ or /wp-content/plugins/, which hold exactly those files. This checker reads the page you enter and tests each of its own CSS and JavaScript files against Googlebot's rules.

AI crawlers and robots.txt

Most AI companies now publish the names of their crawlers and say they honor robots.txt. They usually run several, for different jobs, and blocking one does not block the others:

Google-Extended is a token, not a separate crawler: Googlebot does the crawling and the token decides whether that content may be used for Gemini training and grounding. Google states that it does not affect inclusion or ranking in Google Search. Blocking Googlebot itself, on the other hand, removes you from Google Search entirely.

Whether to block AI crawlers is a business decision, not a technical error. Publishers selling their content often block training crawlers; businesses that want to be recommended in ChatGPT, Claude or Perplexity answers usually keep the search crawlers allowed.

What happens when robots.txt is missing or broken

If /robots.txt answers 404 or another 4xx error (except 429), Google treats the whole site as allowed. If it answers a 5xx server error, Google stops crawling the site for the first 12 hours and keeps trying, then uses its last good copy for up to 30 days. A robots.txt that keeps failing can therefore pause crawling of the whole site. Google also reads only the first 500 KiB of the file, and ignores Crawl-delay and Noindex lines (Bing and some other crawlers do read Crawl-delay).

Sources