Google killed its Search Console robots.txt Tester in 2023. This brings it back with faithful RFC 9309 matching — and adds what the original never had: a one-click verdict for every AI crawler. Check if a URL is allowed or blocked for Googlebot, GPTBot, ClaudeBot or PerplexityBot, and see the exact rule that decided it. Built by the team behind Melbourne SEO Consultant.
# Is this URL crawlable — and by whom? > test https://example.com/reports/q3.pdf --all-bots ✓ fetch https://example.com/robots.txt — HTTP 200, 1.8 KB → match RFC 9309 · most-specific UA group wins · ties → Allow Googlebot ALLOWED group: * Bingbot ALLOWED group: * GPTBot BLOCKED group: GPTBot (Disallow: /) ClaudeBot BLOCKED group: * (Disallow: /*.pdf$) PerplexityBot BLOCKED group: * (Disallow: /*.pdf$) # Same check by hand: $ curl -s localhost:5055/api/test \ -d '{"test_url":"…/q3.pdf","user_agent":"GPTBot","fetch_domain":"example.com"}' { "allowed": false, "ua_group": "gptbot", "matched_rule": { "directive": "Disallow", "pattern": "/" } }
Most online checkers use Python's urllib.robotparser, which silently ignores * and $ wildcards — so they say "allowed" when a real crawler is blocked. This implements Google's actual matching rules instead.
Test against Googlebot, Bingbot, DuckDuckBot and YandexBot — plus the bots that feed AI answers: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bytespider, Applebot-Extended and more.
Run one URL against every well-known crawler at once and get a clean ALLOWED / BLOCKED grid — the fastest way to spot that your PDFs are open to Google but closed to ChatGPT. Google's old tester could never do this.
Not just a verdict — the precise Allow or Disallow line that won, the user-agent group it came from, and every matching rule ranked by specificity. No more guessing why a URL is blocked.
* matches any sequence and a trailing $ anchors the end of the URL — so rules like Disallow: /*?sessionid= are evaluated correctly instead of being treated as no-ops.
A crawler obeys exactly one group — the most-specific matching User-agent — and all others, including *, are ignored for it. A Googlebot group correctly governs Googlebot-Image too.
Type a domain and it pulls the live robots.txt server-side (no CORS headaches), or paste your own draft to test changes before you ship them. Either way you get the same verdict engine.
Two endpoints — /api/test for a single verdict and /api/matrix for the all-bots grid. Wire them into CI, a deploy gate, n8n or your own scripts to catch crawl regressions automatically.
It runs entirely on your machine. No accounts, no API keys, no analytics, nothing sent to a third party. Paste-mode never makes an outbound request at all.
A single self-contained Flask app with just Flask and Requests. Clone it, pip install, run — live on localhost:5055 in under a minute. Set any port with PORT=.
Every matching decision is auditable in plain, commented Python. Fork it, embed it, ship it in your own toolchain — no premium tier, no metering, no lock-in.
No setup beyond two pip packages. No external service ever sees your robots.txt.
Drop in the page you want to check and choose a bot — Googlebot, GPTBot, ClaudeBot, or any custom user-agent token. Then either fetch the domain's live robots.txt or paste your own.
Test this bot gives a single ALLOWED / BLOCKED verdict; Test all bots runs the full search-plus-AI matrix at once so you see the whole picture in one grid.
The result shows the deciding Allow/Disallow line, which user-agent group applied, and every matching rule ranked by specificity — so the "why" is never a mystery.
Since Google retired its tester, the options are a dead tool, online checkers built on a wildcard-blind parser, or reading the spec by hand. Here's how a faithful RFC 9309 implementation compares.
| Capability | robots.txt Tester | Google's old tester | urllib-based checkers | Generic online checkers |
|---|---|---|---|---|
| Still available in 2026 | Yes | Retired 2023 | Yes | Yes |
* and $ wildcards |
Honoured | Yes | Silently ignored | Varies |
| Google's most-specific-group rule | Yes | Yes | Partial | Varies |
| Tests AI crawlers (GPTBot, ClaudeBot…) | Built in | No | Manual | Rarely |
| All-bots matrix in one click | Yes | No | No | No |
| Shows the exact deciding rule | Yes | Partial | No | Rarely |
| JSON API for automation / CI | Yes | No | DIY | Rarely |
| Self-hosted, nothing sent out | Yes | Google-hosted | Yes | External service |
| Price | Free, MIT | Was free | Free | Free / ad-supported |
Comparison is descriptive only. Capabilities of third-party checkers vary by vendor and version.
Paste the proposed file, confirm the key URLs are crawlable, then re-test the live domain after it ships. Pairs with our SEO audit service, where blocked-but-important URLs are a common finding.
As answers move to AI engines, a stray Disallow on GPTBot or ClaudeBot quietly removes you from the conversation. The all-bots matrix shows AI access at a glance — the work behind our AI SEO services.
Indexing fell off a cliff after a release? Test the affected URL and the tester names the exact Disallow line and group responsible — no more bisecting a 300-line robots.txt by eye.
Call /api/test in your pipeline to assert that critical URLs stay ALLOWED for Googlebot. A bad robots.txt edit fails the build instead of silently tanking traffic for a week.
Confirm that Disallow: /*?filter= blocks the noise while leaving clean product URLs open. Real wildcard evaluation means you see what Google actually does, not what a naive parser pretends.
A clean JSON verdict with the deciding rule attached makes robots.txt testable like any other config — in unit tests, staging checks, or a headless build. Feeds neatly into technical SEO workflows.
A single Flask app with two dependencies. No build step, no accounts, no API keys — live on localhost:5055 in under a minute.
git clone https://github.com/\ puneetindersingh/robots-txt-tester cd robots-txt-tester pip install -r requirements.txt python app.py # → localhost:5055
curl -s localhost:5055/api/test \
-H 'Content-Type: application/json' \
-d '{"test_url":"https://example.com/x",
"user_agent":"GPTBot",
"fetch_domain":"https://example.com"}'
# Run on any port you like
PORT=8080 python app.py
# All-bots matrix endpoint:
# POST /api/matrix { test_url, robots_text }
Needs Python 3.8+ with Flask and Requests. Full endpoint docs are in the README.
Google sunset the Search Console tester in 2023 because, by its own admission, it no longer matched live Googlebot behaviour. This tool replaces it using Google's actual matching rules (RFC 9309) — the same logic the open-source parser their crawlers use is built on — so the verdict reflects what a real crawler does.
* and $ wildcards?Yes — and that's the whole point. Most "robots.txt checkers" rely on Python's urllib.robotparser, which does not support wildcards, so a rule like Disallow: /*?sessionid= silently becomes a no-op and the checker reports "allowed" when a real crawler would be blocked. This implementation honours * (any sequence) and a trailing $ (end-of-URL anchor) exactly as Google does.
GPTBot, OAI-SearchBot and ChatGPT-User (OpenAI); ClaudeBot, Claude-User and Claude-SearchBot (Anthropic); PerplexityBot and Perplexity-User; Google-Extended (Gemini); CCBot (Common Crawl); Bytespider; Applebot-Extended; Amazonbot; Meta-ExternalAgent and cohere-ai — alongside the classic search bots. You can also enter any custom user-agent token.
A crawler obeys exactly one group: the one whose User-agent token is the most specific (longest) match for its name. Every other group — including * — is ignored for that crawler. So if you have both a * group and a Googlebot group, Googlebot follows only the Googlebot group. Within it, the longest matching path pattern wins, and on a tie Allow beats Disallow.
No. The tool runs entirely on your own machine. In paste mode it makes no outbound request at all. In fetch mode it requests only the target site's /robots.txt — server-side, so there's no CORS issue — and nothing is logged or sent to any third party.
Yes — MIT licensed, free forever, full source on GitHub. No accounts, no metering, no premium tier. Fork it, embed it, or ship it inside your own toolchain.
Knowing a URL is blocked is step one. Knowing what should be crawlable, indexable and cited is the work.
MIT licensed, built in the open. Search bots and AI crawlers, single verdicts and full matrices, live fetch and paste — with the exact rule that decided each one.
No signup. No credit card. No premium tier.
This is just a testing site. Sorry, I am not taking any clients.
Want some SEO advice? Connect with me on LinkedIn.
