robots.txt tester
One file decides
what gets read.
Paste any address on your site and see whether your robots.txt lets a crawler fetch it — which group applies, which rule decides, and whether the file has lines in it that no crawler acts on. If there is no robots.txt at all, that is an answer too, and this says so rather than shrugging.
This tool works differently from the generator. The generator never uploads your image — it runs in your browser. The checker cannot: it fetches the public address you type, from our server, and reads the robots.txt file at that address’s origin, not the page itself. Only the address you enter is sent. The addresses you check are then kept in this browser, so it can tell you what changed since last time; they go nowhere, and you can clear them.
Questions
I gave it an address with a path. Why does it say it read a different file?
Because robots.txt lives at the origin and nowhere else. Whatever address you type, the file fetched is /robots.txt at that host — a site checked as example.com/blog/ is told which file was read, and it is not one underneath /blog/.
Does it fetch my pages as well?
No page is fetched anywhere. One file is read, and that file is robots.txt.