Loading...
Loading...
Free tool
Paste a robots.txt file, enter a URL and choose a crawler. You see whether the crawler may fetch the URL, which group and rule decided it, and any syntax problems in the file. It runs in your browser and needs no account.
BlockedGooglebot may not fetch this URL.
Pass No syntax problems found.
A crawler first picks one group: the User-agent line that names it most specifically, or the * group if none does. Several groups for the same crawler are treated as one. Crawler names are not case sensitive.
Inside that group, each Allow and Disallow path is compared with the start of the URL path. Paths are case sensitive, so /Admin and /admin are different. The longest matching path wins, and Allow wins a tie. An empty Disallow blocks nothing.
The syntax check flags directives it does not recognize, rules placed before any User-agent line, paths that do not start with / or *, and lines with no colon. To write a new file, use the robots.txt generator, which has presets for AI crawlers. Google's own robots.txt documentation is the reference for how Googlebot reads the file. See the other free SEO tools too.
The rule with the longest path wins, because it is the most specific. If an Allow and a Disallow match with paths of the same length, Google uses the Allow. If no rule matches, the URL is allowed.
One group only: the one that names it most specifically. If no group names the crawler, it follows the * group. Rules in the * group are not added to a named group, so a crawler with its own group ignores the * rules.
An asterisk matches any run of characters, and a dollar sign at the end of a path means the URL must end there. For example, Disallow: /*.pdf$ blocks URLs that end in .pdf but not /file.pdf?download=1.
No. robots.txt controls crawling, not indexing. A blocked URL can still be indexed without its content if other pages link to it. To keep a page out of search, allow crawling and use a noindex tag, or put the page behind a login.
This tool applies the matching Google documents, which follows the RFC 9309 standard. Other crawlers can differ in small ways. For example, some honor Crawl-delay and Google ignores it. robots.txt is a request, not access control.
Neither. It only reads the text you paste, in your browser, and sends nothing to Seology. To see the file Google last fetched from your site, open the robots.txt report in Search Console.
The Seology AI SEO agent audits any public URL, including crawler access, and delivers fixes on Shopify stores with a 90-day rollback or as GitHub pull requests.
Start Free