Free tool

Robots.txt Generator

Choose how AI crawlers should treat your site, list the paths no crawler should fetch, add your sitemap, and download the file. It runs in your browser and needs no account.

AI crawlers

AI search and AI training crawlers may read the site.

Paths to keep all crawlers out of, one per lineSitemap URL

Your robots.txt

Download robots.txt

User-agent: *
Allow: /

How to create a robots.txt file

Pick an AI crawler policy. If you want to appear in ChatGPT, Claude and Perplexity answers, keep the search crawlers allowed.
List paths that waste crawl time or should stay out of the way, such as cart, checkout, admin and internal search pages.
Add the full URL of your XML sitemap.
Upload the file to the root of your domain, then check it in Search Console's robots.txt report.

The AI crawlers this tool knows

AI search: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User. These fetch pages to answer a question and can cite you.

AI training: GPTBot, ClaudeBot, CCBot, Bytespider, Applebot-Extended, Meta-ExternalAgent, Google-Extended. These collect content for model training.

Vendors add and rename crawlers, so confirm the current names in the OpenAI, Anthropic and Perplexity documentation. For the wider picture, read the AI SEO and GEO guide or create an llms.txt file.

Frequently asked questions

Where does robots.txt go?

At the root of the host it applies to, so it loads at yoursite.com/robots.txt. Each subdomain needs its own file. Shopify generates one for you and lets you customize it through the robots.txt.liquid theme template.

Does blocking a page in robots.txt remove it from Google?

No. robots.txt stops crawling, not indexing. A blocked URL can still appear in results if other pages link to it. To keep a page out of search, let it be crawled and add a noindex tag, or put it behind a login.

What is the difference between AI search crawlers and AI training crawlers?

Search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot fetch pages so an assistant can cite them in an answer. Training crawlers such as GPTBot, ClaudeBot and CCBot collect text for model training. Blocking the first group takes you out of AI answers; blocking the second does not.

Does blocking Google-Extended affect Google Search?

No. Google-Extended is a control token for Gemini training and grounding. It does not change how Googlebot crawls or ranks your site in Search.

Do all crawlers obey robots.txt?

The major search engines and the AI companies named here say their crawlers follow it. It is a request, not access control, so never rely on it to protect private content.

Does this tool store what I type?

No. The file is built in your browser. Nothing you enter is sent to Seology.

Not sure what your crawlers can reach today?

The Seology AI SEO agent audits your pages for Google and AI search, including crawler access, and delivers the fixes to your Shopify store or as GitHub pull requests.

Start Free