Managing AI Crawlers: A 2026 Robots.txt Guide for GPTBot, ClaudeBot & PerplexityBot

Table of Contents

1. Introduction
2. What Are AI Crawlers and Why Do They Matter in 2026?
3. How Are AI Bots Different from Traditional Search Engine Crawlers?
4. What Is Robots.txt, and Why Does It Matter for AI Search?
5. Which AI Bots Should You Know About?
6. How Does Robots.txt Work with AI Bots?
7. Should You Block or Allow AI Bots?
8. What Does a Good Robots.txt File Look Like? (Ready-to-Use Templates)
9. How Can You Check That Bots Are Real and Following Your Rules?
10. What Are the Most Common Robots.txt Mistakes?
11. Does llms.txt Replace Robots.txt?
12. What Should Website Owners Keep in Mind?
13. What Is a 30-Day Action Plan?
14. Final Thoughts
15. FAQs

 

Key Takeaways

 

Introduction

To manage GPTBot, ClaudeBot, and PerplexityBot, add a separate User-agent group for each bot in your robots.txt file and set Allow or Disallow rules for the folders you want to open or close. Block the training bots if you want to protect your content from model training, and allow the search bots if you want your pages cited in AI answers.

That is the short version. The longer version matters, because a single wrong line can make your site invisible to ChatGPT, Claude, or Perplexity overnight. This guide explains how AI crawlers work, which bots deserve your attention, and how to write robots.txt for AI bots without hurting your search visibility.

 

What Are AI Crawlers and Why Do They Matter in 2026?

An AI crawler is an automated program that fetches web pages for an artificial intelligence company. AI crawlers do this for three different reasons, and the reason decides how you should treat each one.

For most of web history, the deal was simple. Search engines crawled your site and sent you visitors. Today, that deal has split into several different arrangements, and not all of them send traffic back.

The three types of AI bots

The most prominent AI bots include:

• Training bots collect content to train or improve large language models. Examples: GPTBot, ClaudeBot, Google-Extended (a control token), Applebot-Extended (also a control token), CCBot.

• Search and indexing bots build the index behind an AI search product. They are the ones that can lead to a citation or a link. Examples: OAI-SearchBot, Claude-SearchBot, PerplexityBot.

• User-triggered fetchers visit a page because a person asked a question or pasted a link. Examples: ChatGPT-User, Claude-User, Perplexity-User.

This three-way split is the most useful mental model for any bot policy. When a client says “let’s block AI,” the first question to ask is: “Which kind?” Blocking all three groups at once is rarely what the business actually wants.

 

How Are AI Bots Different from Traditional Search Engine Crawlers?

Traditional search crawlers collect pages to build an index and send visitors to your site. AI crawlers, on the other hand, collect pages to train models or to pull facts into a written answer, and that answer often keeps the reader on the AI platform instead of sending them to you.

Both types request a URL, read the page, and follow robots.txt. The differences show up in purpose, behavior, and what you get back.

Here is what these differences mean in practice:

• Content must be readable in raw HTML. If key text loads only after JavaScript runs, some AI bots may never see it. Learn how pages get rendered by search engines to see why.
• Clicks are not guaranteed. Being cited is a visibility win, but it does not always produce a visit.
• One policy will not fit all. A rule that suits Googlebot may be wrong for a training bot.
• Identification matters more. Because anyone can fake a user-agent, verification is a bigger part of the job.

 

What Is Robots.txt, and Why Does It Matter for AI Search?

Robots.txt is a plain text file at the root of your domain that tells automated bots which URLs they may fetch and which they should skip. Robots.txt for AI bots matters because it is the first place an AI company checks before it reads your pages, so it decides whether your content can be trained on, indexed, or cited.

Here is why it carries more weight now:

1. It controls entry to AI answers. If a search bot cannot fetch your page, it cannot quote it.
2. It separates training from search. One file can allow citation while refusing model training.
3. It is cheap and fast. One text file, no developer sprint, and changes apply within hours or days.
4. It creates a record. A dated, written policy helps if licensing or legal questions come up.
5. It protects server resources. Rules and Crawl-delay can ease pressure from heavy bot traffic.

If you are new to the file itself, this tutorial on how to create a custom robots.txt file walks through the basics. Keep its limits in mind. The file is a request, not a lock. It does not delete content a bot already collected, and user-triggered fetchers may be treated differently. The sections below show how to write it, test it, and back it up with log checks.

 

Which AI Bots Should You Know About?

You do not need to memorize fifty user-agent strings. Focus on the ones below, which cover the three companies in this guide plus a few others that show up often in server logs.

Always confirm names on each company’s official documentation page. These strings do change.

OpenAI bots

OpenAI documents three bots in its crawler overview. GPTBot is for training, while OAI-SearchBot powers search features in ChatGPT. On the other hand, ChatGPT-User fetches pages when someone asks ChatGPT to look at a link.

OpenAI says each setting is independent. You can block GPTBot and still allow OAI-SearchBot. It also notes that robots.txt updates can take around 24 hours to take effect for its search system.

Anthropic bots

Anthropic’s help center lists ClaudeBot for training, Claude-User for user requests, and Claude-SearchBot for improving search results. The company states that its bots follow robots.txt and that it supports the Crawl-delay directive, which is helpful if a bot is putting load on a small server.

A common mistake is blocking only ClaudeBot and forgetting the other two. That is fine if blocking training is your only goal. But if you want to block everything, you need all three lines.

Perplexity bots

Perplexity’s crawler documentation describes PerplexityBot as the crawler that surfaces and links websites in its search results. The company says it is not used to gather data for foundation model training. Perplexity-User is the fetcher that runs when a person asks a question, and Perplexity states that robots.txt generally does not apply to it because a human triggered the visit.

Other bots worth knowing

Besides these three, other common AI bots are:

• Google-Extended is not a crawler. It is a control token. Google says it does not affect inclusion or ranking in Google Search, so blocking it will not remove you from search results. Googlebot itself still handles AI Overviews eligibility.
• Applebot-Extended works the same way for Apple’s AI features.
• CCBot feeds Common Crawl, an open dataset used by many model builders.
• Bytespider (ByteDance) and Meta-ExternalAgent are also common in logs.

 

How Does Robots.txt Work with AI Bots?

Robots.txt is a plain text file at the root of your domain, for example, https://example.com/robots.txt. Bots read it before they crawl. Using robots.txt for AI bots follows the same rules as for Googlebot, which makes it easier than most people expect.

The basic syntax

There are only four directives you need.

• User-agent: names the bot the rule applies to.
• Disallow: blocks a path.
• Allow: permits a path, usually inside a blocked folder.
• Sitemap: points to your XML sitemap (see XML sitemap best practices).

A minimal example that blocks one bot from the whole site:

And one that blocks a single folder:

Rule precedence

This is where most mistakes happen, so read this part twice.

1. A bot follows the group that best matches its own name. If a GPTBot group exists, GPTBot ignores the * group.
2. Inside a group, the most specific path wins. A longer matching rule beats a shorter one.
3. If an Allow and a Disallow tie on length, Allow wins under RFC 9309. Google applies the least restrictive rule when rules conflict.
4. Paths are case-sensitive. /Blog/ is not /blog/.
5. The file should stay under 500 KiB. Google ignores content beyond that limit, as its robots.txt documentation explains.

What robots.txt cannot do

Be honest with yourself about the limits.

• It does not remove content already collected. Blocking today does not delete what a bot saw last year.
• It does not enforce anything. Compliance is voluntary.
• It does not hide pages. A disallowed URL can still be known if other sites link to it.
• It does not control user-triggered fetchers equally. Companies treat these differently.

If content is private, put it behind a login. Robots.txt is a polite sign on the door, not a lock.

 

Should You Block or Allow AI Bots?

Should you block AI crawlers or allow them to access your website? There is no universal answer. The right choice depends on what your content is worth to you and what you gain from visibility.

Reasons to allow

Common situations to allow AI crawlers include:

• You want citations and brand mentions in ChatGPT, Claude, and Perplexity answers.
• Your business earns from leads or sales, not from ad impressions on the content itself.
• Your content is already public marketing material, such as service pages and guides.
• You are building long-term brand recall. People now ask AI tools for recommendations before they search.

This is the core of AI SEO: being findable, readable, and quotable by the systems people now use to get answers.

Reasons to block

Here’s when to block the AI bots:

• Your content is the product. Paid research, premium journalism, and licensed data fall here.
• You have legal or licensing limits on reuse.
• Bot traffic is stressing your server.
• You want to negotiate a licensing deal before access.

A simple decision table

Ecommerce brands should also review product feeds and crawl paths with their ecommerce SEO team. The default advice for most commercial sites: allow search bots, decide on training bots as a business policy, and review it twice a year. Blocking everything can leave competitors as the only sources an AI tool is able to cite.

Treat AI SEO as part of your wider search plan, and read what GEO is and how it differs from SEO if the terms are unfamiliar. It sits beside traditional SEO, not in place of it. Google Search remains a core channel for most commercial sites, so never let an AI-bot rule touch Googlebot.

 

What Does a Good Robots.txt File Look Like? (Ready-to-Use Templates)

What is the best robots.txt configuration for AI search in 2026? It depends on your goals. Here are four templates to use as starting points. Copy one, change the paths, and test it before you publish. A good robots.txt for AI bots is short, readable, and commented.

Template 1: Allow everything

Use this if you want maximum AI visibility.

Template 2: Block training, allow search

This is a frequently used setup for businesses that want citations but do not want to feed model training.

Template 3: Block all AI bots

Use this only if your content is the product.

Stacking several User-agent lines above one rule set is valid. It keeps the file short.

Template 4: Open only selected folders

Good for sites that want AI bots to see public guides but not account pages or pricing tests.

Here, the longer Allow paths beat the short Disallow: /, so only those folders stay open.

 

How Can You Check That Bots Are Real and Following Your Rules?

Writing the file is only half the job. The other half is checking what actually happens. Many AI crawlers are honest about who they are, but anyone can fake a user-agent string. A bad scraper can call itself “GPTBot” in five seconds. Here’s how to check the genuineness of the AI bots:

Read your server logs

Your access logs show every request, the user-agent, and the IP address. Look for these patterns.

• Request volume per bot. A sudden spike from one user-agent deserves a look.
• Hits on disallowed paths. If a bot you blocked keeps fetching those URLs, it is ignoring your rules.
• Requests for robots.txt. Compliant bots fetch this file first. A bot that never asks for it is a warning sign.
• Status codes. Many 429 or 5xx responses tell you the bot is straining your server.

If you use a log tool such as GoAccess, Screaming Frog Log File Analyser, or your CDN dashboard, filter by user-agent and export a 30-day view.

Verify IP ranges

OpenAI publishes IP ranges for its bots in JSON files linked from its documentation, and Perplexity does the same for PerplexityBot and Perplexity-User. Anthropic’s help center also links to a list of source IP addresses for its crawlers. Anthropic cautions that blocking those addresses can stop it from reading your robots.txt file, so an IP block may not work as an opt-out. Use the published lists to confirm that a request is genuine, and use robots.txt to set your policy.

The check is simple: if a request says “GPTBot” but the IP is not in OpenAI’s published list, treat it as fake.

Use a firewall or CDN

If a bot ignores your file, move from asking to enforcing. Options include:

1. A CDN or WAF rule that blocks by user-agent and verified IP.
2. Rate limiting for any bot that crosses a request threshold.
3. Managed bot controls from providers such as Cloudflare, which let you allow or block AI bot categories with a toggle.
4. A Crawl-delay line for bots that support it, as a gentle first step.

 

What Are the Most Common Robots.txt Mistakes?

The same errors show up again and again in robots.txt files. Here are the ones that can cost a site the most, and they apply to any robots.txt for AI bots setup.

1. Blocking everything with one line. User-agent: * followed by Disallow: / blocks Googlebot too. Traffic from Google can vanish within days.
2. Mixing up training and search bots. Blocking OAI-SearchBot when you only meant to block GPTBot removes you from ChatGPT search results.
3. Forgetting the * fallback. If a bot has its own group, it ignores the * group completely. Repeat any shared rules inside each group.
4. Typos in user-agent names. GPT-Bot is not GPTBot. A misspelled name does nothing, silently.
5. Wrong file location. The file must sit at the root: example.com/robots.txt. A copy in a subfolder is ignored.
6. Blocking CSS and JavaScript folders. This can stop Googlebot from rendering your pages properly.
7. Using robots.txt to hide sensitive pages. The file is public. Anyone can read it and see exactly which paths you want hidden.
8. Never testing. One missing slash can change the meaning of a rule. Test before every release.
9. Setting and forgetting. New bots launch regularly. A file written in 2024 may not mention half of today’s bots.
10. Expecting instant results. Companies cache the file. OpenAI, for example, says changes can take about a day to apply.

How to test your file: Open yoursite.com/robots.txt in a browser, then use a robots.txt tester such as the one in Google Search Console or an open-source parser. Check one URL for each bot name.

 

Does llms.txt Replace Robots.txt?

No. They do different jobs.

• Robots.txt tells bots where they may or may not go. It is an access rule.
• llms.txt is a proposed Markdown file that gives AI tools a short, curated map of your best content. It is a guide, not a permission system.

The llms.txt idea was proposed in 2024. Google says you do not need AI text files to appear in its AI features, and other major AI companies have not published commitments to read the file as a ranking or access signal. Check each company’s current documentation before relying on it. Treat it as a low-cost experiment, not a requirement. If you add one, keep it short and accurate, and never rely on it to block anything.

For real impact, put your effort into the basics that AI search systems clearly use. The llms.txt file guide covers the proposal in more detail, and the technical SEO guide for 2025 covers the foundations:

• Fast, crawlable pages that render without heavy JavaScript (server speed matters here, as this guide on how server speed helps SEO explains).
• Clear headings, short paragraphs, and direct answers near the top, as described in this guide to optimizing website structure for AI search.
• Structured data such as Article, FAQPage, Organization, and Person schema.
• Named authors with real credentials and a visible “last updated” date.
• Original data, quotes, and first-hand experience that other pages do not have.
• Consistent brand mentions across trusted sites.

 

What Should Website Owners Keep in Mind?

Here is what site owners should do.

• Check hosting limits first. Shared hosting plans often cap requests. Heavy bot traffic can slow the site for real visitors.
• Decide your policy by content type. Public service pages and blogs often benefit from visibility. Paid reports and member content may not.
• Think about language. If you publish in Hindi, Odia, Tamil, or other languages, AI search coverage is still developing. Being accessible to search bots may help, though results are not guaranteed.
• Keep records. Note the date and reason for each robots.txt change. It helps if a legal or licensing question comes up later.

If you are comparing SEO services in India, ask how a provider handles bot access, not just keywords and links. A capable team should be able to read server logs, explain the difference between training and search bots, and show you a before-and-after view of AI mentions, which AI visibility tools can help track.

Here are five questions worth asking any agency that sells SEO services in India:

1. Can you show me how you verify GPTBot or ClaudeBot traffic in logs?
2. How do you separate AI search visibility from classic Google rankings in reporting?
3. What is your process if a bot ignores robots.txt?
4. Do you test changes on a staging site first?
5. Can you share a past example with real numbers, including cases where results were mixed?

Buyers of SEO services should also be wary of promises like “guaranteed ChatGPT rankings.” No one controls these systems. Honest providers talk about earning citations, not buying them. This is where AI SEO work needs the same patience and evidence as any other channel.

Whether you hire a team or handle it in-house, take one simple step first: open your robots.txt today and read it line by line.

 

What Is a 30-Day Action Plan?

You do not need a big project. This plan fits into four short sprints.

Then repeat the check every quarter. Keep a change log with the date, the edit, and the reason.

Quick checklist before you publish:

• Googlebot and Bingbot are not blocked by accident
• Every bot name is spelled exactly as documented
• Each bot group repeats any rules it needs
• The Sitemap line points to a live URL
• The file loads at the domain root and returns a 200 status
• You have tested at least one URL per bot
• The change is logged with a date

Once the file is live, keep watching the AI crawlers that visit you and compare their activity with your referral traffic, and use AI visibility tools to see whether those visits turn into citations. If a bot takes thousands of pages and sends nobody back, you now have the data to decide what to do next.

 

Final Thoughts

AI crawler management is no longer a side task for the development team. Instead, it is a content and brand decision. The sites that do well in AI answers are not the ones that panic and block everything, or the ones that ignore the issue. They are the ones that make a clear choice, write it down, and check that it works.

Be sure to learn the three bot types, write a separate group for each bot, test before you publish, and verify behavior in your logs. Good robots.txt for AI bots takes an hour to write and a few minutes each quarter to maintain. And thoughtful AI SEO starts with letting the right systems see your best pages, while you keep control of the rest. Keep an eye on how AI crawlers evolve, because the rules, the bots, and the business models are still changing.

Ready to check how AI bots see your website? If you would like a second pair of eyes on your robots.txt, server logs, and AI visibility, the team at SEOTonic can help with a technical SEO audit and, through its AI SEO services, an AI search review. Request a free SEO audit and ask for a bot-access review, with a clear, plain-language action list for your site.

 

FAQs

Q. Does blocking GPTBot remove my site from ChatGPT?

A. Not necessarily. GPTBot is OpenAI’s training bot. ChatGPT search uses OAI-SearchBot, and user-requested page visits use ChatGPT-User. You can block GPTBot and still allow the other two, as long as you write separate groups for them.

Q. Will blocking Google-Extended hurt my Google rankings?

A. No. Google states that Google-Extended does not affect inclusion or ranking in Google Search. It only controls whether your content is used for certain Gemini AI purposes. Googlebot is a separate system, so do not block it unless you want to leave Google Search.

Q. Do all AI bots obey robots.txt?

A. No. Major companies say their main bots follow it, but compliance is voluntary. User-triggered fetchers may be treated differently, and unnamed scrapers may ignore it. Check your logs and use firewall rules if you need real enforcement.

Q. How long does it take for robots.txt changes to work?

A. It depends on the bot. OpenAI says its search system may take about 24 hours to adjust, and Perplexity says changes can take up to 24 hours. Other bots cache the file for different lengths of time. Plan for a day or two, then confirm in your logs.

Q. Should small businesses block AI bots?

A. Usually not completely. Local service businesses often gain from being cited in AI answers, and their local SEO work already depends on being easy to find. A common middle path is to block training bots and allow search bots. If server load is a problem, use rate limits or Crawl-delay instead of a full block.

Q. Can I block AI bots from only part of my website?

A. Yes. Use Disallow for specific folders, such as /members/ or /pricing-tests/, and Allow for public folders. The more specific path wins when two rules conflict. Test the file before you deploy it.