Module 3 · Lesson 3.1 · AI bots and robots.txt · 3 min
ChatGPT and other AI bots in your robots.txt file
Lesson 3.1 of the free AEO and GEO course: what the robots.txt file is and how to read it, the three families of AI bots (search, assistants, training), the names you need to know, and the mistakes that block a site without its owner knowing.
Transcript
Introduction
Lesson 3.1: AI bots and robots.txt.
What robots.txt is
An engine that can't read a page can't cite it. And the first place where you decide who gets to read is a file: robots.txt. It's a text file at the root of your site. It holds rules for each bot, called by name: this one can come in, that one can't. And it says where your sitemap is. It's a standard, RFC 9309, and the bots of the major engines respect it.
Three families of bots
AI bots don't all do the same thing. There are three families. Search bots read your pages so they can cite you in their answers: OAI-SearchBot, PerplexityBot, Claude-SearchBot, and the ones from Google and Bing. Assistant bots open a page because a user asked for it, right at that moment: ChatGPT-User, Perplexity-User, Claude-User. Then there are training bots, which collect text to train the models: GPTBot, ClaudeBot, CCBot. These ones don't cite. Google-Extended is a special case: as we saw, it also affects Gemini's answers.
Reading your robots.txt
Open yours: type your domain, slash, robots.txt, into your browser. First, look for a rule that blocks everyone: user-agent asterisk, and disallow slash. If it's there, no engine can read you. Then look for rules for AI bots, called by name. And check that the sitemap line is there.
The most common mistakes
Many sites are blocked and don't know it. Three mistakes are the most common. The block from the staging site, still there after going live. A plugin, for security or for SEO, that adds rules for AI bots without you knowing. And misspelled names: a rule for a bot that doesn't exist does nothing at all.
Your task for today
Your task for today: open your robots.txt in the browser. For each family of bots, note whether it's allowed, blocked, or has no rules. And mark the mistakes: we'll fix them in the next lesson.
Next lesson
In the next lesson, we write the rules: blocking training, without disappearing from answers. See you there.
Your task
Open your robots.txt in the browser. For each family of bots, note whether it's allowed, blocked, or has no rules. And mark the mistakes: we'll fix them in the next lesson.
CiteProof: AI bots: who may read the site: the AI bots tab, allowed or blocked per crawler, the visit log, and the WordPress plugin writing robots.txt.