Google published the second episode of a new video series on October 21, 2026, and spent most of it correcting common misconceptions about how robots.txt works. The hosts, Gary Illyes and Cherry Prommawin, reported by Search Engine Roundtable, walked through the exact rules Google’s parser follows. They covered how it treats AI crawlers, emojis, and non-English URLs. A lot of the robots.txt rules for AI crawlers that site owners assume are true turned out to be wrong, and the video lays out why in plain terms.
If you have ever written a robots.txt rule and hoped it would do more than it actually does, this is worth reading before you touch the file again. Robots.txt rules for AI crawlers have become a recurring topic in SEO forums this year. More site owners are trying to control GPTBot, ClaudeBot, and similar agents without fully understanding how the file gets parsed.
Google’s New Video Series Breaks Down robots.txt Rules for AI Crawlers
The series is called Non-Canonical Cabinet, and this was its second installment. Illyes and Prommawin used most of the runtime to explain, line by line, how Google’s robots.txt parser actually reads a file sitting at the root of a website.
robots.txt is a plain text file. It tells automated bots, including Googlebot and other crawlers that choose to comply, where they may and may not go on a site. Its behavior is defined by RFC 9309, the internet standard that formalized how the file should be written and interpreted. Google helped push that standard through in 2022, and its own parser still follows it today.
What the hosts spent most of the episode doing was correcting assumptions. Here are seven of them, and what the video says actually happens instead.
Myth 1: Plain English Instructions Will Work If They Are Clear Enough
Some site owners write rules like “don’t crawl secret page” and assume a smart enough crawler will understand the intent. Illyes used that exact phrase as an example of what does not work.
The parser reads one line at a time and only recognizes four directives: user-agent, allow, disallow, and sitemap.
Anything else gets silently discarded, and the crawler moves on as if the line never existed. Illyes described it directly: the crawler looks at “don’t crawl,” decides it is not a valid rule, ignores it, and keeps going.
Use the exact syntax. Disallow: /secret-page/ works. A sentence does not.
Myth 2: Blocking Crawling Also Controls What AI Does With the Content
This is the mix-up that causes the most confusion, and it is the one Illyes was clearest about. He called robots.txt an access preference protocol, not a data use agreement. In his words, “it doesn’t dictate what a system does with the data after it has already crawled it.”
Disallowing a crawler stops it from fetching pages in the first place, assuming the crawler respects the file at all. It says nothing about what happens to content a crawler already has, and nothing about training, summarizing, or reusing it elsewhere. DevByteDaily covered this exact gap earlier this month in our look at the AI crawler blocking myth. Google confirmed there that tools like Cloudflare’s Content Signals and llms.txt files do not change what AI companies do with content either.
One nuance the video skipped: Google’s own Google-Extended token is treated differently internally. Blocking it does tell Google not to use that content for training certain AI features, which is a data-use signal layered onto an access-control file. That exception exists for Google’s own crawler. It does not extend to how other companies treat their tokens.
My take, after watching how this confusion keeps resurfacing, is that the Google-Extended exception is part of why the myth persists. Site owners see one token that genuinely changes data use and assume every other AI crawler token works the same way. In practice, GPTBot, ClaudeBot, and most other named agents only control crawling access, not what happens to the content afterward.
Myth 3: Emoji and ASCII Art in Comments Are Harmless
Lines that start with a hash symbol are comments, and the parser ignores them. Some site owners use that space for decorative ASCII art or emoji. Illyes called this a questionable choice, not because it breaks anything directly, but because of where it can lead.
Google enforces roughly a 500 KiB size limit on robots.txt files and ignores anything past that point. A file padded with long comments or large blocks of ASCII art can push real, functioning rules past that cutoff without any warning. The rule you need most could be the one that gets cut off.
Keep the file short. Save the creativity for somewhere a parser will never read it.
Myth 4: A Misspelled Rule Still Blocks Something
This follows the same logic as myth one. If you type “Disalow” instead of “Disallow,” the parser does not guess what you meant. It checks the line against its list of four recognized directives, finds no match, and drops it.
The practical risk is that your site keeps running as if the rule was never written. A page you believe is blocked from crawling may not be blocked at all, and you will not get an error telling you so.
Open Search Console’s URL inspection tool periodically and check how Google actually sees your robots.txt file, rather than assuming your last edit landed the way you intended.
Myth 5: One Line Can Block “All AI Crawlers” at Once
There is no catch-all setting for AI crawlers as a category. To block a specific bot, you have to name its exact user-agent token: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and so on. A line that tries to describe AI crawlers generally will not be understood, for the same reason a plain-English sentence is not understood.
There is a second layer worth knowing here. Robots.txt compliance is voluntary. The file only works on crawlers that choose to check it, which is why Illyes and Prommawin kept calling them “well-behaved” crawlers rather than assuming universal compliance.
That gap shows up in real measurements. A dataset from TollBit, cited in reporting on this video by PPC Land, found that roughly 15% of identified AI page-fetching agents in Europe still reached URLs marked disallowed. Separate research from HasData, cited in the same coverage, found 234 of 592 sites that disallowed GPTBot were still served pages by it. Naming the right token correctly is necessary, but it is not a guarantee.
What this means for you: writing correct robots.txt rules for AI crawlers reduces unwanted crawling from agents that choose to comply. It does not stop every agent, and it does nothing against a crawler built to ignore the file outright. Treat it as a filter you control, not a lock you can rely on.
Myth 6: International Characters Work the Same Way on Every Crawler
User-agent tokens are restricted to letters and dashes. Emojis and non-English characters will never work there, no matter which bot you are naming.
Paths are different. Google’s parser can read raw UTF-8 characters in a path, including German umlauts, Chinese characters, or Arabic script. Older or third-party crawlers may not handle that as gracefully. The video’s recommendation is to percent-encode non-English characters in rule paths, so every compliant crawler, not just Google’s, can interpret the rule correctly. DevByteDaily’s recent audit of URL structure and character encoding covers the same percent-encoding logic for international and special-character URLs more broadly.
Myth 7: A Disallow Rule Keeps the Page Out of Search Results
Disallow is a crawling control. It is not an indexing control, and the two get confused constantly. A page blocked from crawling can still show up in Google’s results if other pages link to it. It will appear without a normal description because Google never fetched the page to write one.
If you need a page kept out of search results entirely, the correct tool is a noindex directive. Apply it through a meta tag or an HTTP header, not a robots.txt rule. DevByteDaily’s breakdown of how crawling, indexing, and serving actually work walks through why that distinction matters for how long changes take to show up.
One more wrinkle from the video: Google-Agent, the user-triggered fetcher behind certain AI features, ignores robots.txt entirely for a single fetch. That happens because a real person asked for that specific page in that moment. It is a narrow exception, but it is one more reason to stop treating robots.txt as an airtight gate.
How to Audit Your robots.txt Rules for AI Crawlers This Week
None of the above requires rewriting your whole file. It requires reading it the way Google’s parser does, instead of the way you hope it does.
- Open your live robots.txt file and check every User-agent, Allow, Disallow, and Sitemap line for typos. A single dropped letter makes the whole line invisible to the parser.
- Replace any vague or invented AI crawler rule with the exact user-agent tokens you actually want to control, such as GPTBot, ClaudeBot, or PerplexityBot.
- Remove ASCII art, emoji, and long comment blocks. They are harmless individually, but they eat into the roughly 500 KiB limit Google enforces before it stops reading.
- Percent-encode any non-English characters inside rule paths so crawlers other than Google’s can still interpret them correctly.
- Stop relying on Disallow to keep anything out of search results. Use noindex for that job, and save robots.txt for crawling access only.
- Check your server logs against your Disallow rules occasionally. Given how many AI agents still reach blocked URLs in independent testing, a rule you assume is working deserves an occasional second look.
This pairs well with a broader technical pass. If your site runs large category or filter pages, DevByteDaily’s crawl budget audit covers a related problem. It looks at Googlebot spending time on pages that do not matter, which often overlaps with messy robots.txt rules.
What This Video Signals About Google’s 2026 Technical SEO Focus
Non-Canonical Cabinet is not the only technical explainer Google has shipped recently. Earlier in October, Google also rewrote its URL structure guidance and continued its Search Central Live event schedule into new regions. Put together, these point to Google spending more of its 2026 communication budget on crawling fundamentals rather than ranking signals.
That shift makes sense given how much AI agent traffic now hits the average site. More companies now run crawlers that fetch pages for training data, summarization, or live retrieval. That raises the cost of leaving the gap between what robots.txt actually does and what site owners assume it does uncorrected. A misconfigured rule in 2022 mostly affected Googlebot’s crawl pattern. The same mistake in 2026 can mean an AI agent indexes, summarizes, or serves your content in ways you never intended to allow.
In my experience advising on technical SEO audits, the most common failure is not a missing rule. It is a rule that looks correct to a human but gets silently dropped by the parser, exactly the pattern Illyes spent the episode describing. A five-minute manual review catches most of it.
Why This Matters
Robots.txt rules for AI crawlers get treated as a quick fix: add a line, assume it works, move on. Google’s own explainer shows that assumption fails in specific, avoidable ways, from silent syntax errors to the gap between blocking crawling and controlling data use. Read your file the way a parser reads it, name the exact bots you care about, and stop expecting a text file to do a legal agreement’s job.
Frequently Asked Questions
Does robots.txt stop AI companies from training on my content?
No. It can stop a compliant crawler from fetching your pages, but it says nothing about how already-collected content gets used.
Can I block ChatGPT’s or Claude’s crawler without naming it directly?
No. Generic or descriptive rules are not understood by the parser. You need the exact user-agent token, such as GPTBot or ClaudeBot.
Will Google read emojis in my robots.txt rules?
Not in user-agent tokens, which only accept letters and dashes. Google’s parser can read UTF-8 characters, including some emojis, inside rule paths, but percent-encoding is safer for compatibility with other crawlers.
Does a Disallow rule remove a page from Google’s search results?
No. Disallow blocks crawling, not indexing. A disallowed page can still appear in results through links from other pages, just without a normal snippet.
What standard does robots.txt actually follow?
RFC 9309, the internet standard published in 2022 that Google helped formalize after decades of informal convention.
Leave a response