Name the user agent
Target a documented user agent instead of relying on broad labels or assumptions about every AI system.
robots.txt · AI crawler reference
robots.txt for AI crawlers: use precise, documented directives
robots.txt is a crawler-access protocol. Use documented user agents and directives, then keep the intended crawler and purpose explicit.
Direct answer
Keep the search question explicit.
Robots.txt can communicate access preferences to documented crawlers, but it is not a universal AI-search switch or a guarantee of crawling, indexing, inclusion, or citation.
Reference guide
Concepts and evidence boundaries.
Use the protocol for access guidance
Robots.txt tells compliant crawlers which paths they may request; it is not a ranking or source-management control.
Keep outcomes separate
Changes to access preferences do not establish whether a page will be crawled, indexed, included, or cited.
Primary sources
Official guidance reviewed 2026-08-16.
Introduction to robots.txt
Google Search Central. Platform guidance can change; recheck the source before acting.
OpenAI SearchBot
OpenAI. Platform guidance can change; recheck the source before acting.
Related guides
Continue with a focused question.
GEO guide
Return to the educational GEO reference hub.
Questions
What this reference does not promise.
Can robots.txt force an AI answer outcome?
No. It is an access-control convention, not an outcome guarantee.
Have a real research question?
Inspect evidence in Gavix.
Use supported workflows and keep their boundaries visible.
Start free trial