AI crawlers · technical reference

AI crawlers: identify the purpose before setting controls

Crawler controls depend on the crawler and purpose. Separate search discovery from training-related access rather than treating every AI crawler as interchangeable.

Start free trial Read the guide

Direct answer

Keep the search question explicit.

Read the published crawler documentation and apply controls to the specific user agent and purpose. A control is not a guarantee of how any answer experience will use or cite content.

Reference guide

Concepts and evidence boundaries.

Identify the crawler

Use published documentation to identify the user agent and its stated purpose before changing a control, and record the documented token alongside any change.

Separate discovery from training

OpenAI guidance distinguishes OAI-SearchBot for search discovery from GPTBot training controls; do not collapse those purposes.

Verify ordinary access

Robots directives are crawler access guidance. They do not promise crawling, inclusion, sources, or citations.

What crawler controls cannot guarantee

Allowing or disallowing a documented user agent changes what compliant crawlers may request. It does not determine whether content is used in an answer, cited as a source, or summarized, and it cannot verify crawler identity by itself: pair directives with published documentation and ordinary server logs.

Primary sources

Official guidance reviewed 2026-08-16.

Publishers and developers FAQ

OpenAI. Platform guidance can change; recheck the source before acting.

OpenAI SearchBot

OpenAI. Platform guidance can change; recheck the source before acting.

Related guides

What this reference does not promise.

Are all AI crawlers controlled the same way?

No. Controls and meanings depend on the documented crawler and purpose.

Have a real research question?

Inspect evidence in Gavix.

Use supported workflows and keep their boundaries visible.

Start free trial