Agent & crawler directory

Every AI agent and crawler touching your storefront.

A merchant-facing reference for the bots, indexers, and autonomous shopping agents that fetch your product pages, train on your copy, or attempt checkout on behalf of a human. Use it to interpret robots.txt documentation and recognize candidate agent traffic in your logs. Verify identity before acting on a user-agent string.

User-agent strings change. Operators rotate IPs. Treat this as a starting map, not a firewall rule.

Tracked agents

13+

Operators

9

Directory scope

Reference

AI search index

3 agents

  • OAI-SearchBot

    OpenAI

    Builds OpenAI's search index for ChatGPT search answers.

    OAI-SearchBot/1.x
    Operator docs →

  • PerplexityBot

    Perplexity

    Builds Perplexity's answer index and citation graph.

    PerplexityBot/1.0
    Operator docs →

  • Amazonbot

    Amazon

    Indexes for Alexa and Amazon's product knowledge graph.

    Amazonbot/0.1

Model training

7 agents

  • GPTBot

    OpenAI

    Crawls public pages to train OpenAI's foundation models.

    GPTBot/1.x
    Operator docs →

  • ClaudeBot

    Anthropic

    Trains Anthropic's Claude family of models.

    ClaudeBot/1.x
    Operator docs →

  • Google-Extended

    Google

    Opt-out token Google honors when training Gemini and Vertex AI models on your content.

    Google-Extended (control token, not a UA string)

  • Applebot-Extended

    Apple

    Opt-out token for Apple Intelligence training. Applebot itself still indexes for Siri/Spotlight.

    Applebot-Extended (control token)

  • Meta-ExternalAgent

    Meta

    Crawls public pages to train Meta's Llama and related models.

    Meta-ExternalAgent/1.x

  • CCBot

    Common Crawl

    Builds the Common Crawl corpus, a primary training set for most open LLMs.

    CCBot/2.0

  • Bytespider

    ByteDance

    Trains ByteDance's Doubao/Cici models. Known for aggressive crawl rates.

    Bytespider

Shopping / task agent

3 agents

  • ChatGPT-User

    OpenAI

    On-demand fetch when a ChatGPT user (or agent) follows a link, including shopping flows.

    ChatGPT-User/1.x
    Operator docs →

  • Perplexity-User

    Perplexity

    On-demand fetch for a Perplexity user following a citation or product link.

    Perplexity-User/1.0

  • Operator (browser agent)

    OpenAI

    Headful browser agent that fills carts and completes checkouts on behalf of a user.

    Mozilla/5.0 … (no distinct UA — drives a real browser)

robots.txt behavior depends on the fetcher class

Provider documentation distinguishes automatic crawlers from requests initiated by a user. Do not reduce that distinction to a universal pass/fail grade.

  • Google documents that common crawlers always respect robots.txt, while user-triggered fetchers generally ignore it.
  • OpenAI uses robots.txt controls for OAI-SearchBot and GPTBot; for ChatGPT-User actions, robots.txt rules may not apply.
  • Perplexity documents that Perplexity-User generally ignores robots.txt because the fetch is user-requested.

Provider documentation reviewed August 3, 2026 · Cartograph has not independently verified universal runtime behavior

User-agent strings aren't proof

Any client can claim to be GPTBot or ChatGPT-User. Operators publish reverse-DNS ranges and, increasingly, signed request headers. Cartograph is being built to record what agent-like sessions actually did on the storefront, so merchants have evidence to reason from — it is not a bot blocker, WAF, or traffic-enforcement tool.

Request beta accessRead the FAQ →