AI फ़ेचर, एजेंट, और नियंत्रणों का एक स्रोत-समर्थित नक्शा।

User-triggered fetcher used for certain actions in ChatGPT and Custom GPTs; it is not an automatic web crawler.

रोबोट का व्यवहार

OpenAI says robots.txt rules may not apply because these requests are user-initiated.

पहचान का प्रमाण

OpenAI publishes IP ranges for ChatGPT-User requests; the user-agent string remains spoofable.

Retrieves web content in response to user-initiated Claude requests.

रोबोट का व्यवहार

Anthropic says its bots honor robots.txt directives, including Claude-User.

पहचान का प्रमाण

Anthropic's crawler guidance links a current source-IP list; do not trust the token alone.

User-triggered fetcher that may visit a page to help answer a Perplexity user's question; it is not used for web crawling or foundation-model training.

रोबोट का व्यवहार

Perplexity says this fetcher generally ignores robots.txt because the request was initiated by a user.

पहचान का प्रमाण

Perplexity publishes IP ranges for Perplexity-User; combine the token and source IP rather than trusting the token alone.

Used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request.

रोबोट का व्यवहार

Google says user-triggered fetchers generally ignore robots.txt rules because the fetch was requested by a user.

पहचान का प्रमाण

Google publishes user-triggered-agent IP ranges and is experimenting with Web Bot Auth for the agent.bot.goog identity.

Supports user actions such as fetching current web information to answer Alexa queries on a user's behalf; Amazon says it is not used for generative-AI model training.

रोबोट का व्यवहार

Amazon says because Amzn-User actions can be user-initiated, it may not follow all robots.txt directives.

पहचान का प्रमाण

Amazon publishes IP addresses for Amzn-User; use source verification in addition to the token.

Automatic crawler used to surface websites in ChatGPT search results.

रोबोट का व्यवहार

Managed with the OAI-SearchBot token in robots.txt.

पहचान का प्रमाण

Match the token with OpenAI's published SearchBot IP ranges; a user-agent string alone is not proof.

Navigates the web to improve the relevance and accuracy of Claude search responses.

रोबोट का व्यवहार

Anthropic says its bots honor robots.txt directives.

पहचान का प्रमाण

Anthropic's crawler guidance links a current source-IP list; do not trust the token alone.

Automatic crawler designed to surface and link websites in Perplexity search results; Perplexity says it is not used to crawl content for foundation-model training.

रोबोट का व्यवहार

Perplexity recommends managing PerplexityBot through robots.txt.

पहचान का प्रमाण

Perplexity publishes IP ranges for PerplexityBot; combine the token and source IP rather than trusting the token alone.

Apple web crawler used for search experiences including Spotlight, Siri, and Safari; Apple says crawled data may also support foundation-model training and current-content context for AI outputs.

रोबोट का व्यवहार

Applebot respects standard robots.txt directives in general search crawls.

पहचान का प्रमाण

Apple documents reverse-DNS verification under *.applebot.apple.com and publishes Applebot IP CIDR ranges.

Amazon search crawler used to improve search experiences in Amazon products and services, including eligibility for experiences such as Alexa; Amazon says it is not used for generative-AI model training.

रोबोट का व्यवहार

Amazon documents robots.txt allow/disallow support for its crawlers.

पहचान का प्रमाण

Amazon publishes IP addresses for Amzn-SearchBot; use source verification in addition to the token.

Automatic crawler for content that may be used to train OpenAI generative AI foundation models.

रोबोट का व्यवहार

Managed separately with the GPTBot token in robots.txt.

पहचान का प्रमाण

Match the token with OpenAI's published GPTBot IP ranges; a user-agent string alone is not proof.

Collects public web content that could potentially contribute to Anthropic model training.

रोबोट का व्यवहार

Anthropic says its bots honor robots.txt directives.

पहचान का प्रमाण

Anthropic's crawler guidance links a current source-IP list; do not trust the token alone.

Amazon crawler used to improve products and services; Amazon says collected content may also be used to train Amazon AI models.

रोबोट का व्यवहार

Amazon documents robots.txt allow/disallow support for its crawlers.

पहचान का प्रमाण

Amazon publishes IP addresses for Amazonbot; use source verification in addition to the token.

Publisher control for whether Google-crawled content may be used for future Gemini model training and specified grounding uses.

रोबोट का व्यवहार

Google-Extended is a robots.txt product-control token and does not affect Google Search inclusion or ranking.

पहचान का प्रमाण

Not a request identity. Do not look for Google-Extended as a standalone HTTP user-agent in traffic logs.

Publisher control for whether Applebot-crawled content may be used to train Apple's general-purpose foundation models.

रोबोट का व्यवहार

Applebot-Extended is configured in robots.txt but does not itself crawl webpages.

पहचान का प्रमाण

Not a request identity. Traffic should be attributed to Applebot, not Applebot-Extended.

Automated crawler that collects public web data for Common Crawl's open web-crawl repository.

रोबोट का व्यवहार

Common Crawl documents robots.txt support for CCBot.

पहचान का प्रमाण

Common Crawl publishes dedicated IP ranges and reverse-DNS guidance; the project also warns that clients can falsely claim the CCBot user-agent.