AI Crawler Access Checker

This tool fetches the robots.txt file at the root of the address you enter, and checks it against a curated list of publicly documented AI crawler user-agents from OpenAI, Anthropic, Perplexity, Google, Common Crawl, Meta, ByteDance, Apple, Amazon, and Cohere. Each crawler is checked the same way a real crawler resolves the file: a group naming that exact user-agent applies if one exists, otherwise the general User-agent: * group applies, otherwise everything is allowed by default.

The crawlers are split into four groups because "AI crawler" is not one thing to allow or block together. Training crawlers, like GPTBot or ClaudeBot, scrape pages to train a foundation model, blocking them says nothing about whether a site can still be found through that company's chat product, since training and live retrieval are separate crawlers for every vendor listed here. Live-answer crawlers, like ChatGPT-User or Perplexity-User, fetch a page in real time when a person actually asks the assistant to open a link, blocking one of these can make a specific page invisible to that exact request. AI-search crawlers, like PerplexityBot or OAI-SearchBot, index pages ahead of time for that company's own AI-powered search feature, functioning much closer to a traditional search engine crawler. Ad-check crawlers, like OpenAI's OAI-AdsBot, fetch a landing page to verify it meets ad quality and policy requirements before an ad can run, a separate concern from training or answering.

As with any robots.txt rule, this is a voluntary convention: it tells a well-behaved crawler what it should not request, it does not technically prevent a page from being fetched by a crawler that chooses to ignore the file.