See Which Crawlers Your Website Allows
A small robots rule can produce different behavior for different user agents. This checker reads public robots directives and page-level signals so you can make an informed decision about discovery and access. It does not claim to crawl your site from the real IP address of every search engine.
Run the tool
Know which bot you're checking
ChatGPT Search vs OpenAI model-training crawler
OpenAI documents OAI-SearchBot for ChatGPT search discovery and GPTBot for training-related crawling. These policies can be controlled separately. Blocking the latter does not necessarily mean blocking ChatGPT Search. Read the current OpenAI crawler documentation before making configuration changes.
Googlebot and Bingbot
Traditional search crawlers remain important because Google Search features use Google's search infrastructure and Bing supports Microsoft's search and AI experiences. A robots allowance does not guarantee indexing, nor does being indexed guarantee appearance in an AI answer.
What a useful result looks like
The tool should show, for each selected user agent, the applicable robots rule and whether the submitted path is explicitly blocked, appears allowed, or cannot be confidently determined. The result must identify any fetch failures or conflicting signals. Robots directives can differ from page-level indexing directives; the output should not blend them into one unexplained score.
Limitations you need to know
Bot-specific server blocks, authentication, geofencing and WAF challenges may not be visible to a generic diagnostic request. Robots rules can change between checks. The correct response to a blocked path depends on whether you actually want that content public. We do not recommend lifting privacy or account protections in pursuit of an AI citation.
Related reading: AI Crawler Access Guide.