大乐GEOTools
Run a full analysisOpen workspace
Switch tool
Brand diagnosis
AI Visibility SnapshotbrandBrand Mention CheckbrandBuyer Question Finderbrand
Website technical checksNeeds a website of your own
Crawl & Access
AI Bot & Crawler Checkersiterobots.txt Checkersitellms.txt Checker & ValidatorsiteAI Bot Policy BuilderbuildHTTP & Redirect Checkpage
Index & Discovery
AI Indexability CheckpageSitemap ValidatorsiteCanonical URL CheckpageMeta Robots Checkpage
Content & Structure
Page Extractability CheckpageSchema Markup CheckerpageFAQ Structure CheckpageMetadata & Heading CheckerpageCitation Link CheckpageContent Freshness Signalspage
Crawl & Access

AI Bot & Crawler Checker

Check robots permission and real server responses for GPTBot, ClaudeBot, PerplexityBot and more.

Free · instant

Run a deterministic technical check

The result comes from HTTP requests, parsers and maintained rules — no AI guessing, so it is reproducible.

The check runs against the site root, e.g. yourbrand.com

Requests are rate-limited and cached. Only public content is fetched.

Account
Not required
How it runs
HTTP requests + parsers + rules
Data
Public web content and the input above only

About this check

01

What does this tool check?

Four gates decide whether AI can read your site: robots.txt permission, whether the server actually lets the crawler UA through (WAFs often 403 GPTBot even when robots.txt allows it), how much content survives without JavaScript, and meta/header blocking directives.

02

How should I read the result?

We probe from our own servers, so we can only test UA-level blocking. Every probe ships with a normal-browser control request: if the control itself fails, verdicts show "inconclusive" — never a false "blocked". Google-Extended and Applebot-Extended are robots tokens only, so they are marked not probeable.

03

What should I do next?

If a crawler is blocked at the server level, fix it in your CDN/WAF bot settings — robots.txt cannot help there.

Sources: OpenAI crawler docs · RFC 9309 Robots Exclusion Protocol

Check this next

robots.txt CheckerParse allow and disallow rules per crawler and highlight conflicts.Page Extractability CheckMeasure how much of your content an AI crawler can actually read without JavaScript.
大乐GEO

Access rules change without warning

robots.txt edits and CDN bot rules usually ship without anyone telling you. The DaleGEO workspace re-checks on a schedule and alerts you when a crawler starts getting blocked.

Run a full analysis
  • Scheduled re-checks with change alerts
  • Visibility and citations across the mainland AI engines
  • Fixes ranked by impact, with content generated
AI Crawler Checker — Can GPTBot Actually Read Your Site?