robots.txt analyzer

robots_analyzer

Parses robots.txt like Google (RFC 9309) and evaluates the URL for search engines and AI crawlers.

What it does

Fetches and parses the robots.txt of a URL's origin exactly like Google (RFC 9309: most specific group, longest match, allow wins ties, HTTP status rules) and evaluates the URL for search engines and AI crawlers (Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot, Google-Extended and more). Also checks noindex vs. robots conflicts, whether CDN hosts block CSS/JS for Googlebot, and whether Sitemap directives resolve. Use when asked whether a page or bot is blocked, or why Google cannot see a noindex. Returns per-bot allow/deny with the matched rule. Takes 2 to 6 seconds.

~4 s · up to 40 requests · v1.0.0

Call it from code

REST endpoint (POST)

https://tech-seo.de/api/v1/checks/robots_analyzer

MCP tool name

robots_analyzer

cURL

curl -X POST https://tech-seo.de/api/v1/checks/robots_analyzer \
  -H 'content-type: application/json' \
  -d '{"url":"https://example.com/"}'

Parameters and result format →