Skip to content
tech-seo.de

Bot

TechSEOBot

TechSEOBot is the HTTP client behind tech-seo.de. It is not a crawler: it only fetches URLs that a person or an AI agent explicitly asks it to check, and it does not follow links to discover new pages.

User agent

User agent

User-Agent
Mozilla/5.0 (compatible; TechSEOBot/1.0; +https://tech-seo.de/bot)

Several checks exist to show what a specific crawler receives. They send that crawler's user-agent string instead: by default the Googlebot smartphone UA for redirect, robots.txt, sitemap and hreflang checks, and a set of browser, search and AI crawler UAs side by side in the header inspector. These requests come from Vercel cloud IP ranges, never from Google, OpenAI or Anthropic addresses, so they fail reverse-DNS verification. Render diff uses headless Chromium on the same Vercel IPs and loads the page's subresources. The user agent is the selected crawler preset, or TechSEOBot when no crawler is being imitated. If your CDN blocks unverified crawler user agents, that is working as intended.

Requests

What it requests

  • The URL or domain entered for a check, plus its redirect targets (hop by hop).
  • /robots.txt of that host and of the hosts that serve its CSS, JS and images.
  • Sitemaps listed in robots.txt or entered directly, and a sample of their URLs.
  • Pages linked as hreflang alternates of the entered URL.
  • For render diff, the HTML document and its subresources inside headless Chromium. Images, media and fonts are blocked. Bot verifier never contacts the site that produced the log.

Hard limits per check run (45 seconds max)

  • redirect_matrixup to 120 requests
  • header_inspectorup to 60 requests
  • robots_analyzerup to 40 requests
  • sitemap_auditup to 400 requests
  • hreflang_validatorup to 150 requests
  • bot_verifierup to 0 requests
  • render_diffup to 12 requests

Only ports 80 and 443 on public IP addresses. Response bodies are parsed in memory for the check result (headers, status codes, the HTML head, robots.txt and sitemap XML) and are not stored. Anonymous use is limited to 20 check runs per 10 minutes per IP.

Blocking

robots.txt and blocking

Like other user-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User), TechSEOBot does not obey robots.txt for the URLs it is asked to check, because evaluating robots.txt is one of the checks. To block the self-identified requests, match TechSEOBot in the user agent at your firewall or CDN.