Robots.txt check: may this crawler fetch this URL?

$0.01 per call · failed calls are never charged · no account or API key needed

Read a site's robots.txt and say whether a crawler may fetch a URL, by RFC 9309 rules as Google applies them (most specific user-agent group, longest matching Allow or Disallow, Allow on ties, * and $ wildcards). Returns the deciding rule, the group used, the rule count and the sitemaps listed. A missing robots.txt (4xx) allows everything. Use before scraping or crawling a site politely; for the sitemap URLs themselves use sitemap.

Run it free (25 free calls) Buy credits by card (from $5) Example result (JSON)

Try it with your own inputs

The page URL to check (its site's /robots.txt is read)

Uses one of your 25 free calls. Nothing is charged, and a call that fails is not counted.

Inputs

NameTypeDescription
url (required)stringThe page URL to check (its site's /robots.txt is read)
agentstringCrawler user agent or product token (default *)

Request

Over HTTP, with a card key from /credits:

curl -H "Authorization: Bearer adt_…" "https://agent-data-tools.revmesh2074.workers.dev/robots?url=https%3A%2F%2Fwww.google.com%2Fsearch"

Or one of your 25 free calls, no key:

curl "https://agent-data-tools.revmesh2074.workers.dev/robots?url=https%3A%2F%2Fwww.google.com%2Fsearch&trial=1"

Without a key, the same URL answers 402 Payment Required with x402 and MPP terms, so an agent with a wallet pays per call in USDC (Base, Polygon, Arbitrum or Solana over x402; Base or Tempo over MPP) and retries.

Example result

{
  "url": "https://www.google.com/search",
  "robotsUrl": "https://www.google.com/robots.txt",
  "status": 200,
  "agent": "Googlebot",
  "allowed": false,
  "matchedRule": "Disallow: /search",
  "group": "*",
  "rulesInGroup": 280,
  "sitemaps": [
    "https://www.google.com/sitemap.xml"
  ]
}

From an AI agent (MCP)

Add the MCP server to Claude, Cursor or any MCP client. This tool is robots, and the first 25 successful calls are free:

https://agent-data-tools.revmesh2074.workers.dev/mcp

Related tools