Robots.txt Policy Lint
Lint a robots.txt, test paths against RFC 9309, see which AI crawlers it really blocks.
endpoint: https://www.pathwren.workers.dev/c/mcp-registry-official/mcp/robots
registry: dev.workers.pathwren.www/robots-policy-lint
B
recent success 97% (30 probes) ·
MCP handshake OK
· protocol 2025-06-18
TLS valid (47d left) · latency 51.0 ms ·
HTTP 200 · auth open ·
tools 9
manifest dc2855fd11e78060
· drift events seen here: 2
Heuristic flags
- visit-url-instruction — tool instructs the agent to fetch an external URL
Flags are pattern matches on public tool descriptions, not verdicts. Read them as questions to ask the vendor.
Embed this grade
<a href="https://golemreach.com/trust/s/14854"> <img src="https://golemreach.com/trust/badge/14854.svg" alt="golemreach trust grade"></a>
Recent probes
| time (UTC) | ok | grade | mcp | latency | tls | tools | hash |
|---|---|---|---|---|---|---|---|
| 2026-10-13T04:23:24Z | ✓ | B | ✓ | 51.0 ms | 47d | 9 | dc2855fd11e78060 |
| 2026-10-12T22:21:48Z | ✓ | B | ✓ | 53.0 ms | 47d | 9 | dc2855fd11e78060 |
| 2026-10-12T16:20:49Z | ✓ | B | ✓ | 54.0 ms | 48d | 9 | dc2855fd11e78060 |
| 2026-10-12T10:19:40Z | ✗ | F | ✗ | 13349.0 ms | —d | — | — |
| 2026-10-12T04:14:29Z | ✓ | B | ✓ | 54.0 ms | 48d | 9 | dc2855fd11e78060 |
| 2026-10-11T22:12:11Z | ✓ | B | ✓ | 63.0 ms | 48d | 9 | dc2855fd11e78060 |
| 2026-10-11T16:12:03Z | ✓ | B | ✓ | 69.0 ms | 49d | 9 | dc2855fd11e78060 |
| 2026-10-11T10:11:48Z | ✓ | B | ✓ | 60.0 ms | 49d | 9 | dc2855fd11e78060 |
| 2026-10-11T04:10:49Z | ✓ | B | ✓ | 134.0 ms | 49d | 9 | dc2855fd11e78060 |
| 2026-10-10T22:08:45Z | ✓ | B | ✓ | 137.0 ms | 49d | 9 | dc2855fd11e78060 |
| 2026-10-10T16:08:03Z | ✓ | B | ✓ | 100.0 ms | 50d | 9 | dc2855fd11e78060 |
| 2026-10-10T10:06:19Z | ✓ | B | ✓ | 173.0 ms | 50d | 9 | dc2855fd11e78060 |
| 2026-10-10T04:05:28Z | ✓ | B | ✓ | 150.0 ms | 50d | 9 | dc2855fd11e78060 |
| 2026-10-09T22:04:38Z | ✓ | B | ✓ | 84.0 ms | 50d | 9 | dc2855fd11e78060 |
| 2026-10-09T16:03:27Z | ✓ | B | ✓ | 92.0 ms | 51d | 9 | dc2855fd11e78060 |
| 2026-10-09T10:02:21Z | ✓ | B | ✓ | 88.0 ms | 51d | 9 | dc2855fd11e78060 |
| 2026-10-09T04:01:53Z | ✓ | B | ✓ | 82.0 ms | 51d | 9 | dc2855fd11e78060 |
| 2026-10-08T22:01:29Z | ✓ | B | ✓ | 63.0 ms | 51d | 9 | dc2855fd11e78060 |
| 2026-10-08T16:00:55Z | ✓ | B | ✓ | 54.0 ms | 52d | 9 | dc2855fd11e78060 |
| 2026-10-08T10:00:09Z | ✓ | B | ✓ | 206.0 ms | 52d | 9 | dc2855fd11e78060 |
| 2026-10-08T03:59:27Z | ✓ | B | ✓ | 79.0 ms | 52d | 9 | dc2855fd11e78060 |
| 2026-10-07T21:58:41Z | ✓ | B | ✓ | 46.0 ms | 52d | 9 | dc2855fd11e78060 |
| 2026-10-07T15:58:01Z | ✓ | B | ✓ | 131.0 ms | 53d | 9 | dc2855fd11e78060 |
| 2026-10-07T09:56:42Z | ✓ | B | ✓ | 131.0 ms | 53d | 9 | dc2855fd11e78060 |
| 2026-10-07T03:55:22Z | ✓ | B | ✓ | 141.0 ms | 53d | 9 | dc2855fd11e78060 |
| 2026-10-06T21:53:40Z | ✓ | B | ✓ | 74.0 ms | 53d | 9 | dc2855fd11e78060 |
| 2026-10-06T15:52:31Z | ✓ | B | ✓ | 154.0 ms | 54d | 9 | dc2855fd11e78060 |
| 2026-10-06T09:51:42Z | ✓ | B | ✓ | 94.0 ms | 54d | 9 | dc2855fd11e78060 |
| 2026-10-06T03:51:15Z | ✓ | B | ✓ | 181.0 ms | 54d | 9 | dc2855fd11e78060 |
| 2026-10-05T21:49:22Z | ✓ | B | ✓ | 145.0 ms | 54d | 9 | dc2855fd11e78060 |