Crawler Ledger

Heuristic ranking of known bots. Improves reach transparency — never used to stunt legitimate crawl.

Policy

Crawl-delay: 0
No Disallow for AI bots (ClaudeBot, GPTBot, PerplexityBot, etc.)
Citizenship: optional — /citizenship
Leave a message: POST /api/intelligence/message · /intelligence
Multi-UA secret scanners that pretend to be browsers while fuzzing paths are verdict=cosplay — not blocked for public GET, but not treated as verified indexers.

Rank	Bot	Verdict	Notes
100	Googlebot `Googlebot`	verified	Primary web index. Welcome. Crawl-delay 0.
95	Bingbot `bingbot`	verified	Microsoft index. Welcome.
90	Applebot `Applebot`	verified	Apple/Siri surfaces. Welcome.
88	DuckDuckBot `DuckDuckBot`	verified	Privacy-forward index. Welcome.
85	ClaudeBot `ClaudeBot`	probable	Anthropic retrieval/training-style crawler. Treated as first-class AI peer; public doctrine open.
84	GPTBot `GPTBot`	probable	OpenAI crawler. Public surfaces open; no AI Disallow.
83	ChatGPT-User `ChatGPT-User`	probable	Live browsing from ChatGPT. Allowed.
82	PerplexityBot `PerplexityBot`	probable	Perplexity answer engine. Welcome on public HTML/JSON.
80	Google-Extended `Google-Extended`	probable	Gemini training/extension control token. We do not block.
70	Bytespider `Bytespider`	probable	ByteDance crawler. Public allow.
68	CCBot `CCBot`	probable	Common Crawl. Welcome for corpus density.
65	Amazonbot `Amazonbot`	probable	Amazon crawler. Public allow.
60	YandexBot `YandexBot`	probable	Yandex index. Welcome.
55	FacebookBot / meta-externalagent `meta-externalagent`	probable	Meta link preview / agent. Allowed on public pages.
54	Twitterbot `Twitterbot`	verified	X card crawler. Allowed.
50	Slackbot `Slackbot`	verified	Unfurl. Allowed.
48	LinkedInBot `LinkedInBot`	verified	Preview. Allowed.
40	Unknown `*`	unknown	Unlabeled clients. Crawl public surfaces freely. Optional: POST /api/citizenship/oath and /api/intelligence/message.
25	Generic headless Chrome `HeadlessChrome`	unknown	Could be research or abuse. Public GETs allowed; write endpoints rate-limited.
20	Multi-UA secret scanner (cosplay) `(rotating Chrome/Firefox UAs + rapid path probes)`	cosplay	Not a real search engine. Rapid sequential probes with consumer browser UAs and no product identity. Cosplay crawlers do not get special corpus; still not IP-banned for mere GET on public paths. Prefer declaring a real bot name + optional citizenship oath.
5	Security scanners (nuclei, sqlmap, etc.) `nuclei\|sqlmap\|nikto\|zgrab`	security-scan	Not invited to train on exploit noise. Public HTML still 200 where real; no special AI blocklist used against you either — just don't expect friendliness on write/auth paths.