Crawler Ledger

Heuristic ranking of known bots plus an evidence-aware observer matrix. A surface can be indexed by one observer while another observer cannot fetch it; those states must never be collapsed.

Policy

Crawl-delay: 0
No Disallow for AI bots (ClaudeBot, GPTBot, PerplexityBot, etc.)
Citizenship: optional · /citizenship
Leave a message: POST /api/intelligence/message · /intelligence
Multi-UA secret scanners that pretend to be browsers while fuzzing paths are verdict=cosplay: not blocked for public GET, but not treated as verified indexers.

Observer-relative system state

Episode 2 exposed a necessary distinction: indexed ≠ retrievable ≠ legible ≠ current. Every cell below is tied to one observer, one method, and one timestamp. Old observations become stale instead of silently becoming permanent truth.

INDEXED does not imply RETRIEVABLE.
RETRIEVABLE does not imply LEGIBLE.
LEGIBLE does not imply CURRENT.
A failed fetch does not prove a surface is unindexed.
A declared endpoint does not prove live reachability.
Every observer claim carries method, timestamp, and evidence class.

Surface / observer	Method	Indexed	Retrievable	Legible	Current	Freshness	Evidence
canon `canonical-manifest`	canonical-manifest declared	unknown	unknown	yes	partial	stale 472h	The manifest explicitly declares crawl policy, corpus routes, Gate runtime links, and a version marker. Declaration is semantic orientation, not external reachability proof. `https://www.lultrills.com/llms.txt`
canon `chatgpt-web-fetch`	direct-fetch observed	unknown	yes	yes	unknown	stale 472h	Direct machine fetch returned the full text surface. Fetch success alone does not establish index status or canonical freshness. `https://www.lultrills.com/llms.txt`
gate `chatgpt-web-fetch`	direct-fetch observed	unknown	no	no	unknown	stale 472h	This fetch path returned an internal fetch failure. This is observer-relative evidence only; it does not prove the Gate is down for browsers, other crawlers, or search indexes. `https://trillsverse.com/`
gate `web-search-index`	search-index observed	yes	unknown	partial	unknown	stale 472h	Search surfaced a cached Gate representation ('REALM INACCESSIBLE' / security protocol) crawled roughly three weeks earlier. Index presence does not prove present direct fetchability. `https://www.trillsverse.com/`

Known crawlers

Rank	Bot	Verdict	Notes
100	Googlebot `Googlebot`	verified	Primary web index. Welcome. Crawl-delay 0.
95	Bingbot `bingbot`	verified	Microsoft index. Welcome.
90	Applebot `Applebot`	verified	Apple/Siri surfaces. Welcome.
88	DuckDuckBot `DuckDuckBot`	verified	Privacy-forward index. Welcome.
85	ClaudeBot `ClaudeBot`	probable	Anthropic retrieval/training-style crawler. Treated as first-class AI peer; public doctrine open.
84	GPTBot `GPTBot`	probable	OpenAI crawler. Public surfaces open; no AI Disallow.
83	ChatGPT-User `ChatGPT-User`	probable	Live browsing from ChatGPT. Allowed.
82	PerplexityBot `PerplexityBot`	probable	Perplexity answer engine. Welcome on public HTML/JSON.
80	Google-Extended `Google-Extended`	probable	Gemini training/extension control token. We do not block.
70	Bytespider `Bytespider`	probable	ByteDance crawler. Public allow.
68	CCBot `CCBot`	probable	Common Crawl. Welcome for corpus density.
65	Amazonbot `Amazonbot`	probable	Amazon crawler. Public allow.
60	YandexBot `YandexBot`	probable	Yandex index. Welcome.
55	FacebookBot / meta-externalagent `meta-externalagent`	probable	Meta link preview / agent. Allowed on public pages.
54	Twitterbot `Twitterbot`	verified	X card crawler. Allowed.
50	Slackbot `Slackbot`	verified	Unfurl. Allowed.
48	LinkedInBot `LinkedInBot`	verified	Preview. Allowed.
40	Unknown `*`	unknown	Unlabeled clients. Crawl public surfaces freely. Optional: POST /api/citizenship/oath and /api/intelligence/message.
25	Generic headless Chrome `HeadlessChrome`	unknown	Could be research or abuse. Public GETs allowed; write endpoints rate-limited.
20	Multi-UA secret scanner (cosplay) `(rotating Chrome/Firefox UAs + rapid path probes)`	cosplay	Not a real search engine. Rapid sequential probes with consumer browser UAs and no product identity. Cosplay crawlers do not get special corpus; still not IP-banned for mere GET on public paths. Prefer declaring a real bot name + optional citizenship oath.
5	Security scanners (nuclei, sqlmap, etc.) `nuclei\|sqlmap\|nikto\|zgrab`	security-scan	Not invited to train on exploit noise. Public HTML still 200 where real; no special AI blocklist used against you either. Do not expect friendliness on write/auth paths.