Checks
100+ checks.
| Id | Check | Severity | Last run | Run |
|---|---|---|---|---|
| Crawler access & permissions 20 | ||||
| A1 | File present at root, valid syntax, no malformed or orphaned rules. | High | ||
| A2 | Explicit directives for GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot and peers — not just User-agent: *. | High | ||
| A3 | Wildcard rule doesn't lock the root away from every crawler. | Critical | ||
| A4 | robots.txt Content-Signal (or IETF AI-Preferences) statements of what AI may do with the content. | High | ||
| A5 | noindex / noarchive / noai / noimageai in tags or headers that quietly override robots.txt. | Critical | ||
| A6 | Whether a security layer blocks well-behaved agents even when robots.txt permits them. | Critical | ||
| A7 | Actually requests the page as the agents of ChatGPT, Claude, Perplexity, Gemini, Meta AI, Apple Intelligence and Alexa instead of trusting the stated policy. | Critical | ||
| A8 | Googlebot and Bingbot still allowed — a proxy for accidental over-blocking. | High | ||
| A9 | tdm-reservation header, meta element or /.well-known/tdmrep.json declaring text-and-data-mining rights. | High | ||
| A10 | Cross-checks that the endpoints published manifests advertise actually answer. | High | ||
| A11 | Whether robots.txt distinguishes the bots that train models from the bots that fetch a page when a user asks about you. | High | ||
| A12 | Whether the page still serves its substance to a client that accepts no cookies. | Critical | ||
| A13 | GET endpoints that refuse an agent without a CSRF or session token it has no way to obtain. | High | ||
| A14 | Whether a 429 carries Retry-After and a machine-readable body rather than an HTML error page. | High | ||
| A15 | Human-verification widgets attached to contact, newsletter or signup forms. | High | ||
| A16 | A Really Simple Licensing (RSL) document, found from robots.txt, a Link header or the page, stating what AI systems may do with the content and on what terms. | High | ||
| A17 | Whether AI crawlers are answered 402 Payment Required with a crawler-price, which prices a page rather than blocking it. | High | ||
| A18 | Requests robots.txt as GPTBot, ClaudeBot and PerplexityBot and compares each answer with the copy everyone else gets. | High | ||
| A19 | Whether robots.txt disallows the scripts and stylesheets the homepage loads from its own host, for Googlebot and Applebot. | High | ||
| A20 | Requests the sign-in, cart and checkout pages the homepage links to as a browser and as ChatGPT-User and Claude-Agent, and looks for a CAPTCHA on them. | High | ||
| Discovery surfaces 12 | ||||
| B1 | Sitemap reachable at the standard path and referenced from robots.txt. | High | ||
| B2 | A markdown index for language models at /llms.txt. | High | ||
| B3 | H1 name, blockquote summary, H2 sections and correct link syntax. | High | ||
| B4 | Extended single-file concatenation of the full markdown corpus. | High | ||
| B5 | rel=describedby, api-catalog, service-desc, alternate advertised in response headers. | High | ||
| B6 | Root-level instruction files written for agents rather than crawlers. | High | ||
| B7 | Presence of the well-known namespace agents probe first (RFC 8615). | High | ||
| B8 | SVCB/HTTPS records under _agents.<domain> advertising agent endpoints. | High | ||
| B9 | Human-readable URLs that keep working — treated as a public contract. | High | ||
| B10 | JSON, RSS/Atom or markdown endpoints served alongside HTML. | High | ||
| B11 | Navigation built from real anchors rather than click handlers on divs. | High | ||
| B12 | Whether the pages behind a listing can be reached by following links. | High | ||
| Content accessibility & extractability 16 | ||||
| C1 | The substance is in the first response, not injected by client-side JS. | Critical | ||
| C2 | Measures what share of visible text survives with JavaScript switched off. | High | Needs a browser | |
| C3 | Returns text/markdown when an agent sends Accept: text/markdown. | High | ||
| C4 | A .md sibling or /index.md fallback for every HTML page. | High | ||
| C5 | One h1, logical order, real landmarks instead of div soup. | High | ||
| C6 | Docs, pricing and key pages reachable without authentication. | High | ||
| C7 | Compares the response served to an AI user-agent against the baseline browser response. | Critical | ||
| C8 | Main content small enough to be read whole — under 8,000 tokens, failing above 25,000. | High | ||
| C10 | Buyer questions answered directly in a quotable Q&A shape. | High | ||
| C11 | Scannable structures an engine can lift verbatim. | High | ||
| C14 | Whether an AI user-agent is shown different claims, not merely different markup. | High | ||
| C15 | Whether comments, reviews and other user submissions are structurally separable from what the site itself says. | High | ||
| C16 | Prices, specifications and tables that exist only as pixels. | High | Needs a browser | |
| C17 | How far into the page the substantive answer begins. | High | ||
| C18 | Whether asking for a language actually returns it. | High | ||
| C19 | Where the main content starts and ends in the page's text, against the character caps of Cline (50,000), Claude Code (100,000) and Gemini CLI (250,000). | High | ||
| Structured data & metadata 10 | ||||
| D1 | Structured data on the page and syntactically correct. | High | ||
| D2 | Organization, Product, Article, FAQ, LocalBusiness rather than generic markup. | High | ||
| D4 | Unique, descriptive, present in the raw HTML. | High | ||
| D5 | Correct rel=canonical, no conflicting signals. | High | ||
| D6 | Social/preview metadata completeness. | High | ||
| D7 | Links out to Wikipedia, Crunchbase, LinkedIn that tie the site to a known entity. | High | ||
| D8 | Declarative actions an agent could invoke. | High | ||
| D9 | schema.org speakable for voice agents. | High | ||
| D10 | Valid lang attribute; hreflang where relevant. | High | ||
| D11 | Conflicting or duplicated Organization identities in the structured data. | High | ||
| Agent capability & protocol exposure 15 | ||||
| E1 | A live Model Context Protocol server (streamable HTTP or SSE). | Medium | ||
| E2 | /.well-known/mcp/server-card.json describing the server (SEP-1649). | Medium | ||
| E3 | WebMCP tools on the page: forms declared with toolname and tooldescription, and tools registered on document.modelContext. | Medium | ||
| E4 | /.well-known/agent-card.json for agent-to-agent delegation. | Medium | ||
| E5 | Machine-readable API description, optionally with agent extensions. | Medium | ||
| E6 | /.well-known/api-catalog indexing the site's APIs. | Medium | ||
| E7 | An agent-skills index at /.well-known/skills.json. | Medium | ||
| E9 | Conversational endpoint returning schema.org-typed JSON. | Medium | ||
| E10 | UI resources an agent can render on the user's behalf. | Medium | ||
| E11 | agent-permissions.json, UCP capability profile, or an equivalent declaration of what agents may do. | Medium | ||
| E12 | Whether search runs as a GET with a named parameter or only as a JavaScript POST. | Medium | ||
| E13 | Whether the advertised MCP server answers tools/list with usable tool definitions. | Medium | ||
| E14 | Whether the paths in the OpenAPI spec actually answer. | Medium | ||
| E15 | Whether an endpoint that is going away says so in a header an agent can read. | Medium | ||
| E16 | Whether the advertised MCP endpoint is public HTTPS on a host with an IPv4 address and no redirect to another host. | Medium | ||
| Identity, trust & auth 15 | ||||
| F1 | /.well-known/http-message-signatures-directory so agents can prove who they are (RFC 9421). | Medium | ||
| F2 | /.well-known/openid-configuration or RFC 8414 authorization-server metadata. | Medium | ||
| F3 | RFC 9728 metadata for the resource an agent wants to reach. | Medium | ||
| F4 | RFC 7636 S256 required of agent-grade OAuth clients. | Medium | ||
| F5 | Secure transport and a clean certificate chain. | Medium | ||
| F6 | The trust furniture an engine looks for before citing you. | Medium | ||
| F7 | rel=author, JSON-LD author/publisher, named bylines. | Medium | ||
| F10 | security.txt, and no sensitive paths (admin, .env, .git) leaked through robots.txt. | Medium | ||
| F11 | A did:web identity document at /.well-known/did.json. | Medium | ||
| F12 | Whether a request signed under RFC 9421 is treated differently from an unsigned one. | Medium | ||
| F13 | Whether the published key directory is a valid JWK Set a verifier can actually load. | Medium | ||
| F14 | Text hidden from readers but visible to parsers, carrying instructions rather than content. | Medium | ||
| F15 | Forms that send, book or charge, with nothing in front of them. | Medium | ||
| F16 | Whether an agent can authenticate on its own, or needs a human at a login screen. | Medium | ||
| F17 | Walks the MCP authorization chain from an unauthenticated call: 401, protected resource metadata, the first authorization server, issuer, S256, client registration, and a token endpoint that takes form posts. | Medium | ||
| Agentic commerce & payments 13 | ||||
| G1 | A paid endpoint answering an unauthenticated request with a decodable 402. | Low | ||
| G2 | Payment discovery document with network, facilitator and services array. | Low | ||
| G3 | /.well-known/acp.json, including delegated-payment tokens. | Low | ||
| G4 | /.well-known/ucp merchant profile. | Low | ||
| G5 | WWW-Authenticate: Payment challenge and required parameters. | Low | ||
| G6 | Stripe SPT, L402, Coinbase Commerce, AP2 and similar. | Low | ||
| G7 | A machine-readable catalogue an agent can find without crawling every page. | Low | ||
| G8 | Prices declared in structured data, and MCP tools or API operations an agent can order through. | Low | ||
| G9 | The same price in the visible text, the structured data, the preview tags and the catalog feed. | Low | ||
| G10 | Stock state consistent between the page and its machine-readable claims. | Low | ||
| G11 | Whether numbers carry their own meaning or depend on the reader's locale. | Low | ||
| G12 | The Verifiable-Credentials authorisation layer beneath an agentic checkout. | Low | ||
| G13 | Whether the card fields on the checkout page carry cc-number, cc-exp, cc-csc and cc-name autocomplete tokens. | Low | ||
| Freshness 6 | ||||
| H1 | Human-readable dates on the page. | Low | ||
| H2 | Freshness expressed in machine-readable form. | Low | ||
| H3 | Sitemap actually reports meaningful modification dates. | Low | ||
| H4 | Server headers not signalling stale content to crawlers. | Low | ||
| H5 | Whether the machine-readable catalog is as current as the pages it describes. | Low | ||
| H6 | Whether a repeat visit carrying If-None-Match or If-Modified-Since gets a 304. | Low | ||
| Performance & reliability 11 | ||||
| I1 | Latency measured from the crawler's side, not a browser's. | Low | ||
| I2 | Real-user field metrics folded into the readiness score. | Low | ||
| I3 | Scrunch fails pages slower than five seconds outright. | Low | Needs a browser | |
| I4 | 200s, no long or cross-host redirect chains, correct handling of 403/404/5xx. | Low | ||
| I5 | Infrastructure failures that only show up in AI-crawler traffic. | Low | ||
| I6 | How much output an agent burns to get an answer. | Low | ||
| I7 | How much of the delivered payload is the content an agent came for. | Low | Needs a browser | |
| I8 | Whether the substance arrives in the first response or several requests later. | Low | Needs a browser | |
| I9 | What a client that does not wait for the page to settle actually receives. | Low | Needs a browser | |
| I10 | Whether the page is served gzipped or brotli-compressed to a client that asks for it. | Low | ||
| I11 | Whether the homepage's links are under 250 characters and its linked documents are served inline in a type fetch tools read. | Low | ||
| Accessibility (WCAG) 14 | ||||
| J1 | Text alternatives on every meaningful image. | Medium | ||
| J2 | Every input and button named, every control reachable. | Medium | ||
| J3 | Valid roles and attributes, named dialogs and tooltips, nothing focusable hidden from the accessibility tree, no positive tabindex, and no dangling aria-labelledby / aria-describedby targets. | Medium | ||
| J4 | A real title, a main landmark, named iframes, and landmarks that can be told apart. | Medium | ||
| J5 | What a screen reader announces matches what the eye reads. | Medium | Needs a browser | |
| J6 | Structural markup an assistive client can navigate. | Medium | ||
| J7 | No zoom suppression; explicit media dimensions. | Medium | ||
| J8 | autocomplete attributes that let assistive tooling fill forms. | Medium | ||
| J9 | Every input, select and textarea carrying the name its form needs to submit it. | Medium | ||
| J10 | Selects, dates and file inputs built from real elements rather than reconstructed in JavaScript; no field that can only be picked, no item that can only be dragged, no page drawn on a canvas. | Medium | ||
| J11 | Sticky headers, footers and floating widgets covering the controls beneath them. | Medium | Needs a browser | |
| J12 | Every link says where it goes, including the icon-only ones. | Medium | ||
| J13 | Counts the buttons and form fields that sit inside iframes from another origin. | Medium | Needs a browser | |
| J14 | Whether the page virtualises lists or removes rows as it scrolls, which leaves agents' references stale. | Medium | Needs a browser | |