Skip to main content

llms.txt verification test

This page shows the live result of a Doc Detective test that verifies this site's /llms.txt file is reachable, structurally valid, and still tells the truth about the repo it describes.

Why this matters​

llms.txt is the emerging convention for giving AI tools a structured, machine-readable map of a site's content. It works like robots.txt, but for language models and AI-assisted search rather than web crawlers. The file tells AI agents what the site covers, where the key pages are, and who built it.

The problem: llms.txt is a static file. No human clicks it, no browser renders it, and nothing in a normal deployment pipeline checks that it still responds or still says what it's supposed to say. This site's own llms.txt proved that the hard way: a tool swap, a file rename, and two new Vale rules each landed in separate commits over about six weeks, and none of them touched llms.txt, because nothing connected "this fact changed" to "this file also claims that fact." The result was a page claiming three Vale rules when there were five, eleven workflows described as five, and a link to a page that had been renamed out from under it two months earlier. Nobody noticed, because nobody reads llms.txt. It's written for an AI agent, not a person, which is exactly why a person catching the drift by accident was never something to rely on.

The fix wasn't a smarter test bolted onto hand-maintained prose. It was removing the hand-maintenance for the parts that are actually facts. website/static/llms.txt is now generated by scripts/generate-llms-txt.py from a template plus live repo state (the Vale rule count and names, the workflow count) instead of being typed by hand. What still requires human judgment (what each rule does in plain English, which pages are worth linking, the bio) lives in scripts/pipeline-facts.json and the template, hand-maintained on purpose. If a new Vale rule appears with no matching entry in that file, the generator refuses to run rather than silently shipping an inaccurate file.

Generation removes drift for the facts it covers. It doesn't remove the need to test, for two reasons: something has to verify the deployed file actually matches what the generator would currently produce (the generator running correctly once doesn't prove the live site was ever redeployed), and llms.txt also contains root-relative links that this site's own Lychee link-checking config structurally can't see — .lychee.toml resolves root-relative paths against a localhost base specifically so its own localhost-exclude rule skips them. That's how the dead skill.md link survived two months of Lychee runs untouched. This test's link check exists to cover exactly that gap.

Latest result​

Loading latest result…

How the test works​

The test is embedded directly in this page's source as Doc Detective steps and runs the following:

  1. Make a GET request to the live llms.txt endpoint and confirm it returns HTTP 200.
  1. Confirm the response contains at least one ## section heading, which the llms.txt spec requires.
  1. Compare two specific things: the file this step fetches from the live URL (GET https://oddlittlebird.github.io/showcase-site/llms.txt, whatever GitHub Pages currently has deployed), against the content scripts/generate-llms-txt.py computes fresh, right now, from the exact commit this CI run checked out (not a stored file, not a previous run's output — recomputed from the live .vale/styles/Diana/ directory and .github/workflows/ directory on every run). Any difference between those two — stale counts, a stale link, a stale tool name — fails the test. Since a deploy can take a minute or two to finish after a merge, and this step can start running before that deploy has caught up, a first-attempt mismatch retries (up to 10 times, 15 seconds apart) before being treated as real drift rather than a deploy still in flight.
  1. Extract every root-relative link inside the live llms.txt and confirm each one resolves to a real page. This is the check covering the gap in Lychee's own link-checking config, described above.

If the file disappears, loses its required structure, drifts from what the current repo actually contains, or links to a page that no longer exists, one of these steps fails and the badge turns red.

Run the test manually​

If you clone this repo, you can run the test locally against the live site. No local server is needed, because this test hits the deployed URL directly.

npm install -g doc-detective
doc-detective --config doc-detective/config.json --input website/src/pages/doc-detective/llms-txt-test.mdx

To regenerate llms.txt itself from the template and current repo facts:

python3 scripts/generate-llms-txt.py

Run the test on GitHub Actions​

The test runs automatically every day and on every push to main that touches llms.txt, this page, or the generator's source files. To trigger it on demand:

  1. Go to the Actions tab in the GitHub repository.
  2. Select llms.txt verification test.
  3. Click Run workflow.

A separate workflow, Generate llms.txt, regenerates and commits llms.txt whenever its source facts change (a new or removed workflow file, a new or removed Vale rule, or an edit to the template or facts file itself), plus a daily schedule as a safety net.