Two failure emails​

I got a couple of emails the other morning saying that two of my scheduled tests had failed: the llms.txt verification test and the API reference screenshot test. As one does, I asked my robot minion (Claude Code) to find out why.

Neither test had actually run. Every workflow in my repo runs on a temporary machine that GitHub supplies, and the message on both jobs said no machine ever picked the job up. GitHub's status page showed an Actions incident that started about 20 minutes before my tests were scheduled. The tests that ran earlier and later that day all passed. The fix was to wait for the next scheduled run. So, no big deal.

The notice next to the failures​

Claude Code also pointed out a notice that GitHub had attached to every run, passing or failing. Starting October 19, the ubuntu-latest label moves to Ubuntu 26.

Every one of my workflows had the line runs-on: ubuntu-latest. That line doesn't name a version. It asks for whatever GitHub currently calls the latest one. On October 19, all of my workflows would move to a new operating system, and I wouldn't have changed a thing in my repo.

Most of my pipeline probably wouldn't notice. The screenshot tests are the ones I'd worry about. They compare a new screenshot against a saved one, and a different operating system can render fonts and spacing a little differently. A test could fail over a few pixels, and I'd spend a morning chasing a problem that was never in my content. No thank you!

I'd seen this before​

Pinning had come up before. Earlier this year, I'd pinned Doc Detective to a specific version so that an update couldn't break my tests without warning. In September, I found out the hard way that the pin didn't cover a browser driver that Doc Detective downloads on its own (grr). That one's written up in I'm the supervisor, not the engineer.

The fix here made sense to me right away. Claude Code changed every workflow from ubuntu-latest to ubuntu-24.04, the version they were already running on. That's 16 jobs across 14 workflow files. They run exactly as before, but the upgrade now happens when I decide it does, instead of on GitHub's calendar.

The pin isn't permanent. GitHub will eventually retire Ubuntu 24.04, so at some point I'll try Ubuntu 26 on a separate branch, refresh any saved screenshots that come out different, and then take the pin out. In fact, I told Claude Code to remind me in November so that I can test it out after the initial flurry of fixes and release pains are past.

Somebody used to do this for me​

In my previous life, I was a technical writer at a company where an ops team handled this kind of thing. I never saw a runner label. If a build machine changed under me, then someone else whose job it was knew about it first and dealt with it. I never heard about it.

Owning the repo means I'm that someone now. It also means I can talk about the pros and cons of pinning a version with an engineer and actually contribute meaningfully to the conversation. It's incredibly empowering.

If Git and pipelines still feel like black magic to you, set up a project of your own and own it for a while. You'll run into things you never expected to learn about, and you'll be glad you did.

It's a fair question, and I didn't want to just answer it with a pep talk. This is what happened on my own pipeline this week, including the part I'm less proud of.

How I work with Claude Code​

Claude Code is my assistant and pocket engineer. My account settings tell it, "I am a technical writer, not an engineer," so it explains things to me the way it would to a technical layperson. It does the investigating and the building. I decide what gets built, and when an explanation loses me, I ask for further clarification. Unlike a human engineer, Claude Code never gets frustrated or annoyed with my questions.

One more habit, which is my own quirk: Claude Code commits its changes to a separate branch, and sometimes pushes that branch to GitHub, but I'm always the one who does the final merge into the live site. I like having that manual review step, even when I'm confident in the change.

Two red tests​

This site runs a set of Doc Detective tests on a schedule. Each one checks something the site claims, and each has a status badge on its page. Earlier this week, two of them started failing: the Pipeline test and the API reference screenshot test.

I saw the red in the GitHub Actions tab, pasted the link into Claude Code, and told it those two tests were failing. That was the whole request. (Sometimes I actually go into the logs and paste more details, but I didn't this time.)

It found three things, in this order:

• The failures weren't about my content. The pipeline page still had exactly the heading the test looks for.
• It ran the tests on my laptop and got the same failure. Then it pointed a test at example.com, a page with nothing to do with my site, and that failed too. So the problem was in the testing tool.
• Doc Detective drives a real browser, and to do that it downloads some helper software the first time it runs. One of those helpers, a Chrome driver, released a new version on September 14 that broke the step where a test looks for something on the page. The timing lined up almost to the minute: the pipeline test passed at 16:05 UTC, the new driver came out at 16:18, and the API reference test failed for the first time at 16:51.

Here's the part that surprised me: I'd pinned Doc Detective to a specific version so an update couldn't break things without warning. It turns out that pin doesn't cover the helpers Doc Detective downloads. Those still came in at whatever the newest version was.

The fix was to hold that one driver at its previous version. Claude Code's first attempt at that didn't stick, because Doc Detective's own install step quietly replaced it with the new version again. Claude Code caught that in its own testing before it ever showed me the change, tried a different approach, and confirmed both tests passed. It also told me the version hold is temporary. Once a fixed driver ships, the hold should come out, or I'll miss future fixes too.

I merged the change and watched the Actions tab. Both tests went green.

Why were there so many deploys?​

Then I noticed something else in the Actions tab. My one merge had set off a stack of Deploy runs, and several had a gray warning icon. Deploy is the workflow that rebuilds the site and publishes it. I only expected one deploy per merge, so I asked what was going on.

The gray icons turned out to be harmless. Only one deploy can run at a time, so when several line up, GitHub cancels the older ones that are still waiting and runs the newest. The cancelled ones never used any time or resources.

But Claude Code also checked the deploy history and found something I hadn't known: the site was rebuilding and redeploying itself seven times a day. Most of my scheduled tests save a small status file when they finish, and each one triggered its own deploy. Nothing had broken, and because the repo is public, the extra runs didn't cost anything. It was just a lot of redundant work.

Claude Code gave me three options, each with its trade-off:

• Deploy once a day.
• Deploy only when a test result actually changes.
• Leave it alone and just fix two tests whose results weren't getting published at all.

I pointed out that my Doodle test captures Google's homepage every day, so at minimum the site needs a daily deploy. Once a day covered that, so that's what I picked.

The mistake I caught by asking a plain question​

Before merging, I asked Claude Code one more thing: do any of the pipeline descriptions or test write-ups need to change because of this?

That question turned up a real problem in the change itself. Claude Code had scheduled the daily deploy to run after the check it described as the last one of the day. It wasn't the last one. The doodle test runs twice a day, and its second run is a fallback for days when Google hasn't put up the new doodle yet. That second run happens after the deploy would have run, so on exactly the days the fallback exists for, the doodle page would have shown yesterday's image until the next day.

Claude Code said so directly, said it should have checked every schedule before calling anything the last one, and laid out new options. I chose to deploy after each doodle run: two deploys a day instead of seven. The same review also found two descriptions on the site that were already wrong before this change, and I had it fix those in the same branch.

I never wrote a line of that code. What I did was ask whether the documentation still matched the system. As a technical writer, that's the question I'd ask anyway.

Was seven deploys a day a mistake?​

An engineer designing the whole pipeline in one sitting probably would have set up one deploy from the start, but nobody designed my pipeline in one sitting. It grew one test at a time over several months, and each test's deploy trigger made sense when I added it. When I wrote Doc Detective dominos, triggering a deploy after the doodle test was the fix.

No one had ever looked at the deploy setup as a whole, including me.

The "last check of the day" error bothers me more. Claude Code stated it with complete confidence, and it would have gotten past me if I hadn't asked a follow-up question. I wrote about the same gap in What Claude Code didn't notice.

Back to the technical writer's question​

Could they build this? I think so. Everything I did this week came down to noticing that something looked off in the Actions tab and bringing it to Claude Code, then basically having a conversation with it.

At the start of the week, I didn't know that Doc Detective's helpers ignore my version pin, that my site was deploying seven times a day, or how GitHub handles a queue of deploys. None of that was a prerequisite. I found out each one as it came up.

Now, I know more than I did at the start of the week. I'm building my expertise as I build the pipeline, iterating as I go. And that part is working exactly as designed.

Until last week, it had no front matter. None. It opened on an H1. That's one of the dangers when you build something with an AI coding assistant like Claude Code, I guess.

Vale read it. Doc Detective read it. Docusaurus built it. Every check passed, including me physically looking at it.

Adding a third gate​

I had Claude Code add Manny Silva's new tool, docmeta, to the pipeline this week. The split is clean. Vale checks how the prose reads, Doc Detective checks whether the procedures still work, and docmeta checks the metadata. Before this, nothing in my stack looked at front matter at all.

One thing to know before you try docmeta: It needs Node 24. The Get started page says Node 20 or newer, but the package has required Node 24 since its first release, and npm reports that mismatch as a warning rather than an error. So the install says it succeeded, docmeta --version answers normally, and the thing quietly doesn't work. My CI workflows ran Node 22, which would have produced the same silent non-failure. Claude wrote the whole thing up as a bug report that I sent to Manny, so the Get started should be updated very soon.

The first run told me nothing​

docmeta ships with schemas for Docusaurus front matter, so I started with those. Thirty files across my docs, blog, and pages directories:

30 files checked, 30 passed, 0 failed, 0 errors

A clean sweep, which sounds like good news but was actually the entire problem.

The built-in schema is honest about this in its own description: "Docusaurus marks no field as required, so this is a format check, not a presence check." It constrains the shape of every field Docusaurus accepts, and it requires none of them. A file with no front matter whatsoever satisfies it completely, which is how pipeline.md passed.

Eleven lines changed the answer​

So I had Claude write a schema of my own. Eleven lines of properties:

{
  "type": "object",
  "additionalProperties": true,
  "required": ["title", "description", "last_reviewed"],
  "properties": {
    "title": { "type": "string", "minLength": 1 },
    "description": { "type": "string", "minLength": 1 },
    "last_reviewed": { "type": "string", "format": "date" }
  }
}

The first version required all three fields, which failed all 30 files and told me nothing except that I had written a strict schema. I asked for the honest number instead: what fails on title and description alone, two fields I was already using. On a site I built specifically to demonstrate documentation quality:

24 of 30 files failed

Six passed. All six live in src/pages. Not one file in docs/ or blog/ had a description. Fifteen blog posts, zero descriptions, every one of them written by someone who does this for a living.

description is the field that feeds search results and AI retrieval. It had been missing from almost everything for months, and no gate I owned had any opinion about it.

The date that is not a date​

Claude built a test file with a title, a description, and last_reviewed: 2024-13-45, which is month thirteen, day forty-five. Against the Docusaurus schema:

1 file checked, 1 passed, 0 failed, 0 errors

Against my schema:

/last_reviewed  must match format "date"  (line 4)

That gap is the whole argument. If I ever build a staleness report on last_reviewed, the report doesn't error when the field is garbage. It just returns a wrong answer, and nothing in the output marks it as one.

Two things I got wrong​

The check that would never have run. Claude wired docmeta into my existing lint workflow, and the plan was to push the check on its own first, so I could screenshot it failing before the fix landed. It caught the problem on its own, in the last check before pushing: that workflow triggers on changes to content paths, website/docs/**, website/blog/**, and so on. The new commit touched only the config file, the schema, and the workflow itself. None of those matched, so the workflow would have been skipped, not failed. I would have had a green pull request, a check that never ran, and no way to tell the difference from the outside. A check that doesn't run is worse than no check, because the green tick still shows up.

A rule that had been failing invisibly. Vale started complaining about spaced em dashes in a line the metadata work had just added, and I said so, mostly because I wanted the rule gone. Claude went looking and came back with something I hadn't asked about: that same rule was already firing fourteen times in pipeline.md, on text that has been sitting there for months. I had never seen a single one, because reviewdog only comments on lines inside the diff. Those lines hadn't changed since the rule landed, so the errors were real, error level, and completely invisible. My linter had been finding things and telling nobody.

What I actually changed​

Descriptions now exist on every published page, and pipeline.md finally has front matter of its own. Both new files are committed to the repo, so the schema and config are as open to inspection as everything else here.

last_reviewed takes its value from each file's last commit date, not from today's date. Claude proposed that and I agreed immediately, because stamping today would have asserted that someone reviewed twenty-eight files that afternoon. Nobody did. They were read, which is a different claim, and writing the stronger one into metadata would have been the exact failure this check exists to catch, committed on day one by the people installing the check.

Two files also dropped out of validation: the starter blog posts Docusaurus ships with, still sitting in my repo full of placeholder text. Docusaurus already excludes them from the build, and Claude flagged that writing descriptions for pages nobody publishes would be inventing metadata to satisfy a number. That was the right call. The new check's first effect was on the scope of what gets checked, not on any file.

The result:

28 files checked, 28 passed, 0 failed, 0 errors

Adding a tool is never one change​

The pipeline had four tools. Now it has five, and that number is written down in more places than I remembered: the tool table on the pipeline page, a second independent table in the README, the sentence listing the gates on How I built this, the page describing the Doc Detective test, and the test itself, which asserts that the pipeline page names every tool it claims to run. That test needed a fifth assertion, or it would have kept passing while checking four fifths of the truth.

llms.txt needed updating too, but not by hand. It's generated from a template plus live repo facts, which is the fix I wrote about in A baseline is not a fact. The counts can't drift there anymore. The prose around them still can, so the template needed the new gate written into it.

Then the part I didn't expect: scripts/pipeline-facts.json feeds the generated file, and it claimed my terminology rule errors on "AI agent." It didn't, and it hadn't for a while. It also covered a term the file never mentioned, LLM, which I have since dropped from the rule so I can decide that one case by case. So llms.txt had been handing AI agents a wrong description of my own linter, generated correctly from a source that was quietly false. (sigh)

I built the generator specifically so that facts couldn't drift, and it works. What it checks is that every rule file has an entry and every entry has a rule file. It compares filenames. It never opens the rule to see whether the sentence about it is true. The counts can't drift, because they're read off the filesystem every time. The descriptions can, because they're just sentences somebody typed once.

So it faithfully rendered a fact that had already drifted upstream of it. That's the more interesting failure, and it's the next thing I'm building. A substitution rule's swap list is structured data, so a script can pull the terms straight out of the rule and compare them against the terms the prose claims it flags. That only covers substitution rules. Passive.yml's description is genuinely prose, and no script is going to verify it. But it closes the specific gap that let this one through.

What this was actually for​

I should be straight about the stakes. This is a small personal site. I was fine with just titles, and I wasn't especially worried about descriptions. I ran docmeta mostly because I wanted to try it, and I'm glad I did.

A site like mine doesn't come with strict metadata requirements, but plenty of other sites do. For example, enterprise documentation sets, anything under regulatory obligations, anything where metadata decides retention or access or what turns up in an audit. Those have requirements that are written down somewhere and, more often than not, lack enforcement.

That's the gap worth naming, and it isn't that my blog posts lacked descriptions. It's that nothing in a pipeline with four other gates in it had an opinion about whether they did. An unchecked field doesn't announce itself. It sits there looking exactly like a checked one.

The requirements are also going to move. The more that what we write gets retrieved and summarized by AI agents rather than read by people, the more the definition of adequate metadata will shift, and I don't think anyone knows yet where it will settle.

So this was a proof of concept, and it took under two minutes to set up. It's a fun little tool, so I suggest you give it a spin, especially if you're working on documentation with front matter requirements.

Where this started​

This started as a pattern, not a plan. While Claude and I were updating other pipelines and tests, I kept having to remind it to also update some related file it hadn't touched.

CLAUDE.md's own rule about keeping pipeline.md honest is what that rule exists to prevent. Even so, it needed a reminder to update related files, like llms.txt and "How I built this," which also describe the pipeline.

I quickly realized that depending on me (and my fallible human brain) to remember which files were related, every time, wouldn't scale. Depending on Claude to remember on its own wouldn't either, as demonstrated by the fact that the rule in CLAUDE.md wasn't doing it.

Claude and I brainstormed a fix: a small, maintained map lists file pairs that state the same facts and could drift apart. scripts/coupling-map.json holds the pairs of related files. A new CI workflow runs on every pull request and diffs the changed files against it. It comments, only ever asking, whenever one side of a pair changed without the other.

That solved the original problem and created another problem, albeit a smaller one. The map is yet another file someone has to remember to keep current. I mean, it worked for now, but the site and pipeline will continue to expand and evolve, so the dependencies that need mapping will, too.

A reminder for the reminder​

The map needed its own periodic check. Monthly felt right, but that meant answering a harder question first: how do you automate remembering to check this? Both the human and the LLM doing the remembering are the exact failure mode being fixed.

The lowest-effort option was a reminder on my own calendar. It would probably work, if I paid attention to it. However, it would also live entirely outside the pipeline. This site exists to show a working pipeline that catches its own drift, not one that hands the hardest part back to an external calendar app.

The better option was to add something inside the pipeline, on a schedule, that reads the map and the repo. It reports what it finds the same way this site's other tests already do, by opening a GitHub issue. This would require a language model to make the actual judgment call, not a fixed script.

The cloud routine that went nowhere​

I had Claude build a scheduled routine to run that check once a month. It would read the map, check whether its entries still hold, and look for new pairs worth adding. I then ran it manually to test it before waiting for the real schedule.

Nothing happened.

I ran it again. Silence.

The tool I had for managing that routine could list it, fetch its config, and trigger a run. It couldn't show me a transcript, a log, or an error: no setting to flip, nothing hidden behind a flag, just nothing there. Two runs, zero information about why either one failed.

Switching to something debuggable​

We seemed stuck, at least with the current path. I asked whether we could use the mechanism Doc Detective's workflows already use to open issues: plain gh issue create against the repo's own token. That would sidestep the cloud routine's GitHub access problem entirely.

Claude took that further than I'd actually asked. The whole task could run as a GitHub Actions workflow, using Anthropic's own claude-code-action, not just the reporting step. No separate app dependency needed at all.

That mattered for a second reason too. A failure here now shows up as a normal Actions log, like every other workflow in this pipeline. Unlike the cloud routine, it's not a black box with nothing to debug.

Before writing the YAML, Claude checked the action's actual action.yml directly. Two different documentation sources had disagreed on whether its claude_args input took JSON or command-line flags.

Turns out, it's flags, passed straight to the Claude CLI. Trusting prose over the source would have cost a wasted run finding that out the hard way. (And because the new method required Anthropic API, each run would cost me a small amount of money, so fewer runs meant less $$.)

Three wrong guesses, one real answer​

The first real run failed with Claude execution failed: result is_error:true. Nothing else.

Claude had one real lead. The --allowedTools pattern used a colon, Bash(gh issue:*), while the CLI's own --help only ever showed that syntax with a space. It fixed the pattern and asked for a re-run.

Second run, same generic failure, but this time the full log came back. It ruled out Claude's own fix: the run lasted 409 milliseconds, one turn, zero dollars spent, zero permission checks. It had failed before a tool-permission pattern would ever matter.

The log also explained why the real error had been invisible: the action hides its own output behind a show_full_output flag, which was off by default. And it named the model it had actually used, claude-opus-5. I hadn't realized I needed to specify a model, so I never told Claude which one to use. Because I hadn't, neither did it, and the action defaulted to the most expensive option instead of the Sonnet pricing we'd already discussed. (Thanks, Anthropic!)

Claude turned on full output and pinned the model to Sonnet. Third run: the real error, finally visible. Credit balance is too low.

The API key worked. It just had no funded balance behind it, an ordinary billing gap hidden behind two layers of suppressed output until we finally saw it.

What it actually found​

Once billing was sorted, the workflow ran clean and opened its first real issue. (Yay!) The two existing map entries checked out: it re-ran the actual count commands against the live repo and confirmed both pairs still matched (ls .github/workflows | wc -l, ls .vale/styles/Diana | wc -l). It proposed four new pairs.

Two of the four were solid on inspection. .claude/commands/check.md's exact shell commands are quoted verbatim in a separate write-up page, and pipeline.md's tool table is duplicated independently in README.md. Both went straight into the map.

The other two were both about the same thing. intro.mdx and a template file still said skill.md, a name that hadn't existed since a rename weeks earlier. Claude had seen that exact name before, in a different context, and it had already decided what that meant.

The part Claude got wrong​

Weeks earlier, fixing the dead link this whole saga started with, Claude found pipeline.md's own text calling itself "skill.md and llms.txt" in a heading. It read that as intentional, a nickname the file used for itself, and kept the label when it repaired the link that pointed at it.

It never asked whether that was actually true. It inferred, and then carried the inference forward as though it were settled. I'd read that reasoning when Claude wrote it and let the phrase "intentional site branding" go by without asking what it actually meant. One clarifying question, right then, would have caught it.

It wasn't settled. My correction was one sentence: "skill.md refers to Claude skills, not to pipeline.md, and never did."

The heading itself had been the drift, sitting there since the rename. Claude's fix from weeks earlier had quietly preserved it instead of questioning it. That's the same failure the last post was about, just moved from code into Claude's own reasoning: a baseline only proves a file agrees with its own history.

An unverified assumption only proves Claude agreed with itself. Neither one checks against what's actually true, and both will hold up as long as nobody asks.

What actually changed​

Every current-state reference to "skill.md" is now "pipeline.md": the heading, the intro page, both spots in the template. The historical references stay as they were, because they're describing the old name accurately, in the past tense, as part of an incident that already happened. The map grew by two real pairs.

Does this scale?​

This site is a proof of concept for a documentation pipeline with real testing behind it, not a demo. So it's worth asking honestly: does any of this actually scale past one small repo with one contributor?

"If you touch this, also check that" is not a new problem. Larger teams deal with it constantly, across far more files than this repo has. Traditionally, that knowledge has lived in the heads of the writers and engineers who've been around long enough to remember it, not in any system.

That's fine until someone leaves, or until enough time passes that even the person who remembers stops trusting their own memory. As more of a pipeline gets automated, that knowledge needs somewhere more durable to live. The parts of the process that used to force a human to notice, someone reading every diff, are the parts getting automated away.

The monthly reviewer runs again in September. What it's actually testing, every time, is whether a small piece of that tribal knowledge can live in a file instead of a person. And whether anyone still reads the result closely enough to catch what it gets wrong.

What actually happened​

Three separate, unrelated commits landed between late May and mid-June:

• 2026-05-27 — I swapped the API reference renderer from Redoc to Scalar.
• 2026-06-04 — I renamed skill.md to pipeline.md.
• 2026-06-15 — I added two new Vale rules, Acronyms.yml and Headings.yml, bringing the total from three to five.

None of those commits touched llms.txt. Each one was a correct, complete change to the thing it was actually about: a plugin swap, a file rename, a new lint rule. None of them had any reason to know that a separate file elsewhere in the repo was quietly restating facts that had just become wrong.

By the time I noticed, llms.txt was claiming three Vale rules where there were five, and five workflows where there were eleven. It still credited Redoc for a renderer it hadn't used in over two months. It also linked to /docs/skill.md, a URL that had 404'd since the rename.

Nobody caught any of it, because nobody reads llms.txt. That's not a bug. It's the whole design of the file: it exists for an AI agent, not a person.

I'd already written a post about exactly that asymmetry. Which made it more uncomfortable, not less, that the test I built because of that asymmetry didn't catch this either.

The test that already existed​

That earlier post described a real mechanism. A script counts llms.txt's ## headings and writes the count to a baseline file. A Doc Detective test then compares the live file's heading count against that baseline on every run.

If they match, pass. If the live file has drifted from its last known-good state, fail.

It's a reasonable design, and I stand by the reasoning in that post for why I chose it over hardcoding a number. What I didn't realize at the time: "drift from its last known-good state" meant the baseline was the last version of llms.txt. Every time the count script ran, it recorded whatever the file currently said, headings and all.

So the test wasn't checking "does this file still match reality." It was checking "does this file still match itself." Those are different claims, and llms.txt can satisfy the second one perfectly forever while failing the first one from the moment the underlying facts change.

None of the three commits above added or removed a section heading. Each one changed a number in a bullet list, a link target, or a tool name in a sentence — content that lives under a heading. A test built to catch drift was, by construction, blind to the exact shape of drift that happened.

I'll say directly what I got wrong here, because it's worth naming rather than smoothing over. I was under the impression that the test compared llms.txt's heading count against the live page or pages it describes. I didn't catch this until I sat down and closely re-read the test's own write-up.

It doesn't, and never did. It compares the file to a stored copy of a previous version of itself. That's a meaningfully different claim.

The write-up wasn't ambiguous by accident. I just hadn't reread it critically enough to notice which noun "baseline" was doing a lot of work for.

Diligence here doesn't mean trusting that a test does what its name suggests. It means rereading closely enough to be sure, especially when an assistant wrote both the test and its own description of it.

Three ways to fix it​

Once I saw the actual gap, I looked at three ways to close it:

• Keep the self-comparison, widen the scope. Instead of comparing llms.txt to its own last version, compare it against the heading count summed across every page it links to. This is the option I considered first — it sounds obviously right, since a change in the pages llms.txt describes should show up somewhere. But it doesn't hold up under the same test I just described: none of the three drifts changed a heading anywhere on the site. A tool swap, a file rename, and two new lint rules all landed as prose and link edits, not new sections. Summing across more pages doesn't fix a check that's counting the wrong kind of thing. It just makes the noise louder. An unrelated blog post can shift the total and break the baseline for no real reason.

• Drop headings entirely, check the actual claims. llms.txt's riskiest content isn't its structure. It's a small number of specific facts: how many Vale rules exist, how many workflows exist, which tool renders the API reference. Those are all things I can compute directly from the repository, the same way pipeline.md already gets self-checked (ls .vale/styles/Diana | wc -l, ls .github/workflows | wc -l). I'd already written that habit into this repo's CLAUDE.md, after an external collaborator caught pipeline.md undercounting its own workflows months earlier. Comparing the live file against a freshly computed fact, rather than a stored snapshot, catches all three incidents. It's the only one of the three designs that would. That's because it doesn't depend on the file ever having been right to begin with.

• Do nothing, accept the gap. Worth naming as an option even though I didn't take it: llms.txt isn't load-bearing infrastructure. A slightly stale index file on a portfolio site is a low-stakes place to under-invest. I didn't take this option because the whole thesis of this pipeline is that a well-structured file with wrong information is worse than no file. Leaving that thesis unenforced on the one file written specifically to demonstrate it felt like the wrong place to slack off.

I went with the second option.

What actually shipped​

website/static/llms.txt is no longer edited as static text. scripts/generate-llms-txt.py renders it from a template plus whatever the repository currently contains. The Vale rule count and names come from listing .vale/styles/Diana/; the workflow count comes from listing .github/workflows/.

Content that isn't a fact you can list stays hand-written in scripts/pipeline-facts.json and the template, on purpose. That includes what each Vale rule does in plain English, which pages are worth linking, and the bio. Some parts of a file like this should keep requiring a human.

Here's the part I'm more interested in than the generation itself. The generator refuses to run if a Vale rule has no matching description in the facts file. Add a sixth rule and forget to describe it, and nothing gets silently published with five rules listed instead of six.

The build just stops, with a message telling you exactly which file needs an entry. That's the actual fix for what happened in May and June.

It doesn't rely on remembering to update a second file when the first one changes. It makes forgetting loud instead of silent.

Testing the new file against the live site turned up a second gap I hadn't gone looking for. This site already runs Lychee across every link on every page, llms.txt included. It never once flagged the dead /docs/skill.md link, across two months and roughly nine scheduled runs.

Its config resolves root-relative paths against a localhost base, so Docusaurus routes don't get treated as broken external URLs. That same trick means every root-relative link on the site, not just this one, is invisible to it by construction.

Rather than touch a site-wide config that's doing something deliberate for a reason, the new test includes a small, narrowly scoped check. It extracts every root-relative link inside llms.txt and confirms each one resolves. It's the same shape of decision as an earlier test on this site that does the same narrow-scope-on-purpose thing for a different reason.

Does this scale?​

The generator runs as its own workflow, triggered by changes to the specific files that feed it, plus a daily schedule as a backstop. It's not folded into the main site build.

Every other content-generating workflow on this site already works this way, and I kept the pattern rather than reinventing it. doodle.yml, api-reference.yml, and the rest are all decoupled from deploy.yml, wired together only through workflow_run triggers after each one succeeds.

The reason is blast radius. If generation ran as a step inside npm run build, a bug in the generator could fail the entire site's deploy. That bug could be as small as a bad template placeholder or an unhandled edge case in the fact-counting.

A typo in a file almost nobody reads shouldn't be able to take the whole site offline. Keeping it as a separate job means the worst case of a broken generator is a red badge on one page. That's exactly the size of problem it actually is.

The generator was already working when a good question stopped me: were we making this harder than it needed to be? Two existing Docusaurus plugins looked like they might already do this: docusaurus-plugin-llms and docusaurus-plugin-llms-txt. Both do the same thing: scan the /docs folder and concatenate it into a single file.

Neither has any way to know how many Vale rules exist or how many workflows exist. Those aren't docs-page content — they're facts about the repo's tooling that live in .vale/styles/Diana/ and .github/workflows/. Adopting either would have meant losing the specific claims that caused all three incidents in this post, not fixing them.

The maintenance picture made the decision easier, not harder. docusaurus-plugin-llms-txt hasn't been touched since late 2024, with four open issues sitting unanswered for months.

docusaurus-plugin-llms is actively maintained, but its own issue tracker has an open bug, filed days before I checked. Content carried in a component prop gets silently deleted during generation, with no error and no warning. The reporter's own words: the site renders fine, the generated file is well-formed, "it just has less in it than the source did."

That's the exact failure category this whole post is about, just relocated into a dependency I don't control. I can't fix someone else's plugin in the next ten minutes the way I can my own code.

That's also the honest limit of how far this scales. Every new fact llms.txt states falls into one of two categories. Some are derivable from the filesystem — the generator can absorb those, and this whole class of drift goes away.

Others require a human judgment call, like what a rule actually does in plain English, and those live in pipeline-facts.json. The generator checks that an entry exists, not that it's still accurate.

It refuses to run if a rule has no description at all. But it has no way to notice a description that's gone stale next to a rule whose behavior quietly changed underneath it. That's a real gap, just a smaller and more honest one than the file-wide version this post started with.

The fix doesn't eliminate the need to remember. It shrinks what you have to remember. Instead of "does this whole file still describe a system that keeps changing," the question is now "does this one sentence describe this one rule."

And it turns the coarser failure — adding a rule and never describing it at all — into a build failure. That's better than a silence that lasts until someone reads closely enough to notice.

The other habit worth keeping, alongside the mechanism: a test doesn't get to stay trusted forever just because it's still green. This one sat unexamined for seven weeks before I looked at it closely enough to see what it actually checked.

A baseline only tells you a file agrees with its own history. It has never once told you the file was right to begin with.

The plan that sounded right until I looked harder​

The idea came from a list I'd sketched out early on: ten-plus Doc Detective showcases, each demonstrating a different capability. One line on that list read almost exactly like this: portfolio link check, checkLink on the work samples, blog post about running two link-checking tools side by side. It sounded like a clean parallel. Lychee already crawls every link on this site on a schedule; a Doc Detective test doing the same thing on a smaller, curated set felt like an obvious complement.

Then I went to actually build it and pulled up portfolio.mdx to see what I'd be pointing checkLink at. Every work sample is a static PDF or markdown file sitting in this repo's own static/portfolio/ folder. Nothing is external link, so there's nothing that can change the way a link to someone else's site can.

Why that mattered​

Lychee's job is asking one question of every link on this site: does this URL return a status code that I'm willing to accept? It doesn't care what's behind the link, only whether a server answered. For the portfolio's work samples, that question was already being asked, on the same links, on the same schedule. Writing a checkLink test that asked it again would have meant CI time spent re-confirming something already confirmed. Not a wrong test, but a redundant one.

Redundant tests are their own kind of quiet failure. They pass reliably, add a badge to a page, and let you feel like you've covered something you haven't actually added coverage for. If I'd shipped the plain version, then I'd have a page that looked exactly as thorough as the one I ended up with.

The question Lychee can't ask​

A status code doesn't tell you what's actually behind a link. A PDF that got corrupted mid-upload, a file that got accidentally overwritten with zero bytes, a misconfigured redirect that serves a generic error page at a real 200, all of those pass Lychee's check without complaint. The response came back. The number was fine. Lychee has no way to open the file it just confirmed exists.

That gap is the one worth testing, so that's what the portfolio link check actually does. It still runs checkLink against each work sample, six of them, the same claim Lychee makes. Then it goes one step further: a shell script fetches each file and checks whether it's real. For the PDFs, that means reading the first four bytes and confirming they spell %PDF, plus a minimum file size well below what any of the actual documents weigh in at, so a truncated or emptied file can't sneak through. For the CockroachDB tutorial, it's a markdown heading and its own size floor. None of that is possible from a status code alone.

What "two tools, different jobs" actually looks like​

I'd been picturing two tools running the same check on different scopes; Lychee everywhere, Doc Detective on a smaller set. What I built instead is two tools running different checks on the same links: Lychee asks whether the door opens, Doc Detective checks whether there's a room behind it. The overlap in scope, both tools do touch these six files, is fine. It's the overlap in question that would have been the problem.

The test that's live now doesn't prove more coverage exists on this site than actually does. It proves one specific thing an existing tool structurally can't: that a 200 response and a real, intact document are not the same fact, and a portfolio page is exactly the kind of place where nobody notices when they quietly stop being the same fact.

The deterministic layer​

The first thing I added to this pipeline were linters: Vale, codespell, Lychee. All three run on every pull request. Same input, same output, every time. That's what deterministic means here. They catch style violations, misspellings, and dead links, at the last possible moment, right before something merges. I don't think about them day to day. They just run.

The non-deterministic (AI) layer​

CLAUDE.md rules work alongside the linters, but they enforce correctness a different way. A linter matches a pattern. CLAUDE.md gives Claude Code a judgment call to make, and judgment calls don't come back identical every time the way a regex match does.

Right now, there are two rules in CLAUDE.md:

• When I change anything the pipeline overview page describes, CLAUDE.md tells Claude Code to check that pipeline.md, How I built this, and this repo's README still describe it accurately, and fix whichever one drifted.
• When I add a new Doc Detective test page, CLAUDE.md tells Claude Code to match the heading structure the other pages in that section already use, rather than inventing a new shape.

Both rules exist because I got burned once. An external reader (not Claude) found the pipeline page undercounting its own workflows and Vale rules. Writing the second rule, I found a live example of the same kind of drift myself: one test page was missing its own title heading, and nobody had noticed. No linter catches either of those. They're not typos. They're a claim quietly going stale.

The efficiency layer​

The skills don't enforce anything. They save time. Three commands, three chores I used to do by hand:

• /new-post scaffolds a blog post with the slug, title, and frontmatter already filled in.
• /check runs my linters locally, so I see what the PR check would catch before I open the PR, not after.
• /start boots my local dev server, and checks first so I don't end up with two running by accident.

None of these skills are impressive engineering. They just remove friction.

What I don't know yet​

Here's the honest gap: I don't know whether the skills or the CLAUDE.md rules save me tokens, as opposed to time. A skill runs a fixed set of steps. A CLAUDE.md rule asks Claude Code to notice something and decide what to do about it, which is a different kind of expensive. I haven't measured either one, and I wouldn't bet confidently on the answer in either direction.

What's saving you time in your own process right now? A skill, an agent rule, something else? I'd like to know.

Picking a target that actually changes​

I didn't want to reinvent the wheel, so I built on two existing image tests:

• the API reference test and
• the doodle test

Claude Code suggested pairing the API reference test with the responsive screenshot test instead. That probably would have worked, but I wanted to isolate the doodle specifically. My API sample won't change very often, and never without my actions. Google's doodle definitely will.

That distinction turned out to matter more than I expected.

The bug Claude Code didn't notice​

As we worked, Claude Code kept talking about the doodle test as if it changed every day. It hadn't noticed that the test wasn't actually updating on days Google wasn't running an animated doodle.

I suspected — and it confirmed — why: the old test cropped the screenshot to a specific element, #hplogo, before saving it. That element only exists on days Google is running a special doodle. On an ordinary day, the crop target isn't there, the step fails, and the run produces nothing. No red badge, no error thrown. Just silent, invisible failure.

Once I pointed that out, Claude Code checked the dates on the doodle images already committed to the repo and started investigating from there. The evidence was sitting in plain sight: multi-day gaps in the commit history, days at a time with no update, because the crop target simply wasn't there to find. Claude Code's live inspection of Google's homepage confirmed it: no #hplogo, and no fallback either. #lga, an older, once-standard Google logo-container ID, wasn't there. Just obfuscated, build-specific class names, no stable selector of any kind on a plain-logo day.

Claude Code's first suggestion was to stop cropping and just capture the full page. I countered: could we crop it after the fact instead, or use a narrower viewport? I didn't want my Google account photo or bookmarks bar showing up in a screenshot on a public site.

Claude Code checked, and it turned out not to matter — CI runs in a fresh, unauthenticated browser session, so there's no account or bookmarks to leak in the first place. We went with the full page, uncropped, at a fixed viewport instead of a specific element, so it doesn't depend on anything existing that might not be there.

That's a real shortcoming in AI: it only notices what you tell it to. It means we still need to be the observant ones, watching for things that aren't behaving as expected, and the imaginative ones, describing the edge cases worth checking in the first place.

What the test actually does now​

With the crop gone, the test needed a real comparison mechanism, which is the actual point of Test #5: Doc Detective's maxVariation option.

A screenshot step can take maxVariation, a threshold for how much of the image (0 to 1, as a fraction of pixels) is allowed to differ from the image already saved at that path, and overwrite, which decides what happens to that saved image. There are three modes, and they're less alike than they sound:

• "true" replaces the file every run and never compares anything.
• "false" captures once, then skips every later run outright, no comparison at all.
• "aboveVariation" is the only mode that compares every run: it diffs the new capture against the saved one, and only replaces the saved image if the difference crosses the threshold.

That third mode is what both tests use now. The doodle screenshot compares against its reference image at a 10% threshold. Most days, nothing crosses it. On doodle days, it does, and the reference image updates. The API reference screenshot got the same treatment at a tighter 2% threshold, since that page should barely ever change — a comparison flag there is a much stronger signal.

One more detail worth knowing if you use this yourself: exceeding maxVariation doesn't fail the test. It reports WARNING. That's a deliberate severity choice in Doc Detective, not a gap in my setup — "this looks different" is information, not automatically breakage.

It didn't occur to Claude Code that a WARNING which can't redden a badge is also a WARNING nobody would ever see. I had to ask directly: if this doesn't fail and doesn't show up anywhere, how would I actually find out it happened? Once I asked, the fix was easy — both workflows now open, or comment on, a GitHub issue when a comparison exceeds its threshold. But the question had to come from me first.

The investigation found a second bug​

Digging into why the doodle test was failing silently turned up something else: Doc Detective's CLI exits 0 no matter what happens in the test. Claude Code confirmed it directly, running a test it knew would fail on a bad URL and getting a clean exit code back anyway. There's no flag in the current version to change that.

That mattered here because every workflow on this site checks for failure with a pattern like || echo "failed=true", which depends on a non-zero exit code that, it turns out, never comes. I had Claude Code rewrite the workflows this test touches to read Doc Detective's own JSON results file directly instead of trusting the exit code, and it flagged the same fix as still needed for the rest of the site's workflows, since they're all quietly making the same assumption.

That was a fun find. I don't usually get to be this hands-on with testing. I filed it upstream as doc-detective/doc-detective#674.

Writing the page took more rounds than the test did​

When it came time to write up the test page itself, Claude Code dutifully drafted everything it had observed while we worked, the crop bug, the exit-code bug, the whole investigation, right there on the page.

Except no. Like an overeager intern, it got lost in the details. Every test page on this site follows the same shape: why it matters, the latest result, how it works, what makes it fail, how to run it yourself. That consistency is an important convention in technical documentation. A migration story about a bug that no longer exists doesn't belong in a page explaining what the test currently does — it belongs here, in a post, where the history is the content instead of noise on top of it.

It took a few more rounds of edits before the page said only what it needed to say.

Again: human input. AI-assisted workflows are genuinely useful, but they're fragile, and they still need someone deciding what belongs on the page and what belongs in the commit history.

Is this slop?​

Which brings me to this post. I'm skeptical of the "60% of long-form LinkedIn content is AI slop" stat that's been going around, mostly because the so-called AI detectors behind it have an atrociously high false-positive rate. But it got me wondering: does this post count as slop?

I wrote it with the help of an LLM. The test I'm writing about was built with one too, but the ideas are mine. This was collaborative, and I was in the driver's seat the whole time. Would this article add to the AI slop stat? What if I threw in a typo or twoe? Maybe a double space?

Either way, I'll keep building these Doc Detective tests as long as I keep coming up with testing ideas. This one is #5 on a list of 11 — so far.

I'm open to suggestions if you have any ideas of what you want to see tested, network!

Really, now. I do not expect that kind of recalcitrance from my apps.

Why this test is different​

The pipeline test and the API reference test each check a page at one fixed viewport. Neither would notice if a layout change broke the mobile experience while leaving desktop untouched, or the other way around. That's a real gap: the markup and the content can both be fine, and the page can still fall apart at a width nobody happened to be looking at.

The fix seemed straightforward: define two tests on one page, identical steps, different runOn viewport. Navigate to the homepage, take screenshots: one at 375×812, one at 1280×800.

The crash​

The first time I ran it, Doc Detective printed its usual parsed-test summary, both tests looking exactly as I'd written them, and then:

Cannot read properties of undefined (reading 'forEach')

No stack trace pointing at my file, no line number, nothing. I assumed I'd gotten the JSON wrong somewhere, so I stripped the test down to the smallest possible case: one runOn block, one browser, no viewport at all. Same crash. I removed runOn entirely and the test ran fine, screenshot and all.

So the crash wasn't about viewport, or about my specific config. It was runOn itself.

I read the installed package's source rather than guess further. The context-resolution code loops over context.platforms unconditionally:

context.platforms.forEach((platform) => {
  ...
});

The schema marks platforms as optional. Nothing defaults it to an array when it's missing. Any runOn context that specifies only browsers, which is the natural thing to write when you don't care what OS the test runs on, hits undefined.forEach and dies. This is a genuine bug in the installed version (doc-detective 4.9.0), not a mistake in my markup.

The workaround was to always list platforms explicitly, even though the schema doesn't require it:

{"platforms": ["linux", "mac", "windows"], "browsers": [{"name": "chrome", "viewport": {"width": 375, "height": 812}}]}

With that in place, both tests parsed, resolved, and ran.

The width that wasn't​

Both screenshots saved. Both looked like real, distinct layouts. One had a collapsed hamburger nav, one had a full top bar. I checked the file dimensions to confirm before moving on, and the desktop one read exactly 1280×800, as requested. The mobile one read 500×812.

I'd asked for 375. I tried narrower values, 320, even smaller, and kept landing on exactly 500. I tried a wider value, 550, and got exactly what I asked for. That bracketed it precisely: something in my local browser automation setup enforces a 500-pixel-wide floor, silently, regardless of what viewport a test requests. Anything narrower gets rounded up. Anything at or above passes through untouched.

Nothing failed. Nothing logged a warning. The test reported success, because by its own definition, success just means the browser reached the page and a screenshot got saved. It never claimed to guarantee the exact width.

What the page says now​

I could have picked a viewport value this setup happens to honor and called it "mobile." That would have made the numbers line up and told a smaller truth. Instead, the responsive screenshot test still requests 375×812, because that's a real, common mobile breakpoint and the honest target, and it says plainly, right above the screenshots, that requested and rendered aren't always identical, and that the images are the ground truth, not the caption.

I only caught the gap because I checked the file dimensions instead of trusting the green result. The API reference test passed on a similar assumption a few days ago, and that one turned out fine. This one didn't. The only way to tell the difference in advance is to look.

The responsive screenshot test page shows the result and updates it every day:

• A pass/fail badge with the timestamp of the last run
• A screenshot of the mobile-width homepage
• A screenshot of the desktop-width homepage

A different kind of fragile​

The pipeline test checks static content, markdown that Docusaurus renders once at build time. The API reference test checks something that isn't part of this site's build output at all. Every time someone loads the API reference, Scalar fetches a script from a CDN, fetches an OpenAPI spec, and renders the whole reference client-side, on the spot. Any of those steps can break independently of anything in this repo. A static content check would never catch that.

My first draft's plan was to confirm the page loaded, then screenshot it. That was also my first mistake, because I hadn't yet worked out what "confirm it loaded" needed to mean.

The bug that ate an afternoon​

My first version of the test navigated to the page, then used Doc Detective's find step to check that the text "Garden companion API" appeared somewhere in the body, before taking the screenshot. It failed. Every time. Even at a 90-second timeout, which ruled out simple slowness.

I spent a long stretch assuming the problem was the browser, or the CDN, or a shadow DOM boundary hiding the text from a plain query. None of that held up. A screenshot taken a few seconds after page load, with no assertion at all, showed the heading rendered clearly, right where it should be. The content was there. The find step just couldn't see it.

The actual cause was narrower and stranger: matching against live text on a page that Scalar keeps re-rendering seems to break whatever polling mechanism Doc Detective uses to retry the check. The fix wasn't a longer timeout, it was a smaller claim. Instead of asking "does this page contain the text 'Garden companion API,'" I asked "does an element matching h1.section-header-label exist." Same page, same moment, a check on structure instead of content, and it passed immediately.

I don't fully know why the text-matching path fails on this specific page. I know the selector-based path doesn't, and I know which one to reach for next time a client-rendered page needs a check like this.

The second bug was in my own writing​

Once the render check passed, a different set of steps started failing, ones I hadn't written on purpose. Doc Detective doesn't just run the steps you write; it also scans the surrounding markdown for patterns that look like instructions, and turns them into steps automatically. My "run this test manually" section told a human reader to go to the Actions tab and click Run workflow. Doc Detective read that bold text as literal find and click targets, and tried to execute them against a page that had never heard of a "Run workflow" button.

The instructions for a human running the test by hand had become test steps of their own, silently, because of formatting I'd chosen for readability. The fix was to wrap that section in an explicit ignore marker, telling Doc Detective not to scan it. I later found the same bug sitting untouched in an earlier test on this site, unnoticed because its real steps had been failing for an unrelated reason and never got far enough to trigger it.

A test that always passes is worthless​

Once I'd fixed both of those, the test ran cleanly, which is when I got asked the question that mattered: how does the test know the screenshot is valid?

It doesn't, not in any deep sense. The find check proves that something with the right shape rendered before the picture was taken. It does not check that the picture shows the right content, and Doc Detective saves the screenshot unconditionally, with no comparison against the last one. A visual regression that still produces the same heading element, broken styling, wrong body text, a layout that fell apart, would sail through untouched.

That's not a bug I fixed. It's a boundary I hadn't stated. The API reference test page now spells it out directly: the test fails only if Scalar's render pipeline produces nothing at all, not if it produces something wrong. Catching the second case is a different kind of test, one that diffs a new screenshot against a trusted baseline instead of just checking that a screenshot was possible. That's a real gap, and pretending otherwise would have made this test look more capable than it is.

The environment lied more convincingly than the test did​

The last wrinkle had nothing to do with Doc Detective. My local sandbox couldn't run the finished test reliably, failing on browser automation bootstrap issues that had nothing to do with the test logic, three different infrastructure errors across three attempts, no zombie processes, no obvious cause. I'd already proven the underlying check worked in isolation, more than once, but I couldn't get a single clean end-to-end run locally to point to as proof.

The real proof came from GitHub Actions, which ran the finished workflow on a clean runner and passed on the first try, with a genuine screenshot and a status file to show for it. The lesson wasn't about Doc Detective. It was a reminder that a flaky local environment and a flaky test look identical from the inside, and the only way to tell them apart is a clean environment you trust.

What this test is actually for​

This is the narrowest test in the series so far, and I mean that as a description, not a complaint. It proves that a third-party render pipeline, one entirely outside this repo's control, still produces output. It doesn't prove the output is correct, and now it says so, on the page, next to the badge.

A test that claims more than it checks is worse than no test at all, because it teaches you to stop looking. This one only promises what it can back up.

Update (2026-08-04): The baseline-comparison test described in this post has since been retired and rebuilt. It compared llms.txt against a stored copy of its own previous version, which turned out not to catch the kind of drift that actually happened. For a full breakdown of what changed and why. see A baseline is not a fact.

llms.txt is the emerging convention for giving AI tools a structured, machine-readable map of a site's content. The idea is the same as robots.txt, but aimed at language models and AI-assisted search rather than web crawlers. The file describes what the site covers, where the key pages are, and who made it, in plain markdown that a retrieval pipeline or AI agent can fetch and pass along as context without rendering a browser.

Nobody clicks it. Nobody subscribes to it. If it disappears, or drifts out of sync with the site it's supposed to describe, no human notices. An AI agent might start returning wrong information about the site, and nothing in a normal deployment pipeline would catch it, because nothing in a normal deployment pipeline is watching a static file that no human is reading.

That's the problem I wanted to test.

What to assert​

I worked with Claude Code to draft Doc Detective tests. Claude's first suggestion was to assert on content: check that the file names a specific tool, or includes a particular heading, or contains the author's name.

I pushed back. Most documentation doesn't have a named author in the machine-readable index. And if you're maintaining llms.txt files across ten products, content assertions require hand-editing every time something changes — which means they quietly go stale, which is the exact problem you were trying to prevent.

The more useful assertion is structural. The llms.txt spec requires an H1 title and at least one ## section heading. Those are format requirements that hold across any site's implementation. A 200 response with no section headings is either a placeholder or a corrupted file, and you want to know either way.

The tool doesn't decide what's worth testing. That part is still a human judgment call.

Going further without hardcoding​

Structural conformance catches a broken or missing file. It doesn't catch drift: a file that was valid six months ago and is still technically valid now, but no longer reflects the site's current shape because sections were added or removed.

The obvious fix is to count headings and assert on the count. The problem is that hardcoding a number is just a more subtle version of the same mistake. It starts failing the moment the structure changes intentionally, and whoever runs the test has to remember to update the number, which is manual maintenance that someone will eventually skip.

A baseline comparison works better. A script runs on every push that touches llms.txt, counts the headings by level, and writes the result to a JSON file in the repo. A Doc Detective test then fetches the live file, counts the headings, and compares against the stored baseline. If they match, the test passes. If the file has drifted from its last known-good state, the test fails.

When you intentionally add or remove a section, you update llms.txt, the count script runs, the baseline updates, and the test passes again. The test only fails when the live file diverges from what was last committed. The count is never hardcoded anywhere.

Making it visible​

The llms.txt verification test page shows the live heading counts alongside the pass/fail badge. That's the same pattern as the pipeline test: don't leave the result buried in a GitHub Actions log where nobody looks until something breaks loudly. Put it next to the thing it's checking.

The heading counts also make the state of the file visible at a glance, independent of whether the test is passing. If the counts look wrong to a human reader, that's useful information regardless of what the automated assertion says.

The testing surface nobody talks about​

Most documentation testing is aimed at human readers: page loads, links resolve, images render. Those are the things people notice when they go wrong.

llms.txt has a different audience. It's read by retrieval pipelines and AI agents, not people — whatever sits in front of a chatbot that someone has pointed at your documentation. That audience won't open an issue when it gets a 404 instead of a site description. It just uses whatever it got.

Testing for that audience means thinking about what "correct" looks like from the pipeline's side, not the reader's. A test that checks whether llms.txt is reachable, structurally valid, and still has the right shape is a test written for someone who will never tell you when it's broken.

Not "does this page render well," but "is this file still describing something true, to a reader who won't complain if it isn't."

This site has a page that describes its own pipeline: it involves Vale, codespell, Lychee, and Doc Detective, each doing a specific job. I wanted to know if that description would still be accurate next month, not just whether it read well today. So I wrote a test for it.

Treating a description as a claim​

The pipeline overview page has a clean table naming each tool and what it does. Until this week, nothing checked whether that table stayed accurate. If I swapped Lychee for a different link checker and forgot to update the page, then nothing would tell me. The page would keep naming a tool that wasn't there anymore, misleading the reader.

So I wrote a Doc Detective test that treats the page as a claim instead of a description: load the page, confirm the heading is right, and confirm the body still names all four tools. If any of that stops being true, then the test fails.

What happens when the requirement changes​

If I add a fifth tool to the pipeline tomorrow, or swap Lychee for something else, the test doesn't update itself. The page changes, the test does not, and the test fails. That's not a flaw, that's the design. The failure is a forced checkpoint: a human has to look at the diff between what changed and what the test still expects, and decide whether to update the test to match the new reality.

This is the same shape as a requirements test in any other system. Nothing automatically keeps a test in sync with intent, because the test's whole job is to represent intent as it was last confirmed. If it auto-updated whenever the implementation changed, then it would stop being a test and start being a mirror, agreeing with whatever it's pointed at.

The tradeoff shows up at scale. Four strings hand-typed into a JSON file works fine for one table with four rows. It doesn't work for a documentation set with fifty pages each making a handful of claims, because now the claims live in two places, the page and the test, and keeping them in sync is manual labor that someone eventually skips.

How to fix it​

The fix at that scale is to stop hand-typing assertions and generate them from the documentation itself. Doc Detective already has a feature for exactly this, called detected tests. You define markup patterns in your config, things like "bold text generates a find action" or "a Markdown link generates a checkLink action," set detectSteps to true, and Doc Detective scans your docs and regenerates the test steps from the content on every run. The table on the page becomes the source of truth, not a JSON file sitting next to it.

Here's where it gets specific to this repo: the table on the rendered pipeline page isn't the only copy. The repo's README.md has its own version of the same table, written for anyone reading the code instead of the deployed site. If the README is the table I actually trust, the one a person edits first when a tool changes, then that's where detection should point. Mark up the tool names there as bold text, turn on detection, and the test would track the README automatically. Add a fifth tool, and the test picks it up the next time it runs, no edit to sample.json required.

But Doc Detective loads the rendered pipeline.md page, not the README. So detection only solves half the problem. I'd also need something that checks the pipeline page's table against the README's table, so the two don't quietly drift from each other the same way the page and the old hand-typed test could. Detection would fix the test-versus-page gap. It would not fix the README-versus-page gap, that's a second, separate sync problem, and it's the same shape as the first one: two descriptions of one fact, with nothing forcing them to agree.

I didn't build any of that here, because one table with four rows doesn't need it yet, and sample.json is currently the more honest reflection of how this specific test works for a reader trying to follow along. But it's the real answer to "how does this scale": not a generator I'd have to build, but a feature Doc Detective already ships, plus a second check I'd still need to write, because automating one sync point doesn't automatically close the others.

At that point, I'd also want something downstream of detection that opens a pull request when a detected test changes shape, so a tool swap shows up as a reviewable diff instead of a silent regeneration. That's out of scope for this test, but it's the natural next step.

Making the result visible​

The doodle test I wrote earlier updates two images on a schedule. It's a proof of concept for content automation: let the test keep something current so a person doesn't have to.

This test is a proof of concept for something else. It's not about keeping content current, it's about catching the moment content stops being current. The two are different problems, and conflating them is how a lot of stale documentation happens.

I also didn't want the result just sitting invisible inside a GitHub Actions log. So the workflow now writes a small status file with a pass/fail and a timestamp, and a new page reads it and shows a badge. Green or red, right there, next to an explanation of what's being checked and why.

CI results that only live in CI tend to get ignored until something breaks loudly. Putting the result on the page it's protecting makes the check part of the documentation, not a separate system nobody looks at.

The actual lesson​

Style checkers and spell checkers verify that text is well-formed. Link checkers verify that references resolve. None of that tells you whether a sentence is still describing reality.

That's a different kind of test, and it's worth treating as its own category: not "does this page read well," but "does this page still tell the truth." The pipeline page now runs that check on itself, every day.

And if it is lying, then it says so right on the page as well as in an email to the overseer of its repo (me).

My first Doc Detective test for this site was like that. And honestly, I am glad. If it had gone smoothly, then I wouldn't have learned nearly as much.

My concept was pretty simple: Create a test that shows how Doc Detective can update screenshots.

It seemed like the Google doodle would be an obvious and easy use case.

• It's public.
• It updates every day.
• Everyone knows about it.

So I wrote step-by-step instructions, took a screenshot and downloaded the doodle GIF, and then told my robot minion (Claude Code) to help me write the tests.

Mistakes​

First mistake: I did not tell it to use the most recent version of Doc Detective. I'd heard this could be a problem with coding assistants; that if you do not explicitly instruct them to use the latest, then they might not. In this case, it didn't.

I realized the issue when I ran Doc Detective on the tests locally and it failed. So I had the minion update the tests, tested again, and was successful.

Second mistake: I did not understand who I was writing instructions for. I wrote instructions for humans. The testing agent is not a human.

I had instructions like, "Navigate to google.com" and "Observe the doodle," as one would for a human. That doesn't work for the bot.

If I hadn't had Manny Silva demo the functionality for me a while back, then I wouldn't have thought to ask Claude Code why it was writing tests in a separate file rather than inline, as I had seen in the demo.

Turns out that when you write for a human, the robot has trouble parsing instructions. They just aren't specific enough for it to use the human-readable steps as a basis for tests. As a result, the tests were written to a separate file.

And that, for the record, is totally valid. For this use case, I wanted to try the in-line tests. Claude Code helped me rewrite the instructions so that the testing agent would understand them, which meant we didn't need a separate testing file.

Workflow updates​

After all of that, I ran my regular workflow. Vale, bless its little heart, dutifully complained that I hadn't spelled out my acronyms: GIF and PNG.

So since I don't want it to bug me every single time I use those acronyms, I had Claude update Vale to not apply the rule to GIF, PDF, PNG, or JPG.

And then it turned out I somehow hadn't installed Vale or codespell locally, only on the remote repository, so I had to install those and told Claude to make sure the local and remote Vale configs stayed the same.

Once all of that was finished and merged...it didn't work. So I began to troubleshoot.

Troubleshooting and fixing​

Here's a partial list of things that went wrong before it worked:

• The schema I was writing tests against was two major versions out of date
• A sharp/libvips dependency was silently missing from global npm installs in CI, causing cryptic ERR_DLOPEN_FAILED errors across five separate debugging sessions
• The GITHUB_TOKEN that GitHub Actions uses can push commits, but those commits deliberately cannot trigger other workflows — a loop-prevention feature I had never had to think about before
• The workflow was running at midnight Pacific time, before Google had actually updated the doodle for the day
• Two workflow runs fired at the same time and one of them rejected the other's push

Each one of these felt, in the moment, like a dead end.

What eventually worked: upgrading to the latest version of Doc Detective (which bundled its own sharp), adding a workflow_run trigger so the deploy fires after the doodle commit, and moving the schedule to 2 AM — which Google has reliably beaten.

Dominos​

How many dominos did I tip over in this workflow?

Let's see...

• Told Claude Code to always use the most recent version of libraries and tools.
• Installed a couple tools that I missed installing locally.
• Updated Vale configs. Again.

Mind you, this workflow is still pretty immature. As I continue to iterate and improve the site, I expect that there will either be fewer workflow dominos that fall over with each new change, or at least, they will be new and different ones.

The bigger lesson? Automation doesn't save you from debugging. It just means the debugging happens at scale, in CI logs, instead of on your own machine.

But when it works? The page updates itself while I sleep. That still feels like magic.

I've been making updates around the site, as one does, and realized that Vale wasn't running on my home page or portfolio site. I inquired of Claude Code as to why, and it said that those pages were .tsx files, which Vale ignores.

Mind you, I hadn't asked it to make them .tsx files. It just did it. So I asked if there was any reason they shouldn't be .mdx files. It said there wasn't, so I had it change them to .mdx. That way Vale would review them.

No biggie, right?

Well, today I checked the home page again, and lo and behold, there was a new sidebar on it. Grr.

Turns out that when Claude Code changed the home page to .mdx, Docusaurus did its thing and added a sidebar.

Now, a human knows at a glance that is wrong. My robot minion, of course, did not.

So I had it change the home page back to .tsx and all was well.

The moral of this story, and the point of the exercise of building and using an AI-assisted workflow, is to find these stumbling blocks and identify them.

Right now, this POC site is about the smallest use case you could realistically imagine for a production system. It's dead simple: a portfolio site plus a blog. Small things compound — and the earlier you find them, the easier (and cheaper) they are to fix.

Now take this site and multiply it exponentially, pages and problems, to extrapolate what an enterprise production system will look like. How much planning, training, and supervision will that take?

Scary, no?

This is why we practice. This is why we test. This is why we iterate.

Like many of us, I have worked in Git for years. YEARS.

I've managed GitHub teams, reviews, branches, fixed merge conflicts, and so many other tasks.

But none of those repos were mine.

Until now.

As a writer working for a company, I always had someone else to handle all that setup stuff. I told the engineers or ops team what I needed, and they made it happen. Automagically.

But this repo is, while not the first repository I have ever owned, definitely the first that I have seriously managed.

Now I am the one who gets the notifications that a workflow has failed and need to deal with them. I don't have a team to forward them to, it's just me and my robot minion (Claude Code).

Honestly, it's empowering in many ways. Instead of Kermit flailing helplessly at the error messages, I can work with the assistant to figure out where the workflow is broken and how to fix it.

If you haven't done something like this before, it's an experience I highly recommend.

For example, when I merged my previous blog post, my workflow kicked off with lychee, a link checker, throwing a bunch of errors. I had to track down the right logs, feed them into Claude Code, and work through the fix with it.

Every problem that I encounter is a learning opportunity, and I love it.

I mean, it's been online for a while, but I feel like this blog post is what makes it complete.

You see, this post is what marks the beginning of the workflow testing.

Let me explain.

A while ago, I realized that we documentarians need a way to demonstrate what we do, to really show our value. Unfortunately, a lot of our best stuff is frequently in company hands and hidden behind layers of NDAs. My solution?

Build it ourselves.

Hence, this showcase site was born. It's a proof of concept (POC) site that not only has the information about me that a hiring manager might want (resume, samples, and so on), it also has a GitHub repo showing a semi-automated documentation workflow.

GitHub actions, Doc Detective, agent skills, Vale, and so on. They're all here, and they will run on every page that I write and merge. This will allow me to test and improve the workflow over time.

And because the repo is public, potential employers can see the backend as well as the frontend.

Future blog posts will discuss more about the process and lessons I learn. But for now...launch!