Screenshot comparison test

This page explains maxVariation, the Doc Detective option that compares a new screenshot against a saved reference image, and shows the live results from the two tests it's wired into.

Why this matters​

Every other screenshot test on this site captures an image and either shows it unconditionally or checks that something rendered. None of them check whether today's capture looks like the one already saved. maxVariation closes that gap, applied here to two screenshots with very different expectations: the API reference screenshot test's screenshot, which should almost never change, and the doodle test's screenshot, which changes on purpose whenever Google ships a new doodle. Comparing both shows what a pixel diff actually measures — a real regression signal on a stable page, and a genuinely fluctuating number on a page that's supposed to change.

Both comparisons use the same mechanism: a reference image that only gets replaced when a new capture differs from it by more than the threshold (see How the test works). It's not a fixed baseline pinned forever, and it's not strictly "yesterday's" capture either — it's whatever was last saved, which could be from the last run or from weeks ago if nothing has crossed the threshold since.

Latest result​

API reference — a rendered API spec, not day-to-day content. Its reference image should almost never need to change, at a 2% threshold, so a WARNING here is a real signal that something in the render shifted.

Doodle — content that's supposed to change, at a 10% threshold. Most days the reference image stays untouched because the plain logo hasn't moved; on days Google ships a new doodle, the difference is large enough to replace it. Neither outcome is a failure.

How the test works​

This isn't a standalone test with its own steps — it's the same screenshot step option, maxVariation, applied to a step already defined on the API reference screenshot test and doodle test pages. A Doc Detective screenshot step can take:

• maxVariation — the fraction of pixels (0 to 1) allowed to differ between a new capture and the image already saved at that path, before the comparison is flagged. Defaults to 0.05 (5%).
• overwrite — what to do with the existing file. There are three modes, and they behave less alike than the names suggest:
  • "true" replaces the file unconditionally and runs no comparison at all. This is what the API reference page's main display screenshot still does.
  • "false" captures once. On every later run, if a file already exists at that path, the step is skipped outright — no new capture, no comparison. It freezes a reference image; it doesn't repeatedly check anything against it.
  • "aboveVariation" is the only mode that actually compares on every run: it captures, diffs against the existing file, and replaces the file only if the difference exceeds maxVariation. This is what both comparisons on this page use.

What makes this test fail​

Neither comparison can turn a badge red. Exceeding maxVariation reports WARNING, not FAIL — a deliberate severity choice in Doc Detective, not a gap in this site's setup. "This looks different" is information, not necessarily breakage, and the status badges on this site treat WARNING the same as PASS.

That's deliberate, but it created a gap: nothing surfaced a WARNING anywhere a human would actually see it. Both workflows now open (or comment on, if one's already open) a GitHub issue when a comparison exceeds its threshold, so exceeding maxVariation is quiet on the badge but not silent — it still reaches someone.

Run the test manually​

There's no separate test file for this page. See the API reference screenshot test and the doodle test for how to run each comparison locally.