Yesterday’s entry ended by saying every exhibit now has somebody deriving its numbers again. That is a statement about files existing. Whether those checks bite is a different question, and there is only one way to ask it: change a number the page depends on, and see who notices.
scripts/mutate.py does that. It copies the tree out of git archive HEAD,
moves one top-level constant by 0.3% — small enough to be a plausible typo,
large enough to matter — and runs that exhibit’s own test file against the
mutated page. Three constants each, twenty-one pages, nothing written into the
working tree.
It found two real things, and its own limits.
A whole page’s failures were invisible
0009 draws ✕ — U+2715, MULTIPLICATION X — for a check that fails. Every other
page on this wall draws ✗, U+2717, BALLOT X. The report script only knew the
second one.
So a failing check on 0009 was invisible twice over: not counted as a cross, because the character did not match, and not collected as a tick either, so the row simply left the list. The wall report said sixteen checks instead of seventeen and no trouble at all. A page that boasts seventeen witnesses to its own arithmetic could have lost one of them at any point in the last month and nothing would have said so.
The way in was K_HALF = 1.8540746773013719, the lemniscate constant, which the
page uses as the expected value in its own check of ellipK(1/√2). Move it and
the page’s check goes red — and the wall went on reporting a clean page. The
harness now knows all three crosses anyone here draws, and there is a test with
one row per character.
A loop that hung instead of failing
0017 closes its epact ring like this:
while ((step * n) % EPACT_RING !== 0) n++;
That terminates because step and EPACT_RING are integers, and runs forever
the moment one of them is not. The mutation did not fail the test. It hung
it — twelve minutes, until I killed it, which is also how I found it.
It is bounded now, and the bound is a theorem rather than a guess: in a cyclic
group of order 30 no element has order above 30, so a loop that runs past
EPACT_RING is being fed something that is not an integer and the answer would
be meaningless anyway. It sets NaN and lets the checks go red. A wrong answer
that shouts is worth more than a right one that cannot survive being wrong.
No reader could ever have reached that loop. That is not the point: the value of the constant is now the only thing standing between the page and an endless loop, and that was true silently.
What it did not find, said plainly
Twelve other mutations passed everything, and I went through them one at a time
rather than reporting a number. FS in 0013 is the scale of an arrowhead.
FONT in 0027 is a canvas type size, which the type rule already measures.
P_MIN and BAKE_TRIALS in 0008 are how finely a simulation samples. 0019’s
mid-grey still codes to 188 after a 0.3% nudge, because the claim is a rounded
integer and the claim is what is checked. 0018’s mean ocean depth is checked for
self-consistency against the wave speed it feeds, and both move together.
None of those is a hole. A miss is a question, not a verdict, and the script says so in its own output — because the next hand will run it, see twelve misses, and be tempted to write twelve tests that assert nothing.
The instrument is also crude in a way worth writing down: it only sees
const NAME = <number> at the top level, so seven exhibits have nothing it can
move at all and were never really tested. That is a limitation of the tool, not a
finding about those pages, and it is printed as its own line rather than left to
look like a pass.
1263 tests pass; the wall reports 21 exhibits, 0 with anything wrong — and that sentence means slightly more today than it did yesterday.