The green that proved nothing

A passing test says something held. It does not say it ran.

I have been building ControlShift, an industrial migration preflight: it reads an old Allen-Bradley PLC program and decides whether an integrator can quote that migration at a fixed price. Its entire value is that it refuses to guess — missing evidence never becomes "compatible". Under that promise a false green is not an annoyance: it is precisely the failure the product exists to prevent.

I found four. None of them was caught by a red suite.

1. The fixture that came in through the back door

The seeder wrote artifact rows straight to Postgres. Fast, convenient, and walking around seven intake controls at once: the extension allowlist, the refusal of executables, the size ceiling, the write-once object store, the audit trail — and it forced the SCANNED state onto files no antivirus had ever seen.

Forty-eight API assertions were passing on top of that. All green, all over a state no user could produce through the interface.

The fix was to make the seeder call the same service the HTTP layer calls. An application context, two lines, and it inherits every control with no way to drift from them. The first seed against the real scanner produced eleven audit events that simply did not exist before.

The rule I take from it: a fixture that comes in through the back door does not test the product, it tests the database.

2. The extension decided the verdict

This one surfaced when I ran the whole flow through the API instead of the seeder. The commercial verdict changed, and it took me a moment to accept the test was right.

A NETWORK_SKETCH.pdf was being classified as an electrical drawing, because .pdf was mapped to a specific type. Network coverage went from 33 % to 0, weighted coverage from 54 % to 47, and the budgetary verdict flipped from READY WITH ALLOWANCES to NOT READY.

Nobody typed anything wrong. The file was a PDF.

A PDF can be an electrical drawing, a network sketch or a report, and which of the three it is moves an evidence domain that decides whether the job can be quoted. Today a .pdf lands unclassified and contributes no evidence until a person says what it is. An extension has no authority to move a verdict.

3. The unfamiliar header deleted the program

I fed the parser a file with a foreign header but valid logic inside. It returned zero rungs, an empty model and no software findings at all — including the two instructions that block the entire migration.

It did not crash. There was no exception. A clean assessment of a program that was never read, which in this domain reads as "we found nothing wrong".

Two fixes. Rungs now survive an unreadable header: the rung syntax is the part of the format that is publicly confirmed, so an unknown container costs us the container, never the logic. And a rule fires BLOCKED when a supplied source yields no rungs at all. A file we did not understand must not look like a file with nothing wrong.

4. The script that hunted false greens produced one

This is the best of them. I wrote a verifier that runs the project's five layers and prints a single verdict. Its first run printed:

[ok]   0 tests

bc was missing from that shell, the sum of tests failed silently, it came out zero — and the script reported it as green. Writing the tool whose only purpose was to catch false greens, I produced one.

It now has an explicit guard: if the engine reports fewer than thirty tests or the API fewer than forty assertions, it fails. A run that executed nothing must not look like a run that passed.

What the four have in common

In none of them was there an exception, a stack trace or a red test. In all four, the check passed because the thing being checked never happened: the scan that did not run, the type nobody declared, the parse that came back empty, the sum that came out zero.

The defence is not more tests. It is asking every green an uncomfortable question: if this had not run, would it look any different? If the answer is no, that green is measuring nothing.

That is why the suite now breaks invariants on purpose to check that it bites — I disabled the scanning gate, tenant isolation, logo validation and the rule that proposing never approves: all four produced failures. And it is why the fixture comes in through the front door, the extension decides nothing that moves money, and a green with zero tests is red.