A scraper that returns 300 empty cells looks exactly like one that worked

The script finished with no errors. The file was generated with its 312 rows. Names were there, prices were there.

And the availability column said UNKNOWN in all 312.

On the website, every product said "In stock" in plain letters.

The bug was five characters long

The selector pointed at the right place:

availability = row.css(".stock::text").get()

In the HTML, that node looked like this:

<p class="stock">
    <i class="icon-ok"></i>
    In stock
</p>

::text does not return "the element text". It returns the text nodes, in order. And the first one, before the <i>, is the newline with its indentation. get() takes the first and leaves.

So the extractor faithfully collected four blank spaces. Normalized them to an empty string. And the empty string matched no known value, so the normalizer answered the only honest thing it could: UNKNOWN.

The fix is getall() and join:

availability = " ".join(row.css(".stock::text").getall())

Five characters. The bug is not the interesting part — what it took to see it is.

Why this is worse than an exception

An exception is a good citizen. It breaks the run, leaves a trace, shows up in the log, fires the alert. Somebody looks at it.

This failure did the opposite: it produced plausible output without producing an error.

And plausible output is worse than no output, because it moves forward. The file gets delivered. Somebody opens it, sees 312 populated rows and assumes the job was done. The broken column surfaces weeks later, when someone asks why the report says no competitor has any inventory.

By then a decision has been made on that data.

The three that behave the same way

While building the engine, two more of the same family showed up. Neither throws anything.

The hash that included the capture timestamp. Change detection compares today content against yesterday by hash. If collected_at goes into that hash, every product changes every night — because the time is always different. The change report fills with noise and stops being read. The fix is an explicit whitelist of fields that enter the hash, not a whole dict.

The decimal that was not the same number. Decimal("799.00") and Decimal("799.0") are equal when compared and different when serialized. The same price, with no real change, showed up as modified depending on which page it came from. Fixed by normalizing the exponent before hashing.

The letter that ate a currency. The list of symbols to strip included "R" for the South African rand. The text "EUR 45" went through, lost its R and became "EU 45". The parser found no valid number and returned null. One price gone, silently. Now ISO codes are stripped before symbols, and there are no letters in the symbol list.

All four share the pattern: the system does exactly what you asked, and what you asked was not what you meant. There is nothing a try/except can catch.

What does catch them

It was not better code. It was changing what gets delivered.

Every run now emits a QA report next to the file:

Rows extracted         312
Missing price            0
Missing availability   312   <-- here
Missing SKU              7

That third line is the whole system. You do not need to review the file or know the domain: if a column shows the full count in the missing row, something is broken, and you see it before delivering.

There is a rule alongside it: a run that does not reach COMPLETED is not delivered. The state machine refuses to move from PARTIAL to DELIVERED without explicit human approval — not a config flag, a signature.

The question that stayed

Closing the engine left me with one I now use on any process that produces data:

What does this system look like when it fails silently?

If the answer is "the same as when it works", it is not finished. Test coverage does not matter: tests check what you thought to check, and this kind of failure lives precisely in what did not occur to you.

Short version: an error complains; wrong data waits.