Data / Infrastructure

groundtruth — Verified data extraction

A web extraction engine where data arrives verified, not merely collected: every delivery ships with a report of what could NOT be confirmed.

groundtruth — Verified data extraction
Data jobs rarely fail with an error. They fail silently: the script runs, the file arrives, and two hundred rows say "out of stock" because one selector moved. Nobody notices until someone makes a decision with that data. I built the engine around that failure. Every run emits a QA report —rows extracted, rows missing each field, exact counts— and a run that does not reach COMPLETED is never delivered: the state machine refuses to move from PARTIAL to DELIVERED without explicit human approval. The decisions holding that up are small and concrete. The content hash excludes the capture timestamp, or change detection lies every night. Money is Decimal with a mandatory locale and no default, because 1.234 means different things in different countries. Products are matched by SKU before name, and an uncertain match goes to a review list instead of being reported as a price drop that never happened. The test fixtures include the legitimate empty page and the anti-bot page that returns 200 — the two cases nobody saves, and the only ones that tell "this category is empty" apart from "I have been blocked".

Tech stack

Python · SQLite · Pydantic · Playwright · pytest

Outcomes

  • 135 tests, none touching the internet
  • A QA report with every delivery
  • PARTIAL never reaches DELIVERED without human sign-off