Five scars from running payments infrastructure

I found it on a Tuesday, cross-checking a payment provider’s statement against our own books.

The provider said we owed them six figures. Our systems said we owed nothing. The debt existed nowhere at all.

I have spent years running payments and remittance infrastructure. These are the five scars it left me, and what each one taught.

1. The debt that did not exist. The ledger’s opening entry discarded negative balances as "noise". One of those negatives was real debt with a provider. Lesson: a negative custody position is a liability, not an anomaly.

2. The 97% frozen wallet. Releasing a hold depended on a collateral field that a new flow never created. Result: eternal holds with not a single open order. Lesson: holds are first-class ledger entries, not inferences over side tables.

3. Balances summing the whole history, on every request. Slow — and two concurrent requests could spend the same balance. Lesson: the balance is materialized in the SAME transaction as the entry.

4. "1.500" became 1500. A decimal helper guessed the dot was a thousands separator: a x1000 error on a money path. And there was a unit test pinning the bug as the specification. Lesson: money = integers in minor units + asset + exponent. Never floats, never parsers that guess.

5. Weeks of forensic SQL. Mutable rows only store how things look TODAY; the intermediate states were gone. Lesson: append-only is not academic purity, it is being able to query the timeline instead of interviewing the survivors.

The underlying pattern: operational tables hold "current state", not financial truth. It is not carelessness, it is the data model.

That is why I am building LedgerCore: double-entry ledger infrastructure, where an entry that does not balance is never posted, history cannot be edited and money is integers.