What an AI agent must not decide

A customer had a hold of over seventy thousand sitting against an order the system considered complete. It had to be closed. The question was how.

If the payout to the provider went out, the fix is to settle: the money comes off the held balance and off the total, because it is gone. If the payout never went out, the fix is the opposite: release it back to available.

The two fixes are opposites. And from the database they are indistinguishable.

Getting it wrong in one direction means paying the beneficiary twice. Getting it wrong in the other means taking seventy thousand from a customer who actually had it. There is no third option that is "roughly right".

I work with AI agents on this system every day. I run several in parallel, with persistent domain memory and automated change integration. I hold far more surface area than I would alone. And still, not one of them made that call.

The failure mode is not the one you picture

When someone distrusts building with agents, they almost always picture bad code: broken functions, invented syntax, things that do not compile.

That problem barely exists. Tests catch it, the compiler catches it, you catch it by reading.

The real failure mode is different: plausible code, delivered with confidence. A fix that looks right, passes the tests, and is accounting-wrong. An agent has no way of knowing that the operational table says "how things look today" and not "what happened". It will read the state, reason impeccably about what it read, and propose something perfectly coherent with a false premise.

In a CRUD app that costs an afternoon of debugging. On a money path it costs other people’s money.

Three limits that actually work

Domain memory, not code memory. What an agent needs to remember is not where the function lives. It is that this column is left null on purpose, that this "bug" was a feature flag switched off, that this amount is the net and not the gross. I keep over a hundred and fifty such facts, versioned. Without them every session repeats the previous session’s mistake with total self-assurance.

Scope bounded in writing. "Migrate this" is an invitation for the agent to touch seven other things it found improvable. Explicit limits — this architecture, this module, do not mix with the legacy — are not bureaucracy: they are what stops a one-hour task from becoming a diff nobody can review.

A human gate wherever the effect cannot be undone. An agent can find, measure, propose and execute almost everything. What it does not do is move customer money, or switch on an automated process over data nobody measured. Not because it could not write that code. Because the cost of being wrong is not a line in a log: it is a wrongly moved balance in a real person’s account.

The mistake I made writing about this

A few days ago I published a case study about that frozen wallet. The closing line said the balance had been released and the root cause fixed.

The root cause was fixed. The balance was not released: it went to provider reconciliation and to compliance, precisely because releasing it blindly was the mistake this article is about. The published text claimed an outcome that never happened, and it read well.

I caught it by checking against the actual state of the tickets. Not against my memory, and not against what sounded reasonable.

That is exactly the failure mode I have been describing, applied to myself. Plausible, coherent, presentable, and false. Verification is not a ceremonial step for when there is spare time: it is the only thing separating what you believe happened from what happened.

What I measure now

I stopped measuring how much code ships per hour. I started measuring something else: how many of the decisions the system made this week were reversible.

An agent that writes three times the code and makes one wrong irreversible decision has left you worse off than before you started. One that writes half as much but never crosses that line leaves you building for years.

Speed is worth nothing if it breaks the books. And the books do not tell you when they break: you find that out weeks later, when a customer asks about their money.

(This is exactly why I am building LedgerCore: if every hold and every release are first-class entries, the question "did this go out or not?" has an answer in the books. And a decision with an answer no longer needs guessing — human or artificial.)