Money never moves twice

A transactional backend does not fail the day it goes down. It fails the day it works twice.

Almost everything written about payment backends stops at the headline: "wrap it in a database transaction", "add an idempotency_key". That is the headline, not the problem. The real problem shows up when a money operation crosses three systems that share no transaction: your database, the payment provider (PSP) and the job queue. None of the three can roll back the others.

This article is about that seam. What breaks, why, and the design that holds.

There is no distributed commit

Take a minimal operation: charge 100 USD to a card and credit 100 USD to the user's balance.

DB::transaction(function () use ($user, $amount) {
    $charge = $psp->charge($user->card, $amount);   // network
    $user->balance += $amount;                      // database
    $user->save();
});

This code is wrong, and not subtly so. It is wrong for three separate reasons.

The network call sits inside the transaction. While the PSP takes four seconds, you are holding row locks open. Under load that is a connection pileup and a deadlock. A database transaction should last microseconds, not seconds.

The rollback is a lie. If $user->save() fails, the database rolls back. The charge at the PSP does not. You just took 100 USD from someone without crediting them. The transaction gave you the illusion of atomicity over a system that never participated in it.

A timeout is not a failure. If $psp->charge() times out, you do not know whether the charge happened. The response was lost, not necessarily the operation. Retrying blindly charges twice.

The uncomfortable conclusion: atomicity between your database and a third party does not exist. What you can build is something weaker but sufficient: convergence. The system may be temporarily inconsistent, as long as it has a deterministic path back to the correct state.

Idempotency is a contract, not a column

The useful definition is not "do not repeat", it is: running the operation N times produces exactly the same observable effect as running it once.

Two consequences that are almost always missed.

The client generates the key, not the server. If the server generates it, every retry produces a new key and protects nothing. The key is generated by whoever initiates the intent, and stays stable across every retry.

The key must cover the request body, not just the operation. If someone reuses the same key with a different amount, that is not a retry — it is a client bug, and cheerfully returning the cached response hides it. Store a hash of the body alongside the key.

CREATE TABLE idempotency_keys (
    key            VARCHAR(64)  NOT NULL,
    scope          VARCHAR(64)  NOT NULL,   -- endpoint + tenant
    request_hash   CHAR(64)     NOT NULL,   -- sha256 of the canonical payload
    state          ENUM('in_flight','done') NOT NULL,
    response_code  SMALLINT     NULL,
    response_body  JSON         NULL,
    created_at     TIMESTAMP    NOT NULL,
    PRIMARY KEY (scope, key)
);

And the flow:

$row = IdempotencyKey::firstOrCreate(
    ['scope' => $scope, 'key' => $key],
    ['request_hash' => $hash, 'state' => 'in_flight']
);

if ($row->request_hash !== $hash) {
    throw new IdempotencyConflict();   // 422: same key, different body
}

if ($row->state === 'in_flight' && ! $row->wasRecentlyCreated) {
    return response('', 409);          // a run is already in progress
}

if ($row->state === 'done') {
    return response($row->response_body, $row->response_code);
}

// ...actually execute, then mark 'done' with the response

The detail that makes all the difference: the lock is the database's unique index, not an if (exists) in application code. Two concurrent requests with the same key collide on the INSERT; one wins, the other reads the existing row. A pre-check in PHP has a race window between the SELECT and the INSERT that, under real concurrency, fires every single day.

The in_flight state matters too: without it, two simultaneous requests both see "no stored response" and both execute.

A balance is never a column

The most expensive structural mistake in a money backend is this one:

UPDATE users SET balance = balance + 100 WHERE id = ?;

It is atomic, it is fast, and it is indefensible. Because when the balance does not add up — and one day it will not — there is no way to know why. You lost the history. You have a number with no provenance.

Accounting solved this in the 15th century. A balance is not stored: it is derived.

CREATE TABLE ledger_entries (
    id             BIGINT PRIMARY KEY,
    transaction_id BIGINT  NOT NULL,   -- groups the legs of one entry
    account_id     BIGINT  NOT NULL,
    amount_minor   BIGINT  NOT NULL,   -- signed cents. Never a float.
    currency       CHAR(3) NOT NULL,
    created_at     TIMESTAMP NOT NULL
);

With two non-negotiable invariants:

• Entries are immutable. No UPDATE, no DELETE. A mistake is corrected with a reversing entry, exactly as in real accounting. That is what preserves the audit trail.
• Every entry sums to zero. Money does not appear or vanish: it moves between accounts.

A 100 USD charge with a 3 USD fee, then, is not two rows but four:

• psp:clearing → +10000
• user:42:available → +9700
• revenue:fees → +300
• liability:user_funds → -10000

Sums to zero. And now "why does this user have this balance?" has an executable answer instead of a guess.

"Deriving the balance is slow"

Yes, if you derive it every time. The fix is not to go back to the mutable column: it is a snapshot. A table of materialized balances carrying the id of the last entry included. A read takes the snapshot and sums only the entries after it. The snapshot is a rebuildable cache, not the source of truth — and that distinction is what saves you when it gets corrupted, because it is recomputed from scratch.

Stated once: the cache may lie; the ledger may not. If your system has a path where the balance moves without an entry being written, that path is a bug with a pending date.

Rebuilding the atomicity the network took away

We know the charge cannot be atomic. So we split it into steps that are individually atomic, and make the whole thing converge.

Step 1 — record the intent before touching the network.

$payment = Payment::create([
    'state'           => 'pending',
    'amount'          => $amount,
    'idempotency_key' => $key,
]);   // COMMIT. The database transaction ends here.

Now there is durable evidence that the operation started, before it happens. If the process dies in the next step, a pending record exists for someone to reconcile. Without this step, a crash leaves a charge at the PSP that your system does not know exists. That is orphaned money, and it surfaces weeks later as a complaint.

Step 2 — call the PSP outside any transaction, and pass it your idempotency key. Almost every serious provider accepts one. That is your real protection against double-charging on timeout: retry with the same key and the provider returns the original charge instead of creating a new one.

Step 3 — apply the result, idempotently.

DB::transaction(function () use ($payment, $charge) {
    $fresh = Payment::whereKey($payment->id)
        ->where('state', 'pending')   // the guard that makes this safe
        ->lockForUpdate()
        ->first();

    if (! $fresh) {
        return;                       // another process already applied it
    }

    $ledger->post($fresh, $charge);   // the four legs, summing to zero
    $fresh->update(['state' => 'settled']);
});

The where('state','pending') is what turns a dangerous operation into a safe one: the state transition and the ledger entry happen in the same database transaction, and can only happen once. A second attempt finds no row and does nothing — which is precisely the definition of idempotent.

Note that the only remaining database transaction is short and contains no network call.

The part almost nobody builds: reconciliation

Everything above reduces the probability of inconsistency. It does not eliminate it. A window always remains: the process dies between step 2 and step 3.

A payment system without reconciliation is a system that trusts itself not to fail. The ones that survive in production have a boring process running every few minutes:

• Find payments stuck in pending for more than N minutes.
• Ask the PSP about that idempotency key: does this charge exist?
• If it exists, apply step 3 — it is idempotent, so it is safe.
• If it does not exist and the window expired, mark it expired.
• If the provider answers something that matches no expected state, do not guess: flag it for human review and alert.

That last point is what separates a financial system from an ordinary application. With money, failing loudly beats resolving creatively. An ambiguous case settled by a heuristic is a discrepancy discovered at month-end close, when there are already hundreds of them.

And on top of everything, one invariant check running continuously — the only metric that truly matters:

-- Must return zero rows. Always. If it does not, stop and look.
SELECT transaction_id
FROM ledger_entries
GROUP BY transaction_id
HAVING SUM(amount_minor) <> 0;

The traps you only learn in production

Floats do not exist in money. Integers in the minor unit, or fixed-precision decimals. And the rounding decision for fee calculation must live in exactly one place: if two modules round differently, the drift is one cent per operation — invisible for weeks, then impossible to trace.

Provider callbacks arrive out of order, duplicated, and sometimes before your own response. The "charge confirmed" webhook can land while your original request is still waiting. That is why the webhook handler must go through the same state-transition guard as the main flow. If you have two paths applying the same effect, you have two chances to apply it twice.

States must be a machine, not a string. Any transition outside the declared graph must throw. A free-text field that any part of the codebase can overwrite guarantees that sooner or later something goes from refunded back to settled.

Retrying is not free. A retry without exponential backoff and jitter turns a brief provider outage into a denial-of-service attack you run against them yourself, precisely while they are trying to recover.

A disabled feature flag can move money without recording it. If the ledger sits behind a flag and the payment flow does not, the system lands in a state no test covers: the balance changes and the ledger never hears about it. Every flag governing a money path needs to be defined in terms of which invariant stops holding when it is off.

Closing

If I had to compress all of it into four sentences:

• Distributed atomicity does not exist: build convergence, not the illusion of a transaction.
• Idempotency rests on a unique index, not on an if.
• Balances are derived from immutable entries; a mutable column is lost history.
• Without reconciliation there is no system, only a system that has not failed yet.

None of this is exotic. It is what separates a backend that processes payments from one that can also prove which payments it processed.