The Webhook That Always Succeeded

Somewhere in most payment systems there is a piece of code whose only job is to tell somebody else that something happened. A payment settled. A mandate was registered. A refund cleared. It is usually the least interesting code in the repository — a POST, a retry, a log line — and it is the only part of the system whose success you cannot confirm by looking at your own database.

I found one of these on a regulated insurance platform. It had been reporting a hundred percent delivery rate for months. The dashboard was green. The delivery log contained no errors at all. The integration partner on the other end had received nothing for over a week.

Success is not a status code

The credential the delivery used had gone stale. That part is mundane; credentials expire, and a system that can't survive that is a system waiting to fail.

What made it invisible was the shape of the failure. The partner's gateway did not answer an unauthenticated request with a 401. It routed the request to a catch-all that returned 404. So the delivery code received a well-formed HTTP response, over a healthy connection, in a normal amount of time.

And the code did roughly this:

try {
  await http.post(partnerUrl, payload, { headers });
  await ledger.recordDelivered(eventId);
} catch (err) {
  await ledger.recordFailed(eventId, err);
}

There is no bug in that snippet that a linter will find. It is well-formed, defensively written, and wrong. It treats the absence of a thrown exception as evidence of delivery. The client library did what it was asked: it made a request and it got a response. Nobody asked what the response said.

The general form of this mistake is worth naming, because it is not really about webhooks: we conflated transport success with delivery success. The network worked perfectly. The delivery did not happen. Those are different claims, and only one of them was being checked.

Why nothing caught it

A single missing assertion explains the bug. It does not explain how the bug survived for a week in a system with monitoring, structured logs and an append-only ledger. That took four things going wrong together, and every one of them generalises:

• The ledger recorded intent, not outcome. It was genuinely append-only, which felt rigorous. But what it appended was we attempted this and did not throw. An immutable log of a wrong conclusion is still wrong; it is just harder to argue with, because immutability reads as authority.
• Alerting watched the error rate. There were no errors. An error-rate alert on a code path that has stopped producing errors is not quiet because things are healthy. It is quiet because it has nothing to say.
• Nobody was watching for absence. Zero deliveries in an hour and a genuinely quiet hour produce the same graph. Every dashboard we had could distinguish many failures from few failures. None could distinguish no traffic from no successes.
• The receiver never complained. The integration was new enough that the partner had no established expectation of volume. You cannot rely on the other side noticing your outage when the other side has not yet learned what normal looks like.

The last one is the part I think about most. We had implicitly outsourced detection to somebody who had no way to detect it.

What at-least-once actually requires

"At-least-once delivery" gets written into design documents as though it were a property of the transport. It isn't. It is a property of the whole loop, and you only get to claim it if four things are true:

1. "Delivered" is defined, and asserted. Not "did not throw" — an explicit check on the status and on whatever the receiver returns to acknowledge it. If the contract has no acknowledgement, that is the first thing to negotiate, not a detail to work around.
2. The outcome is persisted per attempt, not the intent. One row per attempt, carrying the status code, the response body and the reason for the verdict. The question "what did the receiver actually say on the third try" should be a query, not an archaeology project.
3. Failures land somewhere you are forced to look. A dead-letter queue is not a place to lose things politely; its depth is the alert. If nothing ever reads from it, you have built a bin, not a safety net.
4. You reconcile against the receiver's view. Your ledger describes what you believe. Periodically ask the other side what they have, and treat the disagreement as the signal. This is the only one of the four that would have caught this particular bug on day one.

The related trap sits one layer earlier. If you commit a database transaction and then publish the event, there is a window where the state changed and nobody was told, and no amount of retry logic downstream repairs it — the event was never created. The outbox pattern exists for exactly that gap: write the event into the same transaction as the state change, and let a separate process drain it.

Alert on absence

If you take one operational rule from this, take this one: every outbound integration needs an expected-rate floor.

"Fewer than N successful deliveries in the last hour" is a trivial alert to write and it catches an entire category of silent failure that error-rate alerting structurally cannot see — expired credentials, a partner quietly changing a route, a queue consumer that died without an exception, a feature flag that turned the publisher off. All of these look identical to peace and quiet.

I have written before about watching AI agents at the syscall level, and it turns out to be the same idea in a different costume. A workload's own telling of what it did is not evidence. It is a claim, produced by the same code whose correctness is in question. The only trustworthy account comes from outside the thing being observed — the kernel in that case, the receiver in this one.

The uncomfortable part

I would rather not present this as a solved problem, because the corners are genuinely awkward.

• You often cannot reconcile against a receiver you do not control. Plenty of partners expose no endpoint that will tell you what they hold. Then your options narrow to sequence numbers, periodic acknowledged digests, or a phone call.
• Idempotency only works if the other side honours it. You can send a deduplication key with every event and retry safely in principle; whether the receiver actually deduplicates on it is a question about their implementation, and the honest answer is often that nobody has tested it.
• Replaying a week of events is a product decision, not a technical one. Once you can redeliver, you have to decide whether you should. In payments a stale event is not merely late, it can be actively wrong — a status that has since been superseded, a link that has expired, a notification about money that has already moved twice. "Deliver everything eventually" is sometimes the incorrect answer, and choosing when to give up permanently belongs to whoever owns the customer experience, not to the retry policy.

None of that undermines the four requirements above. It just means the fourth one is frequently the hardest, and the one people skip.

The shape of the lesson

The fix, when it came, was small: assert on the response, record the real outcome, fail loudly, and alert when the successes stop arriving. Nearly all of the value was in the last clause.

A system that reports its own success is reporting a belief. Instrument the boundary, not the intention.