The Dual-Write Problem: Why Your Event Never Fired

The ticket says “customer data is out of sync.” The order shows as paid in the billing system but the fulfillment system never got the message, so nothing shipped. Everyone assumes a bug in the consumer. It usually isn’t. The event was never published, or it was published for an order that then rolled back.

This is the dual-write problem, and after 20 years we can tell you it’s behind a large share of the “systems disagree” incidents we get called into. It’s also one of the most under-taught failure modes in integration, because the code that causes it looks completely correct.

The code that looks fine

Here’s the shape. A service handles a request, and it needs to do two things: update its own database, and tell the rest of the world what happened by publishing an event to a broker like Kafka.

saveOrder(order)          // write to the database
publish("OrderPaid", order)  // write to the broker

Two writes. They look atomic sitting next to each other in one function. They are not. A database transaction covers the database and nothing else. It has never covered your message broker. There is no single transaction that spans both.

So there’s a gap between line one and line two, and things die in that gap.

Two ways it fails, both silent

Run the two orderings in your head.

Database first, then publish. You commit the order as paid. Then, before the publish succeeds, the process crashes, or the broker is briefly unreachable, or the pod gets rescheduled. The state changed but no event went out. Billing thinks the order is paid. Fulfillment never heard. Nothing ships. No error was thrown at the customer (the request succeeded), so nobody knows until the “where’s my order” ticket arrives days later.

Publish first, then database. You emit OrderPaid, then the database write fails and rolls back. Now there’s an event announcing something that never actually happened. Downstream systems ship an order the source system has no record of. This is worse, because the phantom event is loose and you can’t recall it.

Confluent states the trap plainly: sending a message in the middle of a transaction isn’t reliable, and sending it after the transaction commits carries no guarantee the process won’t crash first. Either way, one of your two writes can vanish, and the failure is silent by construction.

The instinct is to reach for a distributed transaction: two-phase commit across the database and the broker. Don’t. It drags in a coordinator, hurts availability and throughput, and most modern brokers don’t support it well anyway. The whole point of the fix below is to avoid 2PC entirely.

The fix: make it one write

The transactional outbox pattern, documented cleanly by Chris Richardson, resolves this with one idea: stop writing to two systems.

Instead of publishing to the broker, the service inserts the event as a row into an outbox table, in the same database transaction that updates the business data. Now both writes hit the same database, so they’re genuinely atomic. Either the order is saved and the event row exists, or neither does. The dual write is gone because there’s only one write.

BEGIN
  saveOrder(order)
  insertIntoOutbox("OrderPaid", order)
COMMIT

The event is guaranteed to exist if and only if the transaction committed. That’s the property you wanted all along.

A separate process, call it the relay, reads unpublished rows from the outbox and pushes them to the broker, then marks them done. The relay is now the only thing talking to Kafka, and if it crashes mid-publish, the row is still sitting in the outbox waiting. Nothing is lost.

How the relay reads the outbox

There are two ways to move rows from the outbox to the broker, and the choice has real consequences.

Polling. A background job queries the outbox for unpublished rows every so often and publishes them. It works with any SQL database and it’s simple to build and reason about. The costs: polling adds load and a little latency, and keeping strict event order across polling cycles is genuinely hard. For a lot of systems, simple and slightly-late beats clever and fragile.

Log tailing with change data capture. Instead of polling, you tail the database’s commit log (the Postgres WAL, the MySQL binlog) and publish each outbox insert as it lands. Debezium is the standard tool here; it reads the log and streams changes to Kafka with no polling code of yours in the loop. Its Outbox Event Router even reshapes the raw change events into clean per-aggregate messages, routing by an aggregatetype column and keying by aggregateid. Lower overhead, accurate, and it scales, at the cost of being database-specific and needing care to avoid duplicate publishes.

We reach for CDC when volume is high or ordering matters, and for polling when the system is small and the operational simplicity is worth more than the latency. Neither is wrong. Picking without knowing which property you need is.

The catch you must design for

The outbox fixes the lost event. It does not give you exactly-once delivery, and pretending it does is the next bug.

The relay can publish a row, then crash before it marks the row as done. On restart it sees an unmarked row and publishes it again. Delivery is at-least-once: every event arrives, but some arrive more than once. Fulfillment might receive OrderPaid twice for the same order.

So the consumer has to be idempotent. Track the IDs of events you’ve already processed and discard duplicates. It’s the other half of the pattern, not optional bolt-on hardening. An at-least-once producer without an idempotent consumer just moves the inconsistency downstream.

One operational note people miss: a high-write outbox table accumulates dead rows fast, and on Postgres that means bloat and vacuum pressure. The outbox needs pruning and monitoring like any hot table. It’s a queue that lives in your database, and you have to treat it like one.

Why this is worth knowing cold

The reason we teach this pattern on almost every event-driven project is that the naive version works in the demo. Two writes next to each other pass every test with a healthy broker and a process that never crashes. The failure only shows up under the exact conditions production guarantees you’ll eventually hit: a broker hiccup, a deploy mid-request, a rollback under load. Then a payment doesn’t ship, and everyone blames the consumer.

The database transaction covers the database. It never covered the broker. Put the event in the same transaction as the data, let a relay carry it out, and make your consumers idempotent. Do that and the “systems disagree” ticket you’ve been firefighting mostly stops arriving.