Transactional outbox

Problem

You need to atomically update your database AND publish an event to Kafka. Two-phase commit isn't available; publishing before the DB commit risks phantom events; publishing after risks lost events on crash.

Context

The dual-write problem: any two systems (DB + event bus, DB + cache) can't be updated atomically without shared transactions.

Solution

Write the event to an 'outbox' table in the same DB transaction as the state change. A separate relay process reads the outbox table and publishes to Kafka. Because the outbox write is in the same transaction, either both succeed or both fail. The relay ensures at-least-once delivery.

Trade-offs

• Event publication is delayed by the relay's poll cycle (usually ms-s)
• Requires an outbox table (extra schema)
• Relay must handle at-least-once delivery — consumers must be idempotent
• Ordering of events matches the transaction order — good
• Adds latency to writes if the outbox table is large

Failure modes

• Relay crashes mid-flush; some events published, offset not committed → duplicates on restart
• Outbox table grows unbounded if relay is slow — needs cleanup
• Idempotency in consumers is non-negotiable

When to use

• Microservices where DB updates must reliably trigger events
• You need at-least-once event delivery guaranteed to match DB state
• You can afford ms-s of event publication latency

When NOT to use

• Fire-and-forget events where loss is OK
• Ultra-low-latency requirements — outbox adds polling delay