Change Data Capture (CDC)

Problem

You want to derive downstream systems (search index, cache, analytics) from your primary database — but polling for changes is expensive and misses deletes.

Context

The DB is the source of truth; you need to keep other systems in sync without changing the primary application code.

Solution

Read the database's transaction log (WAL for Postgres, binlog for MySQL) directly. Every INSERT/UPDATE/DELETE emits an event. Tools like Debezium turn these into Kafka topics. Downstream consumers process the events to update their views.

Trade-offs

• Requires low-level access to the DB (replication slot in Postgres)
• Log format is DB-specific — Debezium handles this per DB
• Consumers see raw table-level changes, not application-level events — must interpret
• Log replication slot must be pruned; otherwise DB disk fills up

Failure modes

• Consumer falls too far behind; replication slot grows and eventually fills disk
• Bad DDL causes CDC pipeline to fail — every schema change is a coordination event
• Events lag behind commits by seconds under load

When to use

• Deriving read models from a primary DB without app changes
• Data lakes / analytics: streaming a snapshot of your DB to a data warehouse
• Zero-downtime migrations: CDC keeps old and new DB in sync during cutover

When NOT to use

• App-level events would be cleaner (use outbox instead)
• Small-scale — polling is simpler operationally
• Team unfamiliar with DB internals