What is change data capture (CDC)?

CDC streams inserts, updates, and deletes from a database's transaction log to downstream systems in near real time. It is commonly used to sync data warehouses, search indexes, and caches without adding write load to application code.

What is the transactional outbox pattern?

The transactional outbox stores events in the same database transaction as business data, then a separate publisher sends them to a message bus. It ensures you never commit a state change without a corresponding event record (or vice versa).

When should I use CDC instead of application events?

Use CDC when you cannot modify a legacy application, when you need all table changes for analytics, or when multiple teams consume the same physical data. Use application events when you want to publish domain intent with richer payloads and versioning tied to your bounded context.

Why must event consumers be idempotent?

Message brokers typically guarantee at-least-once delivery. The same event may be processed twice after crashes, retries, or consumer restarts. Idempotent handlers (dedupe keys, upserts, natural keys) prevent duplicate side effects.

What is wrong with dual writes?

Writing to a database and separately sending a message without a shared transaction leads to inconsistency: one can succeed while the other fails. Recovery requires error-prone reconciliation jobs. Outbox or CDC aligns the two paths.

Architecture guide

Reliable Data Integration: Event-Driven Patterns, CDC, and the Outbox

Reliable data integration avoids dual writes by pairing state changes with outbox rows in one transaction, using CDC when legacy apps cannot emit events, and making every consumer idempotent because brokers deliver at-least-once in production.

Reliable data integration moves facts between services, databases, and analytics systems without losing updates or double-applying them when networks retry. Three dominant patterns are domain events (publish from the app), the transactional outbox (atomic commit with state), and change data capture (CDC—stream changes from the database WAL/binlog). This article explains when each fits, ordering and idempotency requirements, and the operational checklist we use before recommending one on client builds. For transactionally consistent messaging, the outbox pattern is well documented in enterprise integration literature and Debezium's CDC documentation (open-source CDC platform) as complementary tools: outbox for application-intent events, CDC for DB-level change streams.

Key takeaways

Never dual-write: do not commit a row then separately fire a message without a shared transactional guarantee—you will eventually have one without the other.

Use outbox when the message must reflect business intent committed with the same ACID transaction as your state change.

Use CDC when many consumers need all row changes or when you cannot change the legacy application but can read replication logs.

All consumers must be idempotent; assume at-least-once delivery from Kafka, SQS, Pub/Sub, or SNS in real deployments.

Pattern comparison table (text)

Outbox — Consistency: strong with OLTP write. — Best for: explicit domain events, cross-service workflows. — Caveat: application must write outbox rows; schema migrations touch publisher.

CDC — Consistency: eventual from DB commit point. — Best for: search indexes, warehouses, caches fed from existing tables. — Caveat: exposes physical schema; refactors may break consumers without contracts.

Choreographed sagas without outbox — Consistency: fragile. — Best for: rarely. — Caveat: compensations and orphaned steps multiply; prefer outbox or orchestration with a durable process manager for money-moving flows.

Implementing the transactional outbox

Add an outbox table (id, aggregate_type, aggregate_id, payload, created_at) in the same database as your write model. In one transaction: update business row, insert outbox row.

A relay process (polling or log-tailing) publishes to the bus and marks rows published or deletes them; use lease/lock to avoid duplicate publish under crashes.

Messages carry versioning (event schema v2) and correlation IDs for tracing across services.

Implementing CDC safely

Run a CDC connector (Debezium, AWS DMS, Fivetran, etc.) against a read replica or primary per vendor guidance. Monitor replication lag; analytics fed from CDC is only as fresh as lag allows.

Treat CDC events as physical change feeds: renames and drops break downstream parsers. Mitigate with consumer contracts, deprecation windows, or views as stable publication surfaces.

For GDPR and right-to-erasure, ensure downstream sinks honor deletes—CDC emits tombstones in Kafka compacted topics when modeled correctly; batch warehouses may need periodic reconcile jobs.

Ordering and partitioning

If per-order events must stay in order, partition the topic by order_id (or tenant + order). Global ordering is expensive and rarely needed.

Consumer parallelism increases throughput but breaks per-partition ordering guarantees if you shard work incorrectly—preserve partition affinity for ordered handlers.

Operational checklist before go-live

Dead-letter queues and replay tooling for poison messages.

Metrics: publish lag (outbox unprocessed count), consumer lag, CDC connector lag.

Load tests on peak publish rates; brokers and consumers sized with headroom.

Disaster recovery: can you rebuild search or warehouse from a snapshot + CDC backlog? Test it once per quarter.

Limitations

Outbox couples event shape to OLTP schema deployment cadence; CDC couples to physical tables—choose based on who owns evolution.

Cross-database transactions are not possible with a single outbox; cross-database sagas need careful design or consolidation.

Vendor-specific CDC limitations (data types, DDL filters) vary—pilot on a copy of production traffic before committing.

Explore our Product Strategy, Custom Software, and AI Development services, or get in touch to discuss your project.