Handling Schema Evolution in CDC Streams

A column is renamed on a Tuesday afternoon. The migration passes, the application is fine, and the search index quietly stops updating three hours later when the first row on that table changes. This page makes source schema changes survivable for the consumer that feeds your index: which changes are safe, how to enforce that at the boundary, and how to execute the unsafe ones without dropping events. It extends change data capture setup in the data ingestion and synchronization pipelines area.

Prerequisites

  1. A running CDC connector emitting envelope-style events with a schema block (Debezium or equivalent).
  2. A schema registry, or the willingness to add one — enforcement without a registry is convention, not control.
  3. A dead-letter destination for events the consumer cannot handle.
  4. Agreement that captured tables are a published interface, so migrations touching them are coordinated.

Diagnosis: the break is delayed, which is what makes it expensive

Adding a column changes nothing until the first row carrying it flows through, so the failure appears hours or days after the migration, decoupled from the change that caused it. By then the deploy is out of the incident window and nobody connects the two.

The consumer log names the field but not the cause:

ERROR  apply_event failed  topic=shop.products partition=0 offset=88214
       KeyError: 'title'
       payload.after keys: ['id', 'product_title', 'price', 'brand', 'created_at']
#                            ^^^^^^^^^^^^^ renamed from `title` in migration 2026_07_21

The consumer is not wrong to fail — it cannot invent the field — but how it fails decides whether this is a five-minute fix or an afternoon of replaying a topic. A consumer that crashes stops the whole partition. A consumer that dead-letters the event keeps every other document flowing while you fix the mapping.

Which source schema changes are safe for a downstream consumer Adding a nullable column and widening a type are safe, adding a required column and dropping a column need a deploy first, and renaming or narrowing a type requires a coordinated two-phase change. safe add nullable column widen int → bigint add table to publication consumer first drop unused column add NOT NULL column change default two-phase only rename column narrow a type change primary key The rule: a change is safe when an unchanged consumer still produces a correct document. Everything else is a deploy ordering problem, and the ordering differs per category. Encode the category in the migration's review checklist, not in tribal memory.
Three categories, three procedures. Most incidents come from treating a right-hand-column change as if it were a left-hand-column one.

Solution Steps

1. Parse the envelope defensively

Read fields by name with explicit defaults, and never assume the set of keys. A consumer written this way survives every “safe” change without a deploy.

# transform.py — tolerant of unknown fields, explicit about required ones
REQUIRED = ("id",)

def to_document(after: dict) -> dict:
    missing = [f for f in REQUIRED if f not in after]
    if missing:
        raise NonRetryable(f"required source field(s) absent: {missing}")
    return {
        # Accept either spelling during a rename window; drop the old one afterwards.
        "title": after.get("title") or after.get("product_title"),
        "brand": after.get("brand"),
        "price": float(after["price"]) if after.get("price") is not None else None,
        # Unknown fields are ignored by construction — no schema coupling.
    }

2. Enforce compatibility at the registry, not in review

A registry configured for backward compatibility rejects an incompatible schema at produce time, which converts a production incident into a failed migration in CI.

# Require BACKWARD compatibility for the products value schema.
curl -s -X PUT 'http://localhost:8081/config/shop.products-value' \
  -H 'Content-Type: application/vnd.schemaregistry.v1+json' \
  -d '{"compatibility": "BACKWARD"}'

# CI check: does the proposed schema pass against the registered history?
curl -s -X POST 'http://localhost:8081/compatibility/subjects/shop.products-value/versions/latest' \
  -H 'Content-Type: application/vnd.schemaregistry.v1+json' \
  --data-binary @proposed-schema.json | jq
# => {"is_compatible": false}    <- fails the build, not production

3. Dead-letter instead of crashing

An event the consumer genuinely cannot handle must not stop the partition. Route it, record why, and keep going.

# consume.py — one poison event must not halt every other document
def handle(event, sink, dlq):
    try:
        doc = to_document(event["payload"]["after"])
    except NonRetryable as e:
        dlq.send({"offset": event["offset"], "reason": str(e), "raw": event})
        metrics.incr("cdc.dlq", tags={"reason": type(e).__name__})
        return                       # partition keeps advancing
    sink.upsert(doc)

4. Execute renames in two phases

A rename is an add plus a drop, and separating them in time removes the outage entirely.

-- Phase 1: add the new column, backfill, write both from the application.
ALTER TABLE products ADD COLUMN product_title TEXT;
UPDATE products SET product_title = title WHERE product_title IS NULL;
-- Deploy the consumer that accepts either spelling (step 1) — it already does.

-- Phase 2, days later, once no event carries the old field:
ALTER TABLE products DROP COLUMN title;

5. Replay from the dead-letter queue after the fix

The events parked during the window are not lost; they are a work queue. Replay them once the consumer understands the new shape.

# replay_dlq.py — re-run parked events through the fixed transform
def replay(dlq, sink, limit=None):
    replayed = failed = 0
    for record in dlq.read(limit=limit):
        try:
            sink.upsert(to_document(record["raw"]["payload"]["after"]))
            replayed += 1
        except NonRetryable:
            failed += 1              # still broken: leave it parked, investigate
    print(f"replayed={replayed} still_failing={failed}")
Two-phase rename with a dual-read window Phase one adds the new column and backfills while the consumer reads either name, and phase two drops the old column after the window closes. phase 1 add + backfill column dual-read window consumer accepts either phase 2 drop old column close the window only when no recent event carries the old field measure it — do not assume it from the deploy time
The dual-read window is what makes a rename boring. Its length is decided by data — the age of the oldest event still in flight — not by the calendar.

Verification

Confirm a newly added column flows through without a consumer deploy:

psql -d shop -c "ALTER TABLE products ADD COLUMN eco_rating TEXT;"
psql -d shop -c "UPDATE products SET eco_rating = 'A' WHERE id = 1001;"
sleep 3
curl -s 'localhost:9200/products/_doc/1001?filter_path=_source'
# => the document updates normally; eco_rating is ignored until the mapping wants it

Confirm a poison event dead-letters rather than stopping the partition:

# Consumer lag must keep falling even while the DLQ counter increments.
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group search-sink \
  | awk 'NR>1 {print $2, $6}'
# PARTITION LAG
# 0 3
# 1 0        <- still draining
curl -s localhost:9102/metrics | grep cdc_dlq_total
# cdc_dlq_total{reason="NonRetryable"} 1

A useful mental model is that the CDC topic is an API whose schema is owned by the database migration, not by the team consuming it. Everything that makes a REST API change safe applies here too — additive changes are cheap, removals need a deprecation window, and renames need both spellings to coexist for a while. The only difference is that nobody thinks of a database migration as an API change, which is exactly why the discipline has to be imported deliberately rather than assumed.

Common Pitfalls

Treating "the application still works" as proof the change is safe

The application reads the column by its new name because it was deployed together with the migration. The CDC consumer was not, and it is a second, independent reader of the same schema. A migration checklist that asks only “does the app work?” will keep producing these incidents; the question to add is “which non-application consumers read this table?”

Setting the registry to NONE compatibility to unblock a deploy

Disabling compatibility checking removes the one automated guard that catches these changes before production, and it is almost always done under time pressure and never reverted. If a genuinely incompatible change is needed, register it as a new subject version with an explicit consumer deploy ordering rather than turning the check off globally.

Replaying the dead-letter queue without fixing the transform first

Replaying parked events through the same broken code re-parks them, doubling the queue and obscuring which failures are new. Fix and deploy, verify against a single parked record, and only then drain the queue — ideally with a limit on the first run so a still-broken transform is caught after ten records rather than ten million.

Operational notes

The practical test for whether your consumer is genuinely tolerant is a fixture-based one: take a captured event, add an unknown field, remove an optional field, and change a numeric type from integer to string, then assert the transform still produces the expected document or dead-letters cleanly. Three tests, written once, catch the overwhelming majority of real-world schema drift before it reaches production, and they document the contract far better than a paragraph in a wiki does.

Give the dead-letter queue an owner and an alert. A DLQ nobody watches is a silent data-loss mechanism that looks like success: the pipeline reports healthy, lag is zero, and documents are quietly missing from search. Alert on any sustained non-zero rate, and include the failure reason as a label so the page tells you whether this is a schema change, a bad row, or a downstream outage.

It also pays to make the consumer’s expectations visible. A short document listing the fields the index actually needs from each captured table — five fields out of a forty-column table, usually — turns “which migrations affect search?” from a research question into a lookup. Most schema changes touch columns nobody downstream reads, and knowing that quickly is what lets the genuinely risky ones get the coordination they deserve.

Finally, make the compatibility category part of the migration template. A pull-request checklist that asks the author to tick “safe / consumer-first / two-phase” for each captured table forces the five seconds of thought that prevents the incident, and it produces a written record of the intent when something does go wrong. Teams that adopt this stop having schema-evolution incidents almost entirely, not because the rule is clever but because the question gets asked at the only moment when acting on the answer is cheap.

Consumer lag and dead-letter rate during a schema break A crashing consumer shows lag climbing to a backlog, while a dead-lettering consumer keeps lag flat and shows a small spike in the dead-letter rate instead. crash on error lag grows without bound dead-letter DLQ +1 lag flat — one document waits, not all
The same schema break, two consumer designs. Dead-lettering converts an outage into a single parked record and an alert.