Using Version Numbers to Prevent Stale Writes

Two workers pick up two updates for the same product. The one carrying the older value happens to reach the index second, and the search result now shows a price that was correct forty seconds ago. Nothing failed, no error was raised, and the index will stay wrong until the next update happens to arrive. This page eliminates the class of bug with version-guarded writes, so the engine — not your worker pool — decides which update is newer. It sits under conflict resolution strategies in the data ingestion and synchronization pipelines area.

Prerequisites

  1. A monotonically increasing version on the source record — an updated_at in microseconds, a row version column, a CDC log sequence number, or a Kafka offset.
  2. Deterministic document ids so every writer addresses the same document.
  3. A writer you can change to send version metadata alongside each document.
  4. Agreement on what “newer” means when two events share a version — usually “either is fine”, which is what external_gte encodes.

Diagnosis: concurrency reorders, and retries reorder harder

Any writer pool with more than one worker can reorder updates for the same document, because the workers race independently through the network and the queue. Retries make it worse: a request that times out, retries, and succeeds can land after a newer request that succeeded on its first attempt.

The result is invisible in every log — both writes returned 200 — and only shows up as a data discrepancy:

source row 991   price 129.00 -> 119.00 (v18) at 10:04:02.114
source row 991   price 119.00 -> 149.00 (v19) at 10:04:02.630
index write      v19 applied  at 10:04:03.001   price=149.00     <- correct
index write      v18 applied  at 10:04:03.418   price=119.00     <- stale, overwrites v19
# Final index state: 119.00. Final database state: 149.00.

Ordering the workers is not the answer — it throws away the concurrency you built the pool for. The answer is to make every write conditional, so a stale write is rejected by the engine rather than applied by it.

Two concurrent writers with and without a version guard Without a guard the later-arriving older version overwrites the newer one; with a guard the engine rejects the stale write with a conflict. no guard worker A — v19 arrives first worker B — v18 arrives second index holds v18 silently stale external_gte guard worker A — v19 accepted worker B — v18 409 conflict index holds v19 correct, always The guard makes arrival order irrelevant — the version decides, not the network. A 409 here is a success: it means the index already holds newer data.
Same two writes, same race, two outcomes. The only difference is whether the write carried the source version.

Solution Steps

1. Choose a version source that is genuinely monotonic

The version must never go backwards for a given document. Wall-clock timestamps from multiple hosts do not qualify unless clocks are tightly synchronized; a database row version or log position always does.

# version.py — prefer, in order: log position, row version, microsecond timestamp
def version_of(change: dict) -> int:
    src = change["source"]
    if "lsn" in src:                       # Postgres logical replication position
        return int(src["lsn"])
    if "row_version" in change["after"]:   # explicit column maintained by the app
        return int(change["after"]["row_version"])
    return int(src["ts_ms"]) * 1000        # last resort: microsecond-scaled timestamp

2. Send the version with every write

Use external_gte rather than external: replays and at-least-once delivery legitimately re-send the same version, and external rejects an equal version as a conflict.

# writer.py — every index operation carries its source version
def index_op(doc_id: str, source: dict, version: int) -> dict:
    return {
        "_op_type": "index",
        "_index": "products",
        "_id": doc_id,
        "version": version,
        "version_type": "external_gte",   # accept newer or equal, reject older
        "_source": source,
    }
# The same thing over HTTP, for a single document.
curl -s -X PUT 'localhost:9200/products/_doc/991?version=19&version_type=external_gte' \
  -H 'Content-Type: application/json' -d '{"title":"Trail Runner 4","price":149.0}'

3. Treat 409 as an expected outcome, not an error

A version conflict means the index already has newer data. Count it, do not retry it, and never log it at error level — a noisy conflict log trains the team to ignore the one that matters.

# apply.py — classify bulk item results correctly
def classify(item: dict) -> str:
    op = next(iter(item.values()))
    status = op.get("status", 200)
    if status < 300:
        return "applied"
    if status == 409:
        return "superseded"      # expected: a newer version already won
    if status in (429, 502, 503, 504):
        return "retry"
    return "dead_letter"         # 400-class: mapping or parse failure

4. Version deletes as well, so they stay deleted

An unversioned delete followed by a stale update resurrects the document. Deletes carry a version too, and the engine keeps a tombstone long enough to reject late writes.

# The delete carries the version of the change that removed the row.
curl -s -X DELETE 'localhost:9200/products/_doc/991?version=20&version_type=external_gte'

# A stale update arriving afterwards is rejected against the tombstone.
curl -s -X PUT 'localhost:9200/products/_doc/991?version=18&version_type=external_gte' \
  -H 'Content-Type: application/json' -d '{"title":"stale"}' | jq '.status, .error.type'
# => 409
# => "version_conflict_engine_exception"

5. Use sequence-number compare-and-swap for read-modify-write

When the write depends on the current document — incrementing a counter, merging a partial payload — external versioning is not enough. Read _seq_no and _primary_term, then write conditionally on both.

# cas.py — optimistic concurrency for read-modify-write updates
def bump_view_count(es, doc_id: str, attempts: int = 5):
    for _ in range(attempts):
        cur = es.get(index="products", id=doc_id)
        body = cur["_source"] | {"views": cur["_source"].get("views", 0) + 1}
        try:
            return es.index(index="products", id=doc_id, document=body,
                            if_seq_no=cur["_seq_no"],          # CAS on both values
                            if_primary_term=cur["_primary_term"])
        except ConflictError:
            continue                                          # someone else won; re-read
    raise RuntimeError("too much contention on a single document")
Choosing between external versioning and sequence-number compare-and-swap Full-document writes from a source of truth use external versioning; read-modify-write updates use sequence number and primary term compare-and-swap. write replaces the whole document source of truth is elsewhere external_gte with the source version stateless, no read required write depends on current state counters, partial merges if_seq_no + if_primary_term read first, retry on conflict
The two mechanisms answer different questions. External versioning asks "is my data newer?"; compare-and-swap asks "has anything changed since I read?"

Verification

Prove that an older version cannot overwrite a newer one:

curl -s -X PUT 'localhost:9200/products/_doc/t1?version=19&version_type=external_gte' \
  -H 'Content-Type: application/json' -d '{"price":149.0}' | jq -r '.result'
# => created
curl -s -X PUT 'localhost:9200/products/_doc/t1?version=18&version_type=external_gte' \
  -H 'Content-Type: application/json' -d '{"price":119.0}' | jq -r '.error.type'
# => version_conflict_engine_exception
curl -s 'localhost:9200/products/_doc/t1?filter_path=_source.price'
# => {"_source":{"price":149.0}}

Prove that a replayed identical version is accepted rather than rejected:

curl -s -X PUT 'localhost:9200/products/_doc/t1?version=19&version_type=external_gte' \
  -H 'Content-Type: application/json' -d '{"price":149.0}' | jq -r '.result'
# => updated       (external_gte allows equal; plain external would 409 here)

Common Pitfalls

Using `external` where `external_gte` is required

At-least-once pipelines legitimately deliver the same event twice, and with version_type=external the duplicate is a 409. That is harmless if you classify conflicts correctly and disastrous if your error handling retries them, because the retry conflicts again forever. Use external_gte for anything fed by an at-least-once stream, and reserve strict external for sources with exactly-once guarantees you can actually name.

Second-resolution timestamps as versions

Two updates within the same second get the same version, so whichever arrives last wins — which is exactly the bug you set out to fix, now with a false sense of safety. Use microseconds at minimum, and prefer a log position or row version, which is monotonic by construction rather than by clock quality.

Letting the tombstone expire before late writes stop arriving

Deleted-document tombstones are retained for a bounded period (index.gc_deletes, 60 seconds by default). A stale update that arrives after the tombstone is collected is treated as a fresh document and the record reappears. If your pipeline can deliver an event minutes late — and a retry queue can — raise gc_deletes to comfortably exceed the worst-case delivery lag.

Operational notes

When you introduce versioning to an existing pipeline, roll it out in two stages. First send the version metadata while continuing to accept every write, logging what would have been rejected; that log tells you the real reordering rate and surfaces any document whose version source is not actually monotonic. Only once the log is clean should you switch the version type on and start rejecting. Skipping the observation stage tends to reveal a non-monotonic version source the hard way, as a wave of conflicts that block legitimate updates.

Instrument the superseded rate. In a healthy pipeline a small, steady percentage of writes conflict — that is concurrency working as designed — while a sudden spike usually means a replay is in flight or a consumer restarted from an old offset. Because both are normal operations, the metric is more useful as a correlation signal than as an alert: when someone asks why indexing throughput doubled for twenty minutes, a matching spike in supersedes answers the question immediately.

A zero conflict rate deserves as much suspicion as a high one. It usually means version metadata is not being sent at all — the writes are unconditional and the protection you believe you have is not deployed. Add an assertion in the writer’s test suite that every emitted operation carries a version and a version type, because this is the kind of guard that gets dropped in a refactor and is invisible until the data is already wrong.

There is one design consequence worth stating plainly: version-guarded writes make the search index a strict projection of the source, and that is a feature. Any process that wants to modify a document must go through the source of truth and acquire a version, which rules out the well-meaning script that “just patches the index” and drifts silently from the database. When a genuine index-only field is needed — a computed popularity score, a cached embedding — keep it in a separate document or a separate index rather than smuggling it into a projected document, where the next full write will overwrite it.

Interpreting the version-conflict rate Zero conflicts suggests versioning is not deployed, a low steady rate is healthy concurrency, and a spike indicates a replay or an offset reset. exactly zero versioning probably absent low and steady healthy concurrency sudden spike replay or offset reset Use the rate as a correlation signal on dashboards rather than as a page-worthy alert. The only genuinely alarming reading is the leftmost one.
Conflicts are information, not failures. The reading that should worry you is the absence of any.