Backfilling a Search Index Without Downtime

Every index eventually needs a full rebuild: an analyzer change, a new field that cannot be added in place, a corruption you cannot reconcile. The naive procedure — stop writes, delete, reindex, resume — costs an outage measured in hours. The procedure here rebuilds into a shadow index while the live one keeps serving, replays everything that changed during the rebuild, and swaps atomically. It builds on batch versus streaming ingestion in the data ingestion and synchronization pipelines area.

Prerequisites

  1. Every read path already goes through an alias, never a concrete index name. If not, fix that first — it is the whole mechanism.
  2. A change stream or a reliable updated_at column that lets you enumerate everything modified after a timestamp.
  3. Disk headroom for two full copies of the index simultaneously.
  4. Deterministic document ids derived from the source record, so a replay overwrites rather than duplicates.

Diagnosis: why “reindex then swap” is not enough on its own

Reindexing a 200-million-document index takes hours. During those hours the source keeps changing, so the moment the copy finishes it is already stale by however long the rebuild took. Swapping to it silently reverts every change made during the window — an update applied at hour two is present in the live index and missing from the new one.

The staleness is easy to miss because the counts look right:

live index  products_v3   198,441,220 docs   (serving)
new index   products_v4   198,441,220 docs   (rebuild complete)
# Identical counts, yet 41,882 documents differ: everything updated during the 4h rebuild.

The fix is not a faster rebuild; it is a rebuild that overlaps with continuous replay of the changes it is missing, so the gap converges to seconds before you cut over.

Shadow index build with concurrent change replay Reads go to an alias pointing at the live index while a bulk backfill and a live change stream both write into a shadow index, followed by an atomic alias swap. read traffic uses alias only alias products products_v3 live, untouched source of truth database bulk backfill change stream products_v4 shadow, filling atomic alias swap one API call reversible
Two writers fill the shadow index concurrently: a bounded backfill of historical rows and an unbounded replay of live changes. The swap happens only when the second has caught up.

Solution Steps

1. Create the new index with the target mapping

Build it configured for write throughput, not for serving — refresh off, replicas zero. Those settings are restored before the swap.

curl -s -X PUT 'localhost:9200/products_v4' -H 'Content-Type: application/json' -d '{
  "settings": {"number_of_shards": 6, "number_of_replicas": 0, "refresh_interval": "-1"},
  "mappings": {"properties": {
    "title":  {"type": "text", "analyzer": "product_analyzer"},
    "brand":  {"type": "keyword"},
    "price":  {"type": "scaled_float", "scaling_factor": 100},
    "src_version": {"type": "long"}
  }}
}'

2. Start the change replay before the backfill

Order matters. Start consuming changes first and buffer them; if the backfill starts first, changes made in the gap between the two are lost by both writers.

# replay.py — start this BEFORE the bulk backfill begins
def start_replay(stream, target="products_v4"):
    """Consume the change stream from 'now' and write into the shadow index.
    Uses external versioning so a replayed change never overwrites a newer one."""
    for change in stream:
        yield {
            "_op_type": "index",
            "_index": target,
            "_id": change["id"],
            "version": change["src_version"],   # monotonic source version
            "version_type": "external_gte",     # reject anything older; tolerate equal
            "_source": transform(change["row"]),
        }

3. Backfill historical rows with the same versioning rule

The backfill and the replay both write every document, potentially in either order. External versioning makes that safe: whichever write carries the newer source version wins regardless of arrival order.

# backfill.py — paginate by primary key, never by OFFSET
def backfill(db, target="products_v4", page=5000):
    last_id = 0
    while True:
        rows = db.query(
            "SELECT * FROM products WHERE id > %s ORDER BY id LIMIT %s", (last_id, page))
        if not rows:
            return
        yield from ({
            "_op_type": "index", "_index": target, "_id": r["id"],
            "version": r["src_version"], "version_type": "external_gte",
            "_source": transform(r),
        } for r in rows)
        last_id = rows[-1]["id"]      # keyset pagination: stable under concurrent writes

4. Wait for the replay lag to converge

Do not cut over on “the backfill finished”. Cut over when the replay consumer’s lag is a few seconds, which is the real measure of how stale the shadow index is.

# Kafka consumer lag for the replay group — the number that gates the swap.
kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --describe --group search-replay-v4 | awk '{print $1, $2, $6}'
# TOPIC            PARTITION  LAG
# shop.products    0          14
# shop.products    1          9        <- seconds of work, safe to proceed

5. Restore serving settings, then swap atomically

Turn refresh and replicas back on, wait for green, and only then move the alias. The alias update is a single atomic action — readers see the old index on one request and the new one on the next, with no window where the alias is missing.

curl -s -X PUT 'localhost:9200/products_v4/_settings' -H 'Content-Type: application/json' \
  -d '{"index":{"refresh_interval":"5s","number_of_replicas":1}}'
curl -s 'localhost:9200/_cluster/health/products_v4?wait_for_status=green&timeout=600s' | jq -r .status

# Atomic: remove and add in ONE request, never two.
curl -s -X POST 'localhost:9200/_aliases' -H 'Content-Type: application/json' -d '{
  "actions": [
    {"remove": {"index": "products_v3", "alias": "products"}},
    {"add":    {"index": "products_v4", "alias": "products"}}
  ]
}'

6. Keep the old index until you are certain

Deleting products_v3 immediately removes your rollback. Keep it for a full business cycle; rolling back is the same atomic call with the actions reversed.

# Rollback — identical shape, arguments swapped. Practise this before you need it.
curl -s -X POST 'localhost:9200/_aliases' -H 'Content-Type: application/json' -d '{
  "actions": [
    {"remove": {"index": "products_v4", "alias": "products"}},
    {"add":    {"index": "products_v3", "alias": "products"}}
  ]
}'

Verification

Confirm the alias points where you expect and nowhere else:

curl -s 'localhost:9200/_cat/aliases/products?v&h=alias,index'
# alias    index
# products products_v4

Compare counts and a sampled diff before deleting anything:

for i in products_v3 products_v4; do
  echo -n "$i "; curl -s "localhost:9200/$i/_count" | jq .count
done
# products_v3 198441220
# products_v4 198441220

# Spot-check 100 random ids in both indices; any mismatch aborts the cleanup.
python3 compare_sample.py products_v3 products_v4 --sample 100
# => 100/100 identical (fields: title, brand, price)

Common Pitfalls

Reading through a concrete index name somewhere

One forgotten query that targets products_v3 directly keeps reading the old index after the swap, and the resulting bug — “some pages show stale data” — is maddening to trace. Before starting, grep the codebase for the concrete name and enforce the alias with a naming convention: concrete indices always carry a version suffix, and nothing but the swap script may mention one.

Paginating the backfill with OFFSET

LIMIT ... OFFSET n re-executes the scan for every page and, worse, shifts under concurrent inserts and deletes, so rows are silently skipped or duplicated. Keyset pagination on the primary key is both stable under concurrent writes and constant-cost per page, which matters when the table has hundreds of millions of rows.

Swapping before replica recovery finishes

Moving the alias to an index still at number_of_replicas: 0 means every query hits a single copy, and a node failure during that window takes search down entirely. Wait for green — the wait is minutes, and it is the difference between a routine cutover and an incident caused by the cutover itself.

Operational notes

Before anything else, decide whether the rebuild is actually necessary. A surprising share of “we need to reindex” requests are satisfied by an update-by-query, a runtime field, or a new subfield added to an existing mapping — all of which avoid the whole procedure. Reserve the full rebuild for changes that genuinely cannot be applied in place: a different analyzer on an existing field, a change in shard count, or a field type change. Making that determination first costs an hour and occasionally saves a day.

Rehearse the whole procedure in staging against a realistically sized copy, because the failure modes are all about scale and timing rather than logic. The two numbers to take from the rehearsal are the backfill duration and the peak replay lag, since together they tell you the size of the maintenance envelope and whether the change stream retains enough history to cover it. A Kafka topic with 24-hour retention cannot support a 30-hour backfill: the replay consumer falls off the end of the log and you discover it at hour 25.

Plan disk explicitly. Two copies of a large index plus the merge working set can approach three times the steady-state footprint at the peak, and hitting a disk watermark mid-backfill puts the live index into read-only mode — converting a zero-downtime procedure into an outage caused by the procedure. Check the high watermark against projected usage before starting, and prefer running the backfill against a temporarily expanded volume over discovering the ceiling at 80% completion.

Finally, write the cutover and the rollback as one script with two entry points, so the reverse path is exercised by the same code that performs the forward path. Rollbacks that exist only as a paragraph in a runbook tend not to work when they are finally needed, and the whole point of the alias mechanism is that reversing is as cheap as advancing.

Timeline of a zero-downtime backfill Replay starts first, the backfill runs for four hours, lag converges, settings are restored, and the alias swap happens last. replay starts backfill — 4 hours lag converges alias swap Reads never stop. The only instant anything changes for a user is the swap, and it is atomic.
Replay before backfill, swap after convergence. Reversing the first two steps opens a gap that neither writer covers.

One more habit is worth building: name indices with a monotonically increasing suffix and never reuse one. products_v4 followed by products_v5 reads unambiguously in _cat/indices months later, whereas products_new and products_new2 guarantee that somebody eventually swaps to the wrong one. The alias is the stable public name; the concrete indices are disposable, numbered, and expected to accumulate briefly during a migration.

Disk footprint during a shadow-index rebuild Disk usage grows from one index copy to two plus merge working space at the peak, then falls back after the old index is deleted. before 1.0× peak 2.7× after 1.0× old index + new index + merge working space — check the high watermark before starting
Peak disk is closer to three times steady state than two. Hitting the flood-stage watermark mid-rebuild puts the live index into read-only mode.