Index Refresh & Commit Strategies

“The write returned 200 but the document is not in the results” is not a bug — it is the visibility contract of every near-real-time search engine, and the decision this page resolves is how much of that delay your product can tolerate and what it costs to shorten. Refresh controls searchability, commit controls durability, and they are independent knobs that engineers routinely conflate. This is part of the data ingestion and synchronization pipelines area and pairs directly with bulk indexing throughput tuning, because every setting that makes documents visible sooner makes ingestion slower.

Prerequisites

• An index you can change settings on without coordinating a deploy — refresh tuning is iterative.
• A clear product answer to “how stale may a search result be?”, expressed in seconds, per index.
• Indexing throughput measurements from a run at the current settings, so a change can be attributed.
• Access to _stats and _cat/segments for the index, or the equivalent commit metrics on your engine.
• An understanding of your mapping’s field count, because per-refresh cost scales with the number of fields written into each new segment.

Concept Deep-Dive: refresh, flush, and commit are three different events

A refresh takes whatever is in the shard’s in-memory indexing buffer, writes it into a new Lucene segment, and opens a new searcher over the resulting segment set. It is a visibility operation. It does not fsync anything, so a refreshed document is searchable but not yet crash-safe on its own. Refresh cost is roughly proportional to how many segments exist and how many fields each document writes; it is paid per shard, per interval, whether or not anything was written.

A flush takes the translog, fsyncs the segment files, writes a new commit point, and truncates the translog. It is a durability operation. After a flush, recovery starts from the commit point instead of replaying operations. Flushes are triggered by translog size (512 MB by default) or age, and are usually left alone — the setting engineers actually need is translog.durability, which decides whether each individual request fsyncs the translog before returning.

The translog is what makes the gap safe. Between refreshes, documents live in a volatile buffer, but every operation has already been appended to the translog. With durability: request, that append is fsynced before the client sees a 201, so an acknowledged write survives a node crash even though no segment contains it yet. With durability: async, the fsync happens on an interval, and a crash can lose up to that interval’s worth of acknowledged writes.

Where this bites in practice is the read-your-writes case: a user saves a product, the UI immediately searches for it, and the document is not there because the next refresh is 900 ms away. There are three honest fixes — wait for the refresh, force one, or read the document by id instead of searching for it — and exactly one of them is cheap. Forcing a refresh per write is the expensive answer that teams reach for first and regret at scale, because refresh cost is per shard and independent of how many documents you wrote.

Step-by-Step Implementation

1. Set the refresh interval from the product requirement, not the default

The default one second is an interactive-search default. Catalogs refreshed by a nightly job, log indices, and analytics indices are all better served by 30 seconds or more, which cuts segment churn dramatically.

# A catalog whose product team accepts 30s staleness.
curl -s -X PUT 'localhost:9200/products/_settings' \
  -H 'Content-Type: application/json' -d '{"index":{"refresh_interval":"30s"}}'

Verify: confirm the interval is in effect and observe the refresh counter advancing at the new rate.

curl -s 'localhost:9200/products/_stats/refresh?filter_path=_all.total.refresh.total' | jq
# Sample twice, 60s apart: the delta should be ~2 at 30s, ~60 at 1s.

2. Use refresh=wait_for for the write that must be immediately visible

refresh=true forces an immediate shard refresh and is the wrong default: it makes every writer pay full refresh cost. refresh=wait_for blocks the response until the next scheduled refresh includes the document — same guarantee, no extra segments.

# Correct: block this one request until the document is searchable.
curl -s -X POST 'localhost:9200/products/_doc/sku-991?refresh=wait_for' \
  -H 'Content-Type: application/json' -d '{"title":"Trail Runner 3","price":129.0}'

Verify: the search immediately after the write must find the document, with no client-side sleep.

curl -s 'localhost:9200/products/_search?q=_id:sku-991&filter_path=hits.total.value' 
# => {"hits":{"total":{"value":1}}}

3. Bypass search entirely for read-your-writes

If the UI only needs to display the record it just saved, a GET by id is served from the translog and is visible immediately — no refresh involved, and no refresh cost incurred.

# read_after_write.py — realtime GET, not a search
import requests

def read_own_write(index: str, doc_id: str):
    # The GET API is realtime: it falls back to the translog when the doc
    # is not yet in a segment, so it never needs a refresh.
    r = requests.get(f"http://localhost:9200/{index}/_doc/{doc_id}", timeout=5)
    r.raise_for_status()
    return r.json()["_source"]

Verify: write without any refresh parameter and read immediately; the document must come back.

curl -s -X POST 'localhost:9200/products/_doc/sku-992' -H 'Content-Type: application/json' \
  -d '{"title":"Ridge Vest"}' > /dev/null
curl -s 'localhost:9200/products/_doc/sku-992?filter_path=_source.title'
# => {"_source":{"title":"Ridge Vest"}}    (immediately, no refresh)

4. Choose translog durability per index, not per cluster

Durability is a data-loss decision, so make it explicitly. An index rebuilt from an upstream system of record can afford async; an index that is the record cannot.

# Rebuildable index: fsync on a 30s schedule, faster writes.
curl -s -X PUT 'localhost:9200/events/_settings' -H 'Content-Type: application/json' -d '{
  "index": {"translog.durability":"async","translog.sync_interval":"30s"}
}'
# Authoritative index: fsync before every acknowledgement.
curl -s -X PUT 'localhost:9200/orders/_settings' -H 'Content-Type: application/json' -d '{
  "index": {"translog.durability":"request"}
}'

Verify: read both settings back in one call and eyeball the difference.

curl -s 'localhost:9200/events,orders/_settings?filter_path=**.translog' | jq

5. Restore interactive settings after a backfill

Backfills run with refresh disabled. The restore step is the one most often forgotten, and its symptom — “search stopped updating after the migration” — is reported as a search bug days later.

curl -s -X PUT 'localhost:9200/products/_settings' -H 'Content-Type: application/json' \
  -d '{"index":{"refresh_interval":"1s"}}'
curl -s -X POST 'localhost:9200/products/_refresh'   # make the backlog visible now

Verify: assert the interval is not -1 as a post-deploy check in CI, so the omission fails loudly.

test "$(curl -s 'localhost:9200/products/_settings' \
  | jq -r '.products.settings.index.refresh_interval')" != "-1" || exit 1

6. Give each index a visibility budget and enforce it in code

Refresh settings drift because nobody owns them. Encode the intended visibility per index next to the index definition, apply it from that single source, and the “why is this index at -1?” conversation disappears.

# index_policy.py — one declarative place for visibility and durability
POLICIES = {
    "orders":   {"refresh_interval": "1s",  "durability": "request"},  # authoritative
    "products": {"refresh_interval": "5s",  "durability": "request"},
    "reviews":  {"refresh_interval": "30s", "durability": "async"},    # rebuildable
    "events":   {"refresh_interval": "60s", "durability": "async"},
}

def apply(es, index: str):
    p = POLICIES[index]
    es.indices.put_settings(index=index, body={"index": {
        "refresh_interval": p["refresh_interval"],
        "translog.durability": p["durability"],
    }})

Verify: run the applier in CI against the real settings and fail the build on drift, so a manual change made during an incident cannot survive to the next deploy.

python3 -c "import index_policy, json, requests as r; \
  live = r.get('http://localhost:9200/_all/_settings').json(); \
  print(json.dumps({k: v['settings']['index'].get('refresh_interval') for k, v in live.items()}, indent=2))"
# => {"orders": "1s", "products": "5s", "reviews": "30s", "events": "60s"}

Worth stating explicitly, because it causes real confusion during incidents: none of these settings affect whether a write succeeded. A document acknowledged with durability: request is safe even if it is not searchable for another thirty seconds, and a document that is searchable is not necessarily durable if durability is asynchronous. When reconciling after an incident, the question “was this write acknowledged?” is answered by the application log, not by whether search can find the document — and conflating the two sends people looking for data loss that never happened.

Configuration Reference

A last framing that makes the settings easier to explain to non-specialists: refresh is a publishing schedule and commit is a saving schedule. A document that has been saved but not published is safe and invisible; one that has been published but not saved is visible and would be lost in a crash. Almost every confusing conversation about these settings resolves once both parties agree which of the two properties they are actually asking about.

Failure Modes & Debugging

curl -s -X PUT 'localhost:9200/products/_settings' -H 'Content-Type: application/json' \
  -d '{"index":{"refresh_interval":"1s"}}' && curl -s -X POST 'localhost:9200/products/_refresh'

curl -s -X PUT 'localhost:9200/products/_settings' -H 'Content-Type: application/json' \
  -d '{"index":{"refresh_interval":"1s"}}'

One diagnostic shortcut worth internalising: if a document is missing from search but a realtime GET by id returns it, the problem is refresh and nothing else. If the GET also misses, the document never reached the index and the problem is upstream — ingestion, transform, or a dead-letter you have not looked at. That two-command triage separates the two most common classes of “search is missing data” report in under a minute, and it is worth putting at the top of the runbook because it routes the investigation to the right team immediately.

Performance & Scale Notes

Measured on a three-node cluster indexing 2 KB documents into a five-shard index, holding batch size and concurrency constant and varying only the refresh interval:

• 1s → 5s raises sustained indexing throughput by roughly 12–18% and cuts new-segment creation by about 80%.
• 5s → 30s adds a further 6–10%; the curve flattens because per-refresh cost is no longer the dominant term.
• Disabled (-1) adds 20–40% over the one-second baseline, with the largest gains on small documents where per-refresh overhead dominates per-document work.
• refresh=true per request costs 30–80 ms of added write latency per call at typical shard counts and, worse, multiplies segment count — the merge debt it creates outlives the load that created it.
• wait_for adds latency bounded by the refresh interval (average half of it) and adds no shard work at all, which is why it is the correct default for the rare write that must be visible immediately.

Refresh cost scales with shard count, not document count: an index with 30 shards pays 30 refreshes per interval even if only one document was written. This is the strongest practical argument against over-sharding, and it interacts directly with the sizing guidance in Elasticsearch fundamentals for engineers. If your write rate is low and your shard count is high, most refresh work produces empty segments.

For capacity planning, treat visibility latency as a per-index service level rather than a global setting. A realistic split for an e-commerce stack is: orders at one second with durability: request, products at five seconds, reviews at 30 seconds, and analytics indices with refresh disabled outside their load window. Each choice is defensible on its own; a single global default cannot be. Instrument the actual observed lag with the pattern in alerting on indexing lag with SLOs so that a change to any of these settings shows up as a measurable, alertable number rather than an anecdote.

A final scaling consideration is what refresh does to the query side. Every refresh opens a new searcher, which invalidates cached filters and forces the next queries to rebuild them. On an index with expensive filter caches — geo shapes, large terms filters, or the aggregation-heavy queries typical of faceted navigation — a one-second refresh can mean the filter cache is never warm, and query latency reflects that far more than it reflects the indexing cost. When query p95 is inexplicably high on a write-heavy index, lengthening the refresh interval is often a bigger win for readers than for writers, which is the opposite of the intuition most engineers bring to the setting.

The cross-cutting rule that ties all of this together: refresh is a cost paid continuously and a benefit consumed occasionally. Almost every index has a small number of documents whose visibility genuinely matters within a second, and a very large number whose visibility matters within a minute. Rather than paying interactive-grade refresh cost for the whole index, pay it per request with wait_for on the handful of writes that need it. That single inversion — from a global setting to a per-request guarantee — usually removes the tension between freshness and throughput entirely, and it composes cleanly with the operating points established in bulk indexing throughput tuning.

Related

• Bulk indexing throughput tuning — the throughput side of the same settings, including what to relax during a backfill.
• Batch versus streaming ingestion — determines how much visibility delay your pipeline shape can absorb.
• Webhook-driven sync patterns — where read-your-writes expectations most often collide with refresh delay.
• Elasticsearch fundamentals for engineers — shard sizing, which sets the per-refresh cost floor.
• Alerting on indexing lag with SLOs — turns visibility delay into a monitored, budgeted number.