Forcing a Refresh for Read-Your-Writes Consistency

A user saves a record, the UI navigates to the list, and the record is not there. They save again. Now there are two. This is the read-your-writes gap, and the instinctive fix — ?refresh=true on every write — trades a correctness bug for a throughput bug that shows up weeks later as “indexing got slow.” This page resolves the gap with four techniques ordered by cost, only one of which forces a shard refresh. It sits under index refresh and commit strategies in the data ingestion and synchronization pipelines area.

Prerequisites

  1. An index whose refresh interval is deliberately longer than instant — otherwise there is no gap to close.
  2. The ability to return the written document id (and ideally its sequence number) from your write path to the caller.
  3. A UI or API layer you can change: two of the four techniques are client-side, not index-side.
  4. A clear statement of whose writes must be visible — the author’s own, or everyone’s.

Diagnosis: the gap is per-session, but the fix is usually global

Almost every read-your-writes complaint is about the author seeing their own change. Nobody notices that another user’s edit took four seconds to appear. That asymmetry is the whole opportunity: a per-session guarantee costs almost nothing, while a global guarantee costs a refresh on every write for every user.

The failing sequence looks like this in a request trace, and the 200 on the write is what makes it confusing to debug:

POST /api/products          201  38ms   id=sku-4471   (buffered, not yet in a segment)
GET  /api/products?q=trail  200  22ms   hits=17       (sku-4471 absent — refresh is 4.2s away)
POST /api/products          201  41ms   id=sku-4472   (user retried; duplicate created)

Both writes succeeded and both reads were correct with respect to the index state at the time. Nothing is broken except the contract the interface implied. The engineering question is which of four mechanisms to spend on closing it.

Four read-your-writes techniques ordered by cost Realtime GET costs nothing, session merge costs client work, wait_for adds request latency, and forced refresh adds shard-wide indexing cost. realtime GET cost: none detail view only session merge cost: client state list views wait_for cost: write latency true search need refresh=true cost: shard-wide tests only cheap expensive Work left to right and stop at the first technique that satisfies the surface. Most products never need to go past the second box.
Four mechanisms, one ordering rule. The rightmost option is the one teams reach for first and the one that should almost never ship.

Solution Steps

1. Use a realtime GET when the next screen is a detail view

If the user saves and lands on the record’s own page, no search is involved. The GET API reads from the translog when the document is not yet in a segment, so it is consistent immediately at zero indexing cost.

# after_save.py — the save handler returns the canonical record without any refresh
def save_and_return(es, doc_id: str, source: dict) -> dict:
    es.index(index="products", id=doc_id, document=source)   # no refresh parameter
    # Realtime GET: served from the translog if the doc is not yet searchable.
    return es.get(index="products", id=doc_id)["_source"]

2. Merge the pending write into the client’s list view

When the next screen is a list, hold the just-written document in session state and merge it into the first search response that does not contain it yet. This is a few lines of client code and closes the gap for the only user who can perceive it.

// pendingWrites.js — session-scoped read-your-writes for list views
const pending = new Map();            // id -> { doc, until }

export function notePendingWrite(doc, ttlMs = 15000) {
  pending.set(doc.id, { doc, until: Date.now() + ttlMs });
}

export function mergePending(results) {
  const now = Date.now();
  for (const [id, entry] of pending) {
    if (entry.until < now) { pending.delete(id); continue; }   // the index caught up
    if (results.some((r) => r.id === id)) { pending.delete(id); continue; }
    results = [entry.doc, ...results];                         // surface it optimistically
  }
  return results;
}

3. Use wait_for when the write genuinely must be searchable

Some writes really do need to be in search results for everyone — a moderation action that must remove content, for instance. wait_for blocks that one response until the next scheduled refresh includes the change, adding latency to the writer without adding work to the shard.

# Blocks up to one refresh interval; adds no extra refresh to the shard.
curl -s -X POST 'localhost:9200/products/_doc/sku-4471?refresh=wait_for' \
  -H 'Content-Type: application/json' -d '{"title":"Trail Runner 3","status":"published"}'

4. Wait on the sequence number for cross-service consistency

When a second service must observe the write (an audit consumer, a downstream projection), have it wait for the shard’s global checkpoint to reach the sequence number returned by the write, rather than sleeping.

# wait_for_seq.py — deterministic wait instead of time.sleep(1)
def wait_until_visible(es, index: str, doc_id: str, seq_no: int, primary_term: int,
                       timeout_s: float = 5.0) -> bool:
    """Poll the doc with the write's seq_no; returns as soon as a searcher can see it."""
    deadline = time.monotonic() + timeout_s
    while time.monotonic() < deadline:
        r = es.search(index=index, body={
            "query": {"term": {"_id": doc_id}},
            "seq_no_primary_term": True, "size": 1,
        })
        hits = r["hits"]["hits"]
        if hits and hits[0]["_seq_no"] >= seq_no:
            return True
        time.sleep(0.05)
    return False

5. Reserve refresh=true for tests

In an integration test, forcing a refresh is correct: the suite writes a handful of documents and needs them searchable now, and no production traffic is affected. Encode that boundary so it cannot leak into application code.

# conftest.py — the only place refresh=true is allowed to appear
@pytest.fixture
def indexed(es):
    def _index(index, doc_id, source):
        es.index(index=index, id=doc_id, document=source, refresh=True)  # tests only
    return _index
Sequence of a write followed by a session-merged read The browser writes, holds the document in session state, reads a search result that lacks it, merges it optimistically, and drops the pending copy once the index catches up. browser API index save (no refresh) index → buffer 201 + document search — result set still lacks it merge from session state refresh → now searchable
The client covers the window between acknowledgement and searchability. Once the index catches up, the pending entry is dropped and the two views converge.

Verification

Prove the realtime GET path is genuinely consistent by writing and reading with no refresh anywhere:

curl -s -X POST 'localhost:9200/products/_doc/ryw-1' -H 'Content-Type: application/json' \
  -d '{"title":"Consistency probe"}' > /dev/null
curl -s 'localhost:9200/products/_doc/ryw-1?filter_path=_source.title'
# => {"_source":{"title":"Consistency probe"}}     immediately, every time
curl -s 'localhost:9200/products/_search?q=title:probe&filter_path=hits.total.value'
# => {"hits":{"total":{"value":0}}}                until the next scheduled refresh

Prove wait_for closes the search gap without adding refreshes:

before=$(curl -s 'localhost:9200/products/_stats/refresh' | jq '._all.total.refresh.total')
curl -s -X POST 'localhost:9200/products/_doc/ryw-2?refresh=wait_for' \
  -H 'Content-Type: application/json' -d '{"title":"Waited probe"}' > /dev/null
curl -s 'localhost:9200/products/_search?q=title:waited&filter_path=hits.total.value'
# => {"hits":{"total":{"value":1}}}
after=$(curl -s 'localhost:9200/products/_stats/refresh' | jq '._all.total.refresh.total')
echo "extra refreshes: $((after - before))"
# Expected: 0 or 1 — the scheduled refresh, not one caused by the write

Common Pitfalls

Exceeding the shard's `wait_for` listener limit

A shard parks a bounded number of wait_for requests (1,000 by default). Past that limit it silently forces a refresh instead, so a burst of “cheap” waiting writes turns into exactly the shard-wide refreshes you were avoiding — and the switch is invisible in the response. If a bulk path uses wait_for, cap its concurrency well below the listener limit, or apply the wait to one final marker write instead of every document.

Merging pending writes without a TTL

Session-side merging is safe only while it is temporary. Without an expiry, a document that was deleted or rejected server-side keeps reappearing in the user’s list for the life of the session, which is a worse bug than the one being fixed. Bind the pending entry to a short TTL and drop it the moment a search response contains the document.

Using `refresh=true` in a retry path

Retry loops multiply whatever they wrap. A write handler that forces a refresh and is retried three times performs three shard-wide refreshes for one logical write, and under a partial outage the retry rate is exactly when the search cluster can least afford them. Keep forced refreshes out of anything that can be retried automatically.

Operational notes

Be explicit about which of the four techniques each surface uses, and put a test behind it. An integration test that writes and then immediately searches — with no sleep — is the only thing that keeps a per-surface guarantee honest across refactors, because the failure it catches is timing-dependent and therefore invisible in review. Tests that paper over the gap with a fixed sleep are worse than no test: they pass on a fast machine, fail intermittently in CI, and get quietly lengthened until they no longer assert anything.

Read-your-writes is a contract, so write it down per surface rather than leaving it implicit in whichever technique a given handler happens to use. A short table in the service’s README — surface, guarantee, mechanism — prevents the slow drift where one endpoint uses wait_for, another forces a refresh, and a third does nothing, all for the same underlying index. Drift like that is invisible until an incident, at which point nobody can say what the intended behaviour was.

The measurement that keeps the contract honest is end-to-end, from the write’s acknowledgement to the moment a search containing it returns. Instrument it as a histogram on the write path (record the id, poll once at the p99 target, emit a boolean) or sample it synthetically every minute from a probe. Either way you get a number that tracks the promise the interface makes, rather than the refresh interval, which is only one term in it. When the number regresses, the cause is usually an ingestion backlog rather than the refresh setting — and having the metric is what lets you tell the difference instead of guessing.

End-to-end visibility latency broken into its parts A stacked bar splitting visibility latency into application write time, queue time, indexing time and refresh wait, showing refresh as the smallest component. observed app queue wait indexing refresh Shortening the refresh interval attacks the smallest slice of the total. When visibility regresses, look at queue wait first — it is usually the ingestion backlog. 0 ms total p99
The refresh interval is one term in visibility latency and rarely the dominant one. Measuring end to end stops you optimising the wrong slice.