Handling 429 Rejections with Backpressure

429 Too Many Requests from a bulk endpoint is not an error to log and move past — it is the engine telling you its write queue is full and that every additional request you send makes the situation worse. The failure mode this page fixes is the retry storm: a fixed-delay retry loop that re-sends rejected batches in lockstep, keeps the queue permanently saturated, and converts a five-second capacity dip into a twenty-minute outage. The fix is real backpressure that propagates all the way back to the producer. This sits under bulk indexing throughput tuning, part of the data ingestion and synchronization pipelines area.

Prerequisites

  1. A bulk writer that already batches by payload size and runs at a measured operating point.
  2. Visibility of thread_pool.write.rejected and indexing_pressure counters on every data node.
  3. A producer you can actually slow down — a queue consumer, a paginated reader, or a stream with a pull model.
  4. Somewhere durable to park work when the sink stays unavailable: a Kafka topic, an SQS queue, or local disk.

Diagnosis: why fixed retries make rejection worse

A rejection means the bounded write queue on a shard’s node was full at arrival time. The queue drains at whatever rate the shard can index; it does not drain faster because clients are waiting. So a retry that fires after a fixed 200 ms hits a queue that has barely moved, and because every worker retries on the same schedule, the retries arrive synchronized — the classic thundering herd. Effective throughput collapses even though the engine is still indexing at full speed.

A real incident log makes the pattern obvious: rejections climb while successful writes fall, and the retry count climbs faster than either.

t+00s  bulk_ok=412  bulk_429=0    inflight=8   avg_latency_ms=180
t+30s  bulk_ok=388  bulk_429=24   inflight=8   avg_latency_ms=340
t+60s  bulk_ok=210  bulk_429=190  inflight=8   avg_latency_ms=1120
t+90s  bulk_ok=96   bulk_429=402  inflight=8   avg_latency_ms=2600
#                   ^^^ 4x the successful writes are now retries of the same batches

Nothing about the client changed between t+00s and t+90s. The queue crossed a threshold, the fixed retry loop began amplifying, and the pipeline entered a stable-but-terrible equilibrium where most work is re-sending payloads that will be rejected again.

Retry storm feedback loop versus a backpressure loop A fixed retry loop feeds rejected work straight back into a full queue, while a backpressure loop reduces concurrency and slows the producer instead. fixed retry — amplifies writer full queue 429 → immediate resend backpressure — absorbs producer writer queue 429 → shrink window, pause reads The difference is where the signal stops. Retries absorb it at the writer; backpressure passes it up to whoever is producing work.
A retry loop that never reaches the producer is not backpressure — it just relocates the queue into your client's memory.

Solution Steps

1. Retry with full jitter, never a fixed delay

Exponential backoff alone still synchronizes retries; full jitter spreads them across the whole window and is the difference between recovery and oscillation.

# backoff.py — full jitter (AWS architecture blog formulation)
import random, time

BASE, CAP = 0.25, 30.0

def sleep_for(attempt: int) -> float:
    ceiling = min(CAP, BASE * (2 ** attempt))
    return random.uniform(0, ceiling)      # full jitter: uniform over [0, ceiling]

def with_retry(send, payload, max_attempts=8):
    for attempt in range(max_attempts):
        status = send(payload)
        if status != 429:
            return status
        time.sleep(sleep_for(attempt))
    raise RuntimeError("sink still rejecting after full backoff — escalate to the buffer")

2. Make concurrency adaptive (AIMD)

A fixed concurrency cannot be right at all times, because capacity moves with merges, queries, and node health. Additive-increase/multiplicative-decrease finds the current ceiling continuously and is the same control law TCP uses.

# adaptive_limit.py — grow slowly on success, halve immediately on rejection
class AdaptiveLimit:
    def __init__(self, start=4, floor=1, ceiling=32):
        self.limit, self.floor, self.ceiling = start, floor, ceiling
        self._ok_streak = 0

    def on_success(self):
        self._ok_streak += 1
        if self._ok_streak >= self.limit:        # one full window of clean sends
            self.limit = min(self.ceiling, self.limit + 1)   # additive increase
            self._ok_streak = 0

    def on_reject(self):
        self.limit = max(self.floor, self.limit // 2)        # multiplicative decrease
        self._ok_streak = 0

The sawtooth this produces is the point, not a defect. Concurrency climbs until it finds the current ceiling, backs off hard the moment the ceiling moves down, and climbs again — so the pipeline tracks capacity instead of assuming it. A fixed setting can only be right for one cluster state; AIMD is right for all of them, at the cost of spending a small fraction of time slightly below the true ceiling. Choose the multiplicative factor conservatively: halving is aggressive enough to clear a saturated queue in one step, whereas a gentle 0.9 factor takes so many rejections to make a difference that the queue never drains.

Adaptive concurrency sawtooth against the moving capacity ceiling Concurrency rises one step at a time, halves on each rejection, and tracks a capacity ceiling that drops during a merge storm and recovers afterwards. 16 8 1 time → capacity ceiling merge storm — ceiling drops, limiter follows recovery
Concurrency (solid) chasing capacity (dashed). The gap between the two is the throughput you trade for never overwhelming the sink.

3. Trip a circuit breaker on sustained rejection

Backoff handles seconds of pressure. Minutes of it mean something is broken — a node down, a merge storm, a disk watermark — and hammering it is pointless. Open the circuit, stop sending, and probe periodically.

# breaker.py — closed → open → half-open
import time

class Breaker:
    def __init__(self, threshold=20, cooldown=60.0):
        self.threshold, self.cooldown = threshold, cooldown
        self.failures, self.opened_at = 0, None

    def allow(self) -> bool:
        if self.opened_at is None:
            return True
        if time.monotonic() - self.opened_at >= self.cooldown:
            self.opened_at = None      # half-open: let exactly one probe through
            self.failures = self.threshold - 1
            return True
        return False

    def record(self, rejected: bool):
        if rejected:
            self.failures += 1
            if self.failures >= self.threshold:
                self.opened_at = time.monotonic()
        else:
            self.failures = 0

4. Propagate the signal to the producer

This is the step teams skip. If the reader keeps pulling rows while the writer is blocked, memory grows until the process dies. Gate the producer on the same limiter that governs the writer.

# pipeline.py — the producer cannot outrun the sink
import queue, threading

work = queue.Queue(maxsize=64)          # bounded: put() blocks when the writer falls behind

def produce(rows):
    for batch in rows:
        work.put(batch)                 # backpressure reaches the reader here, for free

def consume(limiter, breaker, send):
    while True:
        batch = work.get()
        if not breaker.allow():
            park_durably(batch)         # circuit open: persist instead of buffering in RAM
            continue
        status = with_retry(send, batch)
        limiter.on_reject() if status == 429 else limiter.on_success()

5. Park overflow durably, not in memory

When the circuit stays open, work must go somewhere that survives a restart. A dead-letter topic or an on-disk spool is the difference between “ingestion paused” and “we lost four hours of updates”.

def park_durably(batch, path="/var/spool/index-backlog"):
    # Append-only NDJSON spool; a separate drainer replays it when the breaker closes.
    with open(f"{path}/{int(time.time())}-{id(batch)}.ndjson", "w") as fh:
        fh.write(batch)
Circuit breaker state transitions Closed transitions to open after repeated rejections, open transitions to half-open after a cooldown, and half-open returns to closed on a successful probe or back to open on failure. closed sending normally open parking to spool half-open one probe request 20 rejects 60 s cooldown probe succeeds → closed (probe fails → open again)
The half-open state is what keeps recovery cheap: one probe decides whether to resume, instead of the whole fleet retrying at once.

Verification

Force rejections in a controlled way by shrinking the write queue on a test node, then confirm the client slows down instead of erroring out:

# Make rejections easy to trigger on a disposable test node.
curl -s -X PUT 'localhost:9200/_cluster/settings' -H 'Content-Type: application/json' \
  -d '{"persistent":{"indexing_pressure.memory.limit":"1%"}}'
# Expected client log — the limiter halves and recovers, no exception is raised:
WARN  bulk rejected (429) — limit 8 -> 4, sleeping 0.41s
WARN  bulk rejected (429) — limit 4 -> 2, sleeping 1.87s
INFO  bulk ok — limit 2 -> 3
INFO  bulk ok — limit 3 -> 4

Confirm the producer actually stalled rather than buffering without bound:

# Queue depth must stay at its bound; RSS must stay flat during the rejection window.
ps -o rss= -p "$(pgrep -f ingest_worker)" | awk '{print $1/1024 " MB"}'
# Expected: within a few MB of the pre-incident baseline, not climbing

Common Pitfalls

Retrying non-retryable failures forever

Only 429 and 5xx deserve a retry. A 400 from a mapping conflict will fail identically on every attempt, so retrying it burns the backoff budget that a genuine capacity dip needs. Classify per item — a bulk response mixes both — and send parse failures straight to the dead-letter path.

Raising `queue_size` to make the rejections stop

A bigger queue does make the 429s disappear, which is exactly the problem: the work now waits invisibly on the server instead of being signalled back to you, and p99 write latency grows without bound. Rejections are the feedback channel — removing them removes your only early warning.

Backing off the writer while the reader races ahead

If the producer is not gated on the same limiter, every second of backoff accumulates unsent batches in process memory. The pipeline survives the rejection window and then dies of an out-of-memory error minutes later, which reads like an unrelated crash in the postmortem. A bounded queue between reader and writer fixes this in one line.

Operational notes

Backpressure changes what “healthy” looks like on a dashboard, so update the alerts alongside the code. A small, steady rejection rate with a limiter that recovers within seconds is normal — it means the client is tracking capacity closely. What deserves a page is the derivative: rejections rising for more than a few minutes, the adaptive limit pinned at its floor, or the durable spool growing without draining. Alerting on raw rejection count produces noise; alerting on “limit has been at floor for ten minutes” produces exactly one page per real incident.

Capacity planning also shifts. Once the writer adapts, the pipeline’s throughput is set by the sink rather than by the client configuration, so the meaningful capacity number becomes the drain rate of the spool after an outage. If the sink sustains 20,000 documents per second and an incident parks two hours of a 15,000-per-second stream, the backlog is 108 million documents and needs roughly an hour and a half at full recovery rate — during which the index is stale and users see it. That recovery arithmetic, not the steady-state rate, is what should drive the decision about how much headroom to keep.