Batch vs Streaming Ingestion for Search Indexing

Modern search architectures require deliberate data routing strategies. The choice between batch and streaming paradigms directly impacts Data Ingestion & Synchronization Pipelines and overall system reliability.

Batch processing prioritizes raw throughput and cost efficiency. Streaming architectures target sub-second index freshness. Production teams must evaluate latency tolerance against infrastructure overhead before committing to a single pattern.

# search-pipeline-config.yaml
ingestion:
 mode: "batch" # or "streaming"
 max_concurrent_workers: 4
 retry_policy: "exponential_backoff"
 index_refresh_interval: "30s"

The diagram below contrasts the two control flows: batch accumulates mutations into scheduled micro-batches, while streaming dispatches each event as it lands.

Batch Ingestion: Throughput-Optimized Indexing

Scheduled bulk loads rely on cron jobs, message queues, or ETL frameworks. This approach aggregates mutations into large payloads. It maximizes indexing throughput while minimizing per-request overhead.

Implementation requires strict partitioning and idempotency controls. You must route documents by primary key ranges to avoid hot partitions. Apply version vectors to prevent stale overwrites during concurrent runs.

1. Partition source datasets by index routing keys.
2. Implement idempotent upserts with version vectors.
3. Tune bulk request payloads to 5-10MB.
4. Schedule off-peak reconciliation windows.

Tradeoffs: Throughput is high. Latency ranges from 5 minutes to 24 hours. Infrastructure costs remain low. Error recovery is simple via job re-runs. Mitigate cluster overload by throttling bulk request rates and monitoring indexing thread pool queue depth to prevent heap exhaustion.

# bulk_indexer.py
from elasticsearch import Elasticsearch

def batch_upsert(client: Elasticsearch, docs: list, chunk_size: int = 2000):
    for i in range(0, len(docs), chunk_size):
        chunk = docs[i:i + chunk_size]
        # _bulk requires alternating action/document pairs
        body = []
        for doc in chunk:
            body.append({"index": {"_index": "products", "_id": doc["id"]}})
            body.append(doc)
        client.bulk(body=body, refresh=False)

Streaming Ingestion: Low-Latency Event Processing

Real-time document updates flow through event logs, Kafka topics, or pub/sub systems. Each mutation triggers an immediate indexing action. This pattern eliminates polling delays entirely.

Production deployments require exactly-once processing guarantees. You must implement watermarking to handle out-of-order events. Map raw payloads directly to flattened index schemas before dispatch.

1. Deploy exactly-once stream processors (Flink/Kafka Streams).
2. Implement watermarking and late-arrival handling.
3. Map event payloads to flattened index schemas.
4. Configure dead-letter queues for malformed payloads.

Tradeoffs: Throughput is medium. Latency stays below 1 second. Infrastructure costs scale linearly. Error recovery requires complex state rollbacks. Teams often pair this with Change Data Capture (CDC) Setup to capture row-level mutations without database polling.

// KafkaStreamsWatermarking.java
StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> source = builder.stream("db-changes");

source
 .groupByKey()
 .windowedBy(TimeWindows.ofSizeWithNoGrace(Duration.ofSeconds(5)))
 .aggregate(
 () -> "",
 (key, value, aggregate) -> value,
 Materialized.as("search-index-buffer")
 );

Hybrid Architectures & Trigger Mechanisms

Production systems rarely rely on a single paradigm. High-velocity deltas require immediate propagation. Historical corrections and schema migrations demand bulk reconciliation.

Route critical mutations through streaming channels for instant visibility. Buffer low-priority updates for nightly consolidation runs. Implement dual-write verification using checksum reconciliation to guarantee consistency.

1. Route high-priority mutations to streaming channels.
2. Buffer low-priority updates for nightly batch consolidation.
3. Implement dual-write verification with checksum reconciliation.
4. Decouple ingestion from normalization via intermediate staging.

Tradeoffs: Throughput remains balanced. Latency becomes tiered based on entity type. Infrastructure costs sit at medium levels. Error recovery segments cleanly by pipeline stage. Integrate Webhook-Driven Sync Patterns for immediate UI feedback while background jobs handle compaction.

{
 "routing_rules": {
 "high_priority": ["inventory_updates", "price_changes"],
 "low_priority": ["metadata_edits", "seo_tags"],
 "fallback": "batch_queue",
 "dead_letter_topic": "index-failures-dlq"
 }
}

Decision Matrix & Production Guardrails

Quantify your selection using index staleness tolerance and query SLAs. Batch favors cost efficiency and simpler failure domains. Streaming favors UX responsiveness and complex state management.

Benchmark against 95th percentile query latency before deployment. Monitor indexing queue depth and segment merge rates continuously. Validate downstream normalization and conflict resolution requirements early.

1. Define maximum acceptable index staleness per entity type.
2. Benchmark 95th percentile query latency under load.
3. Monitor indexing queue depth and segment merge rates.
4. Establish circuit breakers for ingestion spikes.

Tradeoffs: Batch SLA targets 99.9% availability with 5m+ freshness. Streaming SLA targets 99.99% availability with <1s freshness. Hybrid SLA delivers tiered guarantees based on data criticality.

# prometheus_circuit_breaker.yml
groups:
 - name: indexing_guardrails
 rules:
 - alert: HighQueueDepth
 expr: sum(index_queue_depth) > 50000
 for: 2m
 labels:
 severity: critical
 annotations:
 summary: "Indexing queue exceeds safe threshold"
 - alert: MergePressureSpike
 expr: rate(segment_merge_time_total[5m]) > 0.8
 for: 5m
 labels:
 severity: warning

The choice is also a staffing decision as much as an architectural one. A batch pipeline is a scheduled job with a log; when it fails, it fails loudly, and rerunning it is usually safe because it is idempotent by construction. A streaming pipeline is a long-lived distributed system with its own state — offsets, partitions, consumer group membership — and it fails in ways that require someone who understands that state to diagnose. The engineering cost is not in writing the consumer, which is a day’s work; it is in the year of operating it. Teams frequently choose streaming for a freshness requirement that the product could not actually articulate, then spend the following quarter learning consumer-group semantics they did not need.

Conversely, batch has a hard ceiling that arrives without warning: the moment the corpus stops fitting in the window. Because the failure is gradual and the symptom is “the morning is slow”, it tends to be diagnosed as a query-performance problem rather than an ingestion one. Knowing which constraint you are approaching — window duration for batch, consumer lag for streaming — is what lets you migrate deliberately instead of under pressure.

Prerequisites

• A stated freshness budget per index, in seconds, agreed with whoever owns the user-facing surface.
• A source that can either be enumerated cheaply (for batch) or emit ordered changes (for streaming) — not both is fine; neither means neither architecture will work.
• Deterministic document ids, so a batch reload and a streaming write address the same document.
• Measured indexing throughput at your current settings, per bulk indexing throughput tuning.
• A rollback path — normally a previous index kept alive behind an alias.

One practical note before the mechanics: decide which pipeline is authoritative for deletes. In a hybrid setup the batch reload sees the complete key set and can remove orphans, while the stream sees individual delete events. If both act on deletes without coordination, a delete event that arrives during a reload can be undone by the reload’s own write of a row that still existed when the export snapshot was taken. Making the reload authoritative for removals and the stream authoritative for updates avoids the ambiguity entirely.

Step-by-Step: Running Batch and Streaming Together

The hybrid architecture is the one most large catalogs end up with, and it is safe only if the two writers cannot fight over the same document. These four steps make that guarantee structural rather than accidental.

1. Derive document ids in one shared place

Both writers must produce the identical id for the same entity. Put the derivation in a module both import, and never inline it.

# ids.py — the ONLY place a document id is constructed
def document_id(entity_type: str, natural_key: str) -> str:
    # Stable across writers, stable across reruns, independent of feed or event id.
    return f"{entity_type}:{natural_key.strip().lower()}"

Verify: assert the batch and streaming paths agree on a sample, in a test that runs on every commit.

from ids import document_id
assert document_id("product", "SKU-991 ") == document_id("product", "sku-991")
# => passes; whitespace and case cannot fork the id

2. Carry a source version on every write from both paths

With versions attached, arrival order stops mattering: the engine keeps whichever write carries the newer version, whether it came from the nightly reload or from a live event.

# writer.py — shared by the batch job and the stream consumer
def op(entity_type, natural_key, source_version: int, body: dict) -> dict:
    return {
        "_op_type": "index",
        "_index": "products",
        "_id": document_id(entity_type, natural_key),
        "version": source_version,
        "version_type": "external_gte",   # older writes are rejected, not applied
        "_source": body,
    }

Verify: replay a batch document with an old version after a newer streaming write and confirm the index keeps the newer one, as covered in using version numbers to prevent stale writes.

3. Give the batch job its own throughput budget

A reload running at full speed will starve the streaming consumer of write capacity. Cap the batch writer explicitly so freshness does not collapse every night at 02:00.

# ingest-config.yaml
batch:
  payload_bytes: 8388608
  concurrency: 3            # capped: leaves headroom for the stream
  window: "02:00-05:00"     # and confined to a low-traffic window
stream:
  payload_bytes: 524288     # small payloads: latency, not throughput
  flush_interval_ms: 500
  concurrency: 2            # partitioned by hash(document_id)

Verify: watch consumer lag during a reload; it may rise, but it must recover within one flush cycle after the batch job ends.

kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --describe --group search-stream | awk 'NR>1 {s+=$6} END {print "total lag:", s}'
# During the reload window: elevated but bounded. After: back to single digits.

4. Reconcile after every reload

The reload is also your reconciliation opportunity: anything present in the index but absent from the source is an orphan, and the reload is the one moment you have a complete key set in hand.

# reconcile.py — delete orphans left by missed deletes
def reconcile(es, source_keys: set[str], index="products", batch=1000):
    orphans, buf = 0, []
    for hit in scan(es, index=index, query={"query": {"match_all": {}}},
                    _source=False, size=batch):
        if hit["_id"] not in source_keys:
            buf.append({"_op_type": "delete", "_index": index, "_id": hit["_id"]})
            orphans += 1
        if len(buf) >= batch:
            bulk(es, buf); buf = []
    if buf:
        bulk(es, buf)
    return orphans

Verify: the orphan count should be small and stable. A growing count between runs means the incremental path is losing deletes and needs the structural fix, not a nightly cleanup.

python3 reconcile.py --dry-run | tail -1
# => orphans=14 (0.000007% of corpus) — stable across the last 30 runs

Configuration Reference

The settings below are the ones that actually change behaviour when moving between batch and streaming shapes. Everything else is downstream of these five.

The interaction that catches people is between max_in_flight_requests and ordering. A batch job can safely run sixteen concurrent writers because it is loading disjoint key ranges. A streaming consumer cannot, because two updates to the same document may be in flight simultaneously and land out of order — unless work is partitioned by a hash of the document id, or every write carries a source version so the engine rejects the stale one.

Failure Modes & Debugging

curl -s 'localhost:9200/_cat/tasks?actions=*reindex*&v&h=action,running_time'

Performance & Scale Notes

Measured on a three-node cluster with 8 vCPU per node, 2 KB documents, five shards, replicas restored after load:

• Batch reload sustains roughly 28,000 documents per second with refresh disabled and replicas at zero — about 100 million documents per hour, so a 400-million-document corpus needs a four-hour window at full utilisation and a six-hour window in practice.
• Streaming sustains roughly 12,000 documents per second at a 500 ms flush interval, because small payloads amortise per-request overhead poorly. Dropping the interval to 100 ms costs a further 35% of throughput and buys 400 ms of freshness — rarely a trade worth making.
• Hybrid runs both, and the binding constraint becomes disk bandwidth for merges rather than either writer. Expect the streaming path’s effective rate to fall by 30–50% while a batch reload is running, which is exactly why the reload belongs in a low-traffic window.
• Consumer parallelism scales linearly up to the partition count and not one step beyond it. A topic with six partitions cannot use a seventh consumer, and the eighth through sixteenth simply idle — a surprisingly common cause of “we scaled up and nothing improved”.
• Recovery rate is the number nobody measures until they need it. With 30% headroom, a one-hour outage on a 12,000 per second stream leaves a 43-million-document backlog that takes about three hours to drain. Halving the headroom roughly doubles that.

Benchmark methodology matters as much as the numbers: run each configuration for at least thirty minutes, discard the first two, hold corpus and mapping fixed, and change one variable at a time. Report throughput, p99 write latency, rejection count, and segment count together, because a configuration that wins on throughput while tripling segment count has borrowed against future query latency rather than earned anything.

Related

• Change Data Capture (CDC) Setup — the log-tailing source that feeds most streaming pipelines.
• Webhook-Driven Sync Patterns — event triggers for the low-latency lane of a hybrid design.
• Conflict Resolution Strategies — reconcile the dual-write paths a hybrid architecture introduces.
• Data Normalization & Cleaning — the staging step both paradigms share before indexing.
• Observability & SRE for Search — alert on queue depth and ingestion lag across either mode.