Sizing Shards and Replicas for a New Index

Shard count is fixed at index creation and changing it means rebuilding, which makes it the highest-stakes number in a new index and the one most often copied from an example. This page derives it from two measurements — bytes per document and projected document count — and sizes replicas from read throughput and failure tolerance rather than from habit. It sits under Elasticsearch fundamentals for engineers in search engine selection and architecture.

Prerequisites

  1. A representative document sample — at least 50,000 records with the production mapping applied.
  2. A document-count projection for 12 and 36 months, from whoever owns the data.
  3. Node specifications: available RAM per data node, and therefore the JVM heap that follows from it.
  4. A read-throughput estimate at peak, and an answer to how many node failures the index must survive.

Diagnosis: the two failure directions

Under-sharding produces shards that grow past the point where merges complete comfortably and recovery takes hours; the practical ceiling for search workloads is 30–50 GB per shard. Over-sharding is the more common error and its costs are subtler: every query fans out to every shard, every refresh runs per shard whether or not it has work, and each shard carries fixed cluster-state overhead replicated to every node.

The overhead is easy to see once the numbers are laid out:

10M documents, 2.1 KB each on disk = 21 GB primary data

1 shard    21 GB/shard   1 fan-out per query    minimal state
5 shards    4.2 GB/shard  5 fan-outs per query   5× refresh work for the same writes
20 shards   1.05 GB/shard 20 fan-outs per query  most refreshes produce empty segments
# The default of 5 in many examples is 5× the necessary overhead for this corpus.

The right answer for that corpus is one shard today and two if the projection says it will triple, which is a very different configuration from what a tutorial suggests.

Cost of under-sharding and over-sharding Too few shards produce slow merges and long recovery, too many multiply query fan-out and refresh overhead, and the workable band sits between them. over 50 GB/shard slow merges hours to recover 20–40 GB/shard merges keep up recovery in minutes under 5 GB/shard fan-out on every query refresh cost per empty shard the right-hand error is far more common and its cost is paid on every single query forever, because shard count cannot be reduced without a rebuild
Both extremes hurt, but only one of them is the default in most examples. Over-sharding is the error to guard against.

Solution Steps

1. Measure bytes per document with the real mapping

Estimating from JSON size is unreliable — the index stores postings, doc values and _source, and the ratio to raw JSON varies from 0.6× to 3× depending on the mapping.

# Index a sample with the production mapping, then read the actual size.
curl -s -X PUT 'localhost:9200/sizing_sample' -H 'Content-Type: application/json' -d @mapping.json
./bulk_load.sh sizing_sample sample-50k.ndjson
curl -s -X POST 'localhost:9200/sizing_sample/_forcemerge?max_num_segments=1' > /dev/null
curl -s 'localhost:9200/_cat/indices/sizing_sample?v&h=docs.count,pri.store.size'
# docs.count pri.store.size
# 50000      106.4mb            -> 2.18 KB per document

2. Project forward and divide

bytes/doc                     2.18 KB
documents at 36 months        45,000,000
projected primary size        98 GB
target shard size             30 GB
shards = ceil(98 / 30)        4

Size for the 36-month projection rather than today’s corpus, because adding shards later means a rebuild. Sizing for three years costs a little over-sharding early and avoids the migration entirely.

3. Check the shard count against heap

A separate constraint applies at the node level: roughly 20 shards per GB of JVM heap across everything a node holds. With a 31 GB heap that is about 620 shards per node, which sounds generous until an index-per-day pattern is added to the same cluster.

curl -s 'localhost:9200/_cat/nodes?v&h=name,heap.max,shards' 
# name       heap.max shards
# es-data-1    30.9gb    184     -> comfortable

4. Size replicas from reads and failure tolerance

Replicas serve reads and provide failover, and they multiply write cost. One replica is the minimum for any index whose loss would matter; more is a read-throughput decision.

replicas = max(failure_tolerance, ceil(peak_read_qps / per_copy_qps) - 1)

peak read           2,400 qps
per copy sustains     900 qps
by throughput       ceil(2400/900) - 1 = 2
by failure tolerance                    1
=> 2 replicas

5. Create the index with both numbers stated

curl -s -X PUT 'localhost:9200/products_v1' -H 'Content-Type: application/json' -d '{
  "settings": {
    "number_of_shards": 4,
    "number_of_replicas": 2,
    "refresh_interval": "5s"
  },
  "aliases": { "products": {} }
}'

Creating the alias in the same call is what makes every future sizing change a swap rather than a migration, per zero-downtime mapping migrations with aliases.

6. Decide what happens when the projection is wrong

Projections are wrong routinely, in both directions, and the plan for that is part of the sizing decision rather than a separate concern. If the corpus grows faster than expected, shards exceed the target size and the fix is a rebuild at a higher shard count — the same alias-swap procedure, executed under mild time pressure rather than during an incident. If it grows more slowly, the index is over-sharded, which costs a little fan-out and is not worth correcting.

That asymmetry argues for erring slightly on the high side when the projection is uncertain: the cost of being 50% over-sharded is small and continuous, while the cost of being under-sharded is a rebuild. It is one of the few places in search architecture where the conservative choice is genuinely cheap.

Verification

Confirm shards landed at the size you designed for once the corpus is loaded:

curl -s 'localhost:9200/_cat/shards/products_v1?v&h=shard,prirep,docs,store' | head
# shard prirep docs     store
# 0     p      2500431  5.4gb
# 1     p      2499880  5.4gb
# => on track for ~30 GB at the 36-month projection

Confirm the shards are distributed across nodes rather than concentrated:

curl -s 'localhost:9200/_cat/shards/products_v1?h=shard,prirep,node' | sort -k3 | uniq -c -f2 | head
# => each node holds a comparable number; a skew means allocation awareness needs setting

Confirm the search cluster is green, which proves replicas are actually allocated:

curl -s 'localhost:9200/_cluster/health/products_v1' | jq '{status, active_shards, unassigned_shards}'
# => { "status": "green", "active_shards": 12, "unassigned_shards": 0 }

Routing changes the calculation

One option is worth knowing before settling on a shard count, because it changes the arithmetic completely for a specific shape of workload. If every query is scoped to one tenant, one customer, or one region, routing documents by that key sends each query to a single shard rather than fanning out to all of them.

The effect is substantial: a query that would have touched eight shards touches one, which removes the fan-out cost and lets you use a higher shard count without paying for it on every request. The cost is that shard sizes now depend on how evenly your key distributes — one very large tenant produces one very large shard, and the search cluster cannot rebalance it.

# Index and query with an explicit routing key.
curl -s -X POST 'localhost:9200/products/_doc?routing=tenant-42' \
  -H 'Content-Type: application/json' -d '{"tenant_id":"tenant-42","title":"Trail Runner"}'
curl -s 'localhost:9200/products/_search?routing=tenant-42' -H 'Content-Type: application/json' \
  -d '{"query":{"match":{"title":"trail"}}}'

Use it when the tenant distribution is reasonably even and every query carries the key. Avoid it when a handful of tenants dominate the corpus, because the resulting shard skew is worse than the fan-out it saves.

Query fan-out with and without routing Without routing a query touches every shard, while a routed query touches one, at the cost of uneven shard sizes when the key is skewed. no routing 4 shards hit routed 1 shard hit The saving is real and so is the risk: a dominant tenant produces a shard the search cluster cannot rebalance.
Routing trades even shard sizes for single-shard queries. It is excellent with an even key distribution and painful with a skewed one.

Common Pitfalls

Requesting more replicas than nodes

A replica cannot be allocated to the node holding its primary, so an index with two replicas needs at least three data nodes. Configure more and the surplus copies stay unassigned and the search cluster sits yellow indefinitely — which teams then learn to ignore, defeating the point of the health signal.

Sizing from raw JSON bytes

The stored index is not the JSON. A mapping with many analysed fields and doc values can exceed the source size; one with index: false on large fields can be well under it. The difference is easily 3× in either direction, which is the difference between one shard and four. Measure with the real mapping.

Ignoring the total shard count on the search cluster

A well-sized index on a search cluster already carrying thousands of shards from a daily-index pattern still degrades the search cluster, because cluster state is shared. Check the node-level shard count before adding an index, not just the index’s own arithmetic.

Operational notes

Record the sizing calculation next to the index definition, including the measured bytes per document and the projection it assumed. When the index needs revisiting in two years, the question is whether the projection held — and that is answerable in seconds with the numbers written down and requires re-deriving everything without them.

Revisit sizing when the corpus crosses roughly double the projected size, not on a schedule. Growth is the only thing that invalidates the calculation, and tying the review to a measured threshold rather than a date means it happens when it matters and not otherwise.

Inputs to the shard and replica calculation Measured bytes per document and a growth projection determine shard count, while read throughput and failure tolerance determine replicas. measured bytes/doc × projected count → shard count read throughput + failure tolerance → replica count both written down next to the index definition so the next engineer can check the assumption rather than re-derive it
Two independent calculations from four inputs. Recording the inputs is what makes the numbers reviewable rather than arbitrary.

The one thing worth avoiding entirely is choosing a number because an example used it. Five shards was a default in older versions and appears in a great deal of documentation; on a corpus of a few million documents it is four shards more than necessary, forever.