Tracking Search Latency Percentiles

Average search latency is the least useful number a search system produces: it hides the tail, it blends incomparable query shapes, and it moves for reasons unrelated to performance. This page replaces it with percentile histograms that are segmented, correctly bucketed, and measured at the boundary users actually experience. It sits under observability and SRE for search in search engine selection and architecture.

Prerequisites

  1. A metrics backend that supports histograms — a percentile computed from pre-aggregated averages is not a percentile.
  2. A query-shape label available at the point of measurement, since latency is not comparable across shapes.
  3. Latency budgets per shape, agreed with whoever owns the interface.
  4. Access to the engine’s own reported took value as well as your wall-clock measurement.

Diagnosis: three latencies, and only one is the user’s

Three different numbers get called “search latency” and they differ by an order of magnitude on a typical stack.

The engine’s took value covers query execution inside the search engine and excludes the network, the coordinating overhead outside the timed region, and everything your application does. It is the right number for tuning the engine and the wrong number for describing user experience.

Your server-side wall clock around the search call adds network time to the engine and connection acquisition. It is the number that maps to your service’s own performance work.

The client-observed time adds your application’s own processing, enrichment calls, serialisation and the network hop to the user. It is what the user experiences and it is frequently three to five times the engine’s took.

engine took            18 ms
server wall clock      31 ms      (+13 ms network and pooling)
client observed       214 ms      (+183 ms enrichment, render, network)
# Optimising the engine here would target 8% of what the user feels.

Measuring only the first is how teams spend a quarter tuning an engine that was never the constraint.

Three latency measurements around one search Engine execution time is a small part of server wall clock, which is a small part of what the client observes. engine took 18 ms — tuning target for the engine server wall 31 ms — your service's number client observed 214 ms — what the user experiences Alert on the bottom bar; tune against whichever segment of it the traces say is largest.
Three legitimate measurements at different boundaries. Reporting the top one as "search latency" understates the user's experience by an order of magnitude.

Solution Steps

1. Record a histogram, not a gauge

Percentiles must be computed from a distribution. A metric that reports an average per interval cannot produce a p99 afterwards, no matter what the dashboard offers.

# metrics.py — explicit buckets sized around the budgets, not powers of two
from prometheus_client import Histogram

SEARCH_LATENCY = Histogram(
    "search_request_duration_seconds",
    "End-to-end search latency as observed by the API",
    labelnames=("shape", "index", "cache"),
    buckets=(0.005, 0.010, 0.020, 0.030, 0.050, 0.075, 0.100,
             0.150, 0.250, 0.400, 0.750, 1.500, 5.000),
)

with SEARCH_LATENCY.labels(shape=shape, index=index_alias, cache=cache_state).time():
    results = engine.search(body)

Bucket boundaries decide the resolution of your percentiles. Placing several boundaries either side of each budget — 30 ms for autocomplete, 100 ms for instant results — is what makes a breach visible rather than smeared across a wide bucket.

2. Label by shape and keep the cardinality bounded

# shape.py — a small, closed set of labels
def query_shape(req) -> str:
    if req.is_autocomplete:            return "autocomplete"
    if req.filters and req.aggregations: return "faceted"
    if req.filters:                    return "filtered"
    return "plain"

Four values is right. Labelling by user, query text or tenant produces a cardinality explosion that will eventually take down the metrics backend rather than the search service.

3. Record engine time alongside as a separate metric

ENGINE_TOOK = Histogram(
    "search_engine_took_seconds", "Engine-reported execution time",
    labelnames=("shape",), buckets=SEARCH_LATENCY._upper_bounds[:-1],
)
ENGINE_TOOK.labels(shape=shape).observe(results["took"] / 1000.0)

Two histograms with the same buckets make the gap between them directly readable, which is what turns “search is slow” into “the engine is fine, the enrichment is not”.

4. Query percentiles per shape

# p95 per shape over five minutes
histogram_quantile(0.95,
  sum by (le, shape) (rate(search_request_duration_seconds_bucket[5m])))

# The gap between total and engine time — usually the most informative panel
histogram_quantile(0.95, sum by (le) (rate(search_request_duration_seconds_bucket[5m])))
  -
histogram_quantile(0.95, sum by (le) (rate(search_engine_took_seconds_bucket[5m])))

5. Alert on budget breach, not on a fixed threshold

# rules/search-latency.yml
groups:
  - name: search_latency
    rules:
      - alert: AutocompleteLatencyBudgetBreached
        expr: |
          histogram_quantile(0.95,
            sum by (le) (rate(search_request_duration_seconds_bucket{shape="autocomplete"}[5m])))
          > 0.030
        for: 10m
        labels: { severity: page }
        annotations:
          summary: "autocomplete p95 {{ $value | humanizeDuration }} over the 30 ms budget"

Separate rules per shape, each with its own budget, is the whole point of the segmentation — a single global threshold either ignores autocomplete regressions or alerts constantly on faceted queries.

Verification

Confirm the bucket boundaries give useful resolution around the budget:

curl -s localhost:9102/metrics | grep 'search_request_duration_seconds_bucket{shape="autocomplete"' | head -6
# ...le="0.02"} 41203
# ...le="0.03"} 48891     <- the budget boundary is a real bucket edge
# ...le="0.05"} 49102

Confirm the percentile responds to a deliberately slow request:

python3 -c "import requests,time; requests.get('http://localhost:8080/search?q=test&debug_delay=500')"
sleep 30
curl -s -G localhost:9090/api/v1/query --data-urlencode \
  'query=histogram_quantile(0.99, sum by (le) (rate(search_request_duration_seconds_bucket[1m])))' \
  | jq -r '.data.result[0].value[1]'
# => 0.48...  (the injected delay is visible in p99)

Confirm the gap between total and engine latency is being computed:

python3 report.py --panel latency_gap --days 1
# p95 total 214 ms · p95 engine 18 ms · gap 196 ms
# => the engine is not the constraint

Why the tail matters more here than elsewhere

For most services a p99 regression is a nuisance; for search it is disproportionately expensive, because the slow requests are not random. Latency correlates with query complexity, result-set size and filter selectivity, which means the users experiencing the tail are systematically the ones doing the most work — the shopper with six filters applied, the analyst running a broad query, the customer with the largest catalog.

Those are frequently your most engaged users, and the ones whose experience matters most commercially. A p50 that looks excellent while the p99 sits at two seconds describes a system that is fast for casual browsing and slow for anyone actually trying to accomplish something.

The practical consequence is that latency work should start at the tail rather than the median. Pull the ten slowest requests from the last hour, look at what they have in common, and fix that — which is usually one query shape, one unselective filter, or one enrichment call that fans out. Optimising the median rarely moves the tail, while fixing the tail usually improves the median as a side effect.

python3 slow_queries.py --window 1h --limit 10
# 2.14s  faceted  filters=7  results=48210  agg=brand,category,price,size,colour
# 1.98s  faceted  filters=6  results=51044  agg=brand,category,price,size,colour
# => five aggregations over a large result set; the shape, not the engine
Who experiences the latency tail Median requests are simple browsing queries while the slowest requests come from heavily filtered, engaged sessions. p50 — casual browsing few filters, small result sets p99 — engaged users many filters, large aggregations The tail is not a random sample of your traffic — it is concentrated on the users doing the most. Start latency work from the slowest ten requests, not from the median.
Search latency correlates with user intent, which is why the tail deserves more attention here than in a typical service.

Common Pitfalls

Averaging percentiles across instances

The mean of each instance’s p95 is not the fleet’s p95 and can be far from it. Percentiles must be computed from summed bucket counts across instances, which is what sum by (le) before histogram_quantile does. Dashboards that average a pre-computed percentile are showing a number with no statistical meaning.

Buckets that straddle the budget

With buckets at 10 ms and 100 ms, a p95 anywhere between them is reported by interpolation and can be wrong by tens of milliseconds — enough to hide a breach entirely. Place boundaries deliberately around each budget rather than accepting a default exponential scale.

Alerting on p99 for a low-traffic shape

A shape receiving twenty requests a minute has a p99 determined by a single request, and it will fluctuate wildly for reasons that are not performance. Alert on p95 for low-traffic shapes, or aggregate over a longer window, and reserve p99 alerting for shapes with enough volume to make it stable.

Operational notes

Record whether a response came from a cache as a label rather than excluding cached requests from the metric. Cached and uncached latencies are genuinely different populations, and blending them means the percentile moves whenever the hit rate moves — which happens for reasons entirely unrelated to performance, such as a change in traffic mix or a cache warm-up after a deploy.

Publish the budgets next to the dashboard, and make them the panel titles. “Autocomplete p95 (budget 30 ms)” is self-explanatory to anyone who opens it, while “search latency” invites a discussion about whether the number is good. The budgets are also what make the dashboard usable by people outside the team, which matters during incidents.

Keep one panel showing the gap between client-observed and engine-reported latency permanently visible. It is the single most diagnostic view in search performance work, because it immediately answers the question every latency investigation starts with — is this the engine’s problem or ours? — and that answer routes the investigation to the right team in seconds rather than after an afternoon.

One budget and one alert per query shape Autocomplete, instant results and faceted search have budgets an order of magnitude apart, so each needs its own alert threshold. autocomplete p95 budget 30 ms instant results p95 budget 100 ms faceted page p95 budget 500 ms A single global threshold is either useless for the first column or deafening for the third.
Budgets an order of magnitude apart mean thresholds an order of magnitude apart. Segmenting the metric is what makes alerting possible at all.

Finally, keep a long retention on the histogram data even if raw traces expire quickly. Latency questions are frequently retrospective — “was this slow before the migration?” — and a year of bucket counts costs very little storage while making that question answerable in seconds.