Relevance Evaluation & Offline Testing

Every other page in ranking algorithms and relevance tuning assumes you can tell whether a change helped. This one builds that capability. Without it, relevance work is a sequence of confident opinions: someone tries a boost, the results “feel better”, and the change ships with no way to defend it six months later or to notice when it stops working. The engineering decision resolved here is what to measure, how to collect the labels that make measurement possible, and where in the development cycle each measurement belongs.

The distinction that organises everything below is between offline measurement, which is cheap and approximate, and online measurement, which is expensive and authoritative. Both are necessary and each is misused when substituted for the other.

Prerequisites

• Query logs with enough history to sample a representative query mix, including the tail.
• Impression and click logging tied to a stable request identifier, so interactions can be attributed to the results that produced them.
• A search API you can call from a test harness with an arbitrary configuration, not just the production one.
• Somewhere to store judgments as versioned data — a file in the repository is entirely adequate to start.

Concept Deep-Dive: what the metrics actually measure

NDCG — normalised discounted cumulative gain — is the standard because it captures two things at once: whether relevant documents appear, and whether they appear near the top. Each result contributes its relevance grade discounted by its position, and the sum is normalised against the best possible ordering so scores are comparable across queries with different numbers of relevant documents. Its weakness is that it needs graded judgments, which are more expensive to collect than binary ones.

MRR — mean reciprocal rank — considers only the position of the first relevant result, which makes it the right metric for known-item search where the user wants one specific thing. On a navigational query, a result at position one scores 1.0 and position five scores 0.2, and everything after the first relevant hit is ignored. Using MRR on an exploratory query set will mislead you, because finding several good results matters there and MRR cannot see them.

Recall@k measures whether the relevant documents are present at all within the top k, ignoring their order. It is the metric for evaluating a retrieval stage that feeds a reranker: if the right document is not in the candidate set, no reranking can recover it, so recall at the candidate depth bounds everything downstream.

The practical arrangement is to track all three on the same judgment set, because they answer different questions and a change that improves one while damaging another is exactly what you want to notice.

Where each measurement belongs

The three measurement stages have different costs and different authority, and using the wrong one at the wrong moment is the usual source of wasted effort.

During development, the offline harness runs on a laptop against a fixed judgment set. It is instant, deterministic, and directionally right — which is exactly what you need while iterating on a boost or a weight. It cannot tell you whether users will prefer the result, and it is not supposed to.

In continuous integration, the same harness runs against a frozen baseline and fails the build on a regression. Its job is not to find improvements but to prevent unnoticed damage, which is a much easier problem and one that automation handles well.

In production, an interleaving experiment or a canary measures real user preference on real traffic. It is slow, expensive in traffic, and the only source of truth about whether users prefer the change. Reserve it for candidates that have already survived the two cheaper stages.

The mistake that costs the most time is skipping the first stage and taking every idea to an online experiment. Online tests are rate-limited by traffic: a product with modest search volume can run perhaps two meaningful experiments a month, so the offline harness is what converts “two ideas a month” into “twenty ideas a month, two of which reach traffic”.

Step-by-Step Implementation

1. Sample queries by stratum, not by volume

A judgment set drawn from the top thousand queries measures the head and tells you nothing about the tail, where most dissatisfaction lives. Stratify: take a proportion from head, torso and tail so each is represented well enough to report separately.

# sample.py — stratified query sample for a judgment set
def stratify(query_counts: dict[str, int], n=600):
    ranked = sorted(query_counts.items(), key=lambda kv: -kv[1])
    head, torso, tail = ranked[:100], ranked[100:2000], ranked[2000:]
    take = lambda pool, k: random.sample(pool, min(k, len(pool)))
    return (take(head, n // 3) + take(torso, n // 3) + take(tail, n // 3))

Verify: the sampled set’s query-length distribution should resemble production. If the sample is dominated by one- and two-word queries while production is not, the strata boundaries need adjusting.

2. Collect graded judgments cheaply

Full manual grading is expensive, so combine sources. Click-derived labels are free and biased; explicit human grading is accurate and slow. Use clicks for breadth and human grading for a smaller, carefully chosen core that anchors the metric.

# judgments.py — combine click-derived and human grades, human wins
def merge(click_grades: dict, human_grades: dict) -> dict:
    merged = dict(click_grades)
    merged.update(human_grades)          # explicit labels override inferred ones
    return merged

Verify: measure agreement between the two sources on their overlap. Low agreement means the click model is wrong for your surface, and the click-derived labels should be reweighted rather than trusted.

3. Score a configuration in a repeatable harness

The harness must be able to score an arbitrary configuration, not just production, or you cannot compare candidates before shipping.

# evaluate.py — one function, any configuration
import math

def dcg(grades: list[float]) -> float:
    return sum(g / math.log2(i + 2) for i, g in enumerate(grades))

def ndcg_at_k(ranked_ids: list[str], grades: dict[str, float], k=10) -> float:
    got = [grades.get(doc_id, 0.0) for doc_id in ranked_ids[:k]]
    ideal = sorted(grades.values(), reverse=True)[:k]
    denom = dcg(ideal)
    return dcg(got) / denom if denom else 0.0

def evaluate(config, judgments, k=10) -> dict:
    scores = []
    for query, grades in judgments.items():
        hits = search(query, config=config, size=k)
        scores.append(ndcg_at_k([h["id"] for h in hits], grades, k))
    return {"ndcg": sum(scores) / len(scores), "n": len(scores)}

Verify: run the harness twice against the same configuration and confirm identical output. Any variation means something non-deterministic — a now-relative boost, a random tie-break — is leaking into the measurement.

4. Gate changes in CI

Once the harness is deterministic, it belongs in the pipeline. A relevance change that lowers NDCG should fail the build in the same way a failing unit test does.

# .github/workflows/relevance.yml
- name: Evaluate relevance
  run: |
    python3 evaluate.py --config config/production.yaml --judgments judgments/v7.jsonl \
      --baseline .relevance-baseline.json --tolerance 0.005

Verify: deliberately introduce a bad boost and confirm CI fails. A gate nobody has seen fail is a gate nobody trusts.

5. Confirm survivors online

Offline metrics filter candidates; they do not prove a win. Anything that passes offline goes to an interleaving experiment or a canary before it reaches everyone, using the procedure in canary deploying relevance models.

Verify: track the correlation between offline deltas and online outcomes over several changes. If offline wins routinely fail online, the judgment set does not represent your traffic and needs rebalancing.

The statistics that stop you shipping noise

Two questions decide whether a measured difference means anything, and both are routinely skipped because the metric produced a number and numbers feel definitive.

The first is variance. Run the same configuration against two disjoint halves of the judgment set and observe the difference between them; that spread is your noise floor, and any candidate whose improvement is smaller than it has not been shown to help. On a 600-query set the floor is typically 0.01–0.02 NDCG, which is larger than many changes teams celebrate.

The second is multiple comparisons. Sweeping twenty parameter combinations and picking the best guarantees that the winner is partly lucky, because the maximum of twenty noisy samples is biased upward. The correction is to hold out a portion of the judgment set, choose on the first portion, and confirm on the second — the same discipline as a train/test split, for the same reason.

Neither of these requires statistical sophistication, and both are the difference between a measurement programme that compounds and one that generates confident noise. A team that ships only changes exceeding the noise floor on a held-out set will improve steadily; a team that ships every positive delta will oscillate around a fixed point while believing it is progressing.

Configuration Reference

The set of parameters above is deliberately small. A measurement system with many configuration options becomes a system whose results depend on how it was configured, which defeats the purpose of having one — the whole value of the harness is that two people running it get the same answer.

Failure Modes & Debugging

Starting from nothing

The most common reason a team has no relevance measurement is that building one looks like a project. It is not, if the first version is deliberately crude.

Take the fifty highest-traffic queries. For each, have one person write down the one or two documents that obviously should rank first — not a graded judgment, just the correct answer. That is a binary judgment set of a hundred pairs and it takes an afternoon. Score MRR against it. That single number already catches the majority of catastrophic regressions: an analyzer change that breaks tokenisation, a boost that promotes noise, a mapping change that stops a field being searched.

From there the set grows in the direction the failures point. When a support ticket describes a bad result, add that query and its correct answer. When a new product category launches, add a few of its queries. Within a quarter the set is a few hundred queries drawn from real dissatisfaction, which is a considerably better sample than any set designed up front.

The alternative — waiting until there is time to build the proper thing with graded judgments and stratified sampling — reliably produces no measurement at all, and the relevance work proceeds on opinion for another year.

Performance & Scale Notes

• Harness runtime scales with the number of queries times query latency. Six hundred queries at 60 ms is under a minute serially and seconds in parallel, which is well within a CI budget.
• Judgment collection is the real cost. Human grading runs at roughly 60–120 pairs per hour depending on domain complexity, so a 2,000-pair core set is a few days of work — worth budgeting explicitly rather than hoping it happens.
• Statistical resolution improves with the square root of the sample size. Distinguishing a 0.01 NDCG difference reliably needs low thousands of queries; distinguishing 0.05 needs only a few hundred.
• Judgment decay is real: a set built against a catalog loses accuracy as products change. Refresh a portion each quarter rather than rebuilding wholesale, and always record which version a score came from.
• Interleaving is more sensitive than A/B testing for ranking comparisons — often by an order of magnitude in required traffic — because both rankings compete within the same result list and population variance is removed.

The most common failure in this area is not a wrong metric but an absent one. A team with a rough judgment set of 300 queries, scored automatically on every change, will out-improve a team with no measurement and better intuitions, because the first team notices regressions and the second ships them.

One last framing worth carrying into every relevance conversation: the judgment set is not a test suite, it is a specification. It states, in data, what your product considers a good answer — and that is a claim about the business, not about the code. Treating it that way changes who is involved in maintaining it, and it is the reason the most effective search teams have a merchandiser or a domain expert contributing labels rather than an engineer guessing at them.

Related

• Ranking algorithms and relevance tuning — the parent area whose every technique depends on the measurement built here.
• Building a judgment list from click logs — how to derive labels from interactions without inheriting position bias.
• Measuring NDCG and MRR in CI — wiring the harness into the pipeline so regressions fail the build.
• Interleaving experiments for ranking changes — the online confirmation step for changes that pass offline.
• Learning-to-rank — the technique that consumes judgments as training data rather than only as a scoreboard.