Measuring NDCG and MRR in CI

A relevance regression is the only kind of production defect that ships without failing anything. Tests pass, the build is green, latency is unchanged, and the results are quietly worse. This page closes that gap by making relevance a build-time check: a deterministic scorer, a committed baseline, and a gate that fails when a change makes ranking worse. It belongs to relevance evaluation and offline testing within ranking algorithms and relevance tuning.

Prerequisites

  1. A judgment set committed to the repository as a versioned file, not fetched from a mutable service.
  2. A search instance CI can reach — a container started by the pipeline is ideal, because it removes the shared-environment variability that makes scores drift.
  3. A fixture corpus small enough to index in under a minute and representative enough that scores are meaningful.
  4. Ranking configuration that lives in the repository, so the thing being scored is the thing being reviewed.

Diagnosis: why relevance regressions escape review

A pull request that changes a boost weight, a field mapping, or an analyzer looks like a small diff. Nothing about it signals that it will change what users see for thousands of queries, and no reviewer can predict the effect by reading it. The change ships, and the feedback arrives weeks later as a gradual decline in a business metric that a dozen other things could explain.

The pattern is visible once you look for it in the history:

2026-04-02  bump title boost 4 -> 6          "felt better on a few queries"
2026-04-18  add popularity boost              "merch request"
2026-05-07  switch analyzer to light_stem     "fixes plurals"
2026-05-22  raise max_boost 2.0 -> 4.0        "campaign needed it"
# Net effect on ranking: unknown. Each change was reviewed and none was measured.

Each of those is defensible in isolation. The aggregate is a ranking configuration nobody can reason about, and there is no point in the history where a regression would have been caught. A CI gate turns each of those lines into a measured claim.

Relevance gate in the continuous integration pipeline A pull request starts a container, indexes a fixture corpus, scores the judgment set, and compares against a committed baseline before allowing a merge. pull request config change index fixtures throwaway container score judgments NDCG, MRR, recall compare baseline pass or fail the delta is posted on the pull request, whether it passes or not so reviewers see the ranking effect of a diff they cannot otherwise evaluate
The gate's main value is not blocking bad changes but attaching a measured effect to every ranking diff, which is information reviewers currently do not have.

Solution Steps

1. Make the corpus and the clock deterministic

Scores must depend only on the configuration under test. Index a fixed fixture corpus, and freeze anything time-dependent, because a recency boost relative to now produces a different ranking on every run.

# conftest.py — one fixed corpus, one fixed clock
FROZEN_NOW = "2026-01-15T12:00:00Z"      # every recency boost resolves against this

def build_index(client):
    client.indices.delete(index="ci", ignore=[404])
    client.indices.create(index="ci", body=load_yaml("config/index.yaml"))
    bulk(client, load_ndjson("fixtures/corpus.ndjson"), refresh=True)

2. Score both metrics in one pass

NDCG and MRR answer different questions, and computing them together costs nothing beyond the search you already issued.

# metrics.py — both metrics from one ranked list
import math

def ndcg_at_k(ranked, grades, k=10):
    dcg = lambda gs: sum(g / math.log2(i + 2) for i, g in enumerate(gs))
    got = [grades.get(d, 0.0) for d in ranked[:k]]
    ideal = sorted(grades.values(), reverse=True)[:k]
    return dcg(got) / dcg(ideal) if ideal and dcg(ideal) else 0.0

def reciprocal_rank(ranked, grades, threshold=1.0):
    for i, doc in enumerate(ranked, start=1):
        if grades.get(doc, 0.0) >= threshold:
            return 1.0 / i
    return 0.0

3. Report per stratum, not as one number

A single site-wide average hides the case that matters: a change that improves head queries and damages the tail. Report each stratum separately and gate on all of them.

# report.py — one row per stratum, plus the overall figure
def score_all(judgments, strata):
    out = {}
    for name, queries in strata.items():
        rows = [(ndcg_at_k(search(q), judgments[q]), reciprocal_rank(search(q), judgments[q]))
                for q in queries]
        out[name] = {
            "ndcg": round(sum(r[0] for r in rows) / len(rows), 4),
            "mrr":  round(sum(r[1] for r in rows) / len(rows), 4),
            "n": len(rows),
        }
    return out

4. Commit the baseline and diff against it

The baseline is a file in the repository, updated deliberately by the pull request that improves the metric. That makes every change to expected relevance visible in review, exactly like a snapshot test.

{
  "judgments_version": "v7",
  "head":  { "ndcg": 0.7412, "mrr": 0.8103, "n": 200 },
  "torso": { "ndcg": 0.6188, "mrr": 0.6642, "n": 200 },
  "tail":  { "ndcg": 0.5031, "mrr": 0.5507, "n": 200 }
}

5. Fail the build on regression, with a readable message

# gate.py — exit non-zero with an explanation a reviewer can act on
TOLERANCE = 0.005          # just above measured run-to-run variance

def gate(current, baseline):
    failures = []
    for stratum, base in baseline.items():
        if not isinstance(base, dict):
            continue
        for metric in ("ndcg", "mrr"):
            delta = current[stratum][metric] - base[metric]
            if delta < -TOLERANCE:
                failures.append(f"{stratum}.{metric}: {base[metric]:.4f} -> "
                                f"{current[stratum][metric]:.4f} ({delta:+.4f})")
    if failures:
        raise SystemExit("relevance regression:\n  " + "\n  ".join(failures))

6. Post the delta on every pull request

Even when the gate passes, the numbers belong in the review. A reviewer who can see “tail NDCG +0.012, head unchanged” understands the change in a way the diff alone does not permit.

# .github/workflows/relevance.yml
- name: Comment relevance delta
  run: python3 report.py --format markdown >> "$GITHUB_STEP_SUMMARY"

Verification

Confirm the harness is deterministic before trusting the gate:

python3 score.py --config config/index.yaml > a.json
python3 score.py --config config/index.yaml > b.json
diff a.json b.json && echo "deterministic"
# => deterministic

Confirm the gate actually fails on a real regression:

# Deliberately break ranking, then run the gate.
yq -i '.fields.title.boost = 0.1' config/index.yaml
python3 gate.py
# => relevance regression:
#      head.ndcg: 0.7412 -> 0.6104 (-0.1308)
#      head.mrr:  0.8103 -> 0.6620 (-0.1483)
# exit status 1

Common Pitfalls

Scoring against a shared environment

A staging cluster shared with other work changes underneath the harness — someone reindexes, a colleague pushes a mapping change, a background job adds documents — and the scores move for reasons unrelated to the pull request. Start a container in the job and index the fixtures fresh; the extra minute buys results that mean something.

Updating the baseline to make the build green

When the gate fails, the fastest path is to regenerate the baseline, and doing so silently converts the gate into a formality. Require baseline changes to be a deliberate, separately reviewed part of the diff, with a note explaining why the new numbers are correct. A baseline that changes in most pull requests is a gate that is no longer doing anything.

Gating on an average that hides a stratum

A blended figure can improve while the tail degrades, because head queries usually dominate the mean. Gate on each stratum independently — the whole reason for stratifying is that the strata behave differently, and averaging them away discards exactly the signal the split was created to expose.

Keeping the gate fast enough to keep

A relevance gate that adds ten minutes to every build gets disabled within a month, so runtime is a design constraint rather than an afterthought. Three choices keep it inside a minute or two.

Index the fixture corpus once per job rather than per test, and keep it in the low thousands of documents — enough for scores to be stable, small enough to index in seconds. Run the queries concurrently, since they are independent and the harness is almost entirely waiting on the search API. And skip the gate entirely on pull requests that touch no ranking-relevant path, using a path filter, so the cost is paid only when it can find something.

# Only run the relevance job when something that can affect ranking changed.
on:
  pull_request:
    paths:
      - 'config/index.yaml'
      - 'config/ranking/**'
      - 'src/search/**'
      - 'judgments/**'

The last of those is worth doing early. Most pull requests in a codebase cannot possibly change ranking, and running the gate on all of them trains everyone to ignore a job that is almost always irrelevant to their change.

Keeping the relevance job inside a usable runtime Indexing once per job, running queries concurrently and filtering by path together keep the gate to about a minute. naive index per test, serial queries, every PR — 11 min tuned index once, concurrent, path-filtered — 70 s A gate that is slow and usually irrelevant is a gate that gets disabled.
Runtime is what determines whether the gate survives its first busy week. All three optimisations are one-line changes.

Operational notes

Keep the fixture corpus small and the judgment set separate from it. The corpus exists to make search return something plausible; the judgments encode what “correct” means. Conflating them — grading only documents that happen to be in the fixtures — produces a harness that cannot detect a retrieval failure, because every relevant document is guaranteed to be present.

Version the judgment set explicitly in the baseline file, as in step 4. Scores computed against different label sets are not comparable, and without the version stamp a refreshed judgment set silently invalidates every historical number while the graph continues to look continuous. When the set changes, re-baseline in a dedicated commit that changes nothing else, so the discontinuity has an obvious cause in the history.

Baseline updates in the commit history Ordinary changes leave the baseline untouched, deliberate improvements update it with an explanation, and judgment-set refreshes get their own commit. ordinary change baseline untouched measured improvement baseline raised, explained judgment refresh its own commit A baseline that moves in most pull requests is a gate that has quietly stopped gating.
Three legitimate reasons for the baseline to change, each with a different signature in the history. Anything else is the gate being worked around.

Finally, treat a failing relevance gate the way you would treat a failing test: as information rather than an obstacle. The correct response is to look at which queries moved and why, which the per-query output makes easy, and the answer is frequently that the change is right and the baseline should move. What matters is that the conversation happens at all, which it does not when nothing measures the effect.