Implementing Reciprocal Rank Fusion

Reciprocal rank fusion is twelve lines of code and one constant, which makes it easy to ship and easy to ship wrong. The problems are not in the arithmetic but around it: what k actually controls, what happens when one retriever returns nothing, how to keep the merged list debuggable, and which queries should not be fused at all. This page covers those. It belongs to hybrid search and fusion strategies inside search engine selection and architecture.

Prerequisites

1. Two or more retrievers that return ranked document identifiers for the same query.
2. A shared identifier space — both retrievers must refer to the same document by the same id, which sounds obvious and is a common source of silent zero-overlap.
3. A judgment set for tuning k and the weights, since neither has a universally correct value.
4. Request-level logging with room for per-document provenance.

Diagnosis: what the constant actually does

The formula assigns each document weight / (k + rank) from every list it appears in, and sums those contributions. The constant k sets how quickly the contribution decays with rank, and choosing it is choosing how much a top-one placement is worth relative to a top-five.

With k = 0, first place contributes 1.0 and fifth contributes 0.2 — a five-fold difference, so a single retriever’s top result almost always wins. With k = 60, first contributes 0.0164 and fifth 0.0154, barely a 6% difference, so agreement between retrievers matters far more than any single ranking’s confidence.

rank    k=0      k=10     k=60
1     1.0000   0.0909   0.01639
2     0.5000   0.0833   0.01613
3     0.3333   0.0769   0.01587
5     0.2000   0.0667   0.01538
10    0.1000   0.0500   0.01429
# At k=60 a document ranked 1st by one retriever and absent from the other (0.01639)
# loses to one ranked 3rd and 5th by both (0.01587 + 0.01538 = 0.03125).

That last line is the whole design. A high k rewards documents both retrievers liked, which is usually what you want from a hybrid system — the two retrievers are being combined precisely because their agreement is more trustworthy than either alone.

Solution Steps

1. Implement the merge with provenance from the start

Adding provenance later means changing the return shape everywhere. Build it in now; the cost is one dictionary.

# fusion.py — RRF with per-document provenance
def rrf(rankings: dict[str, list[str]], k: int = 60,
        weights: dict[str, float] | None = None) -> list[dict]:
    weights = weights or {name: 1.0 for name in rankings}
    scores: dict[str, float] = {}
    sources: dict[str, list[tuple[str, int]]] = {}
    for name, ranked in rankings.items():
        w = weights.get(name, 1.0)
        for rank, doc_id in enumerate(ranked, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + w / (k + rank)
            sources.setdefault(doc_id, []).append((name, rank))
    order = sorted(scores, key=lambda d: (-scores[d], d))    # deterministic tie-break
    return [{"id": d, "score": round(scores[d], 6), "from": sources[d]} for d in order]

The tie-break on document id matters more than it looks: without it, documents with identical scores come back in arbitrary order, which makes the harness non-deterministic and any CI gate unusable.

2. Handle an empty or failed retriever explicitly

If the vector retriever times out, the correct behaviour is to return lexical results rather than nothing. RRF handles this naturally — a missing list simply contributes nothing — but only if the caller does not treat the failure as fatal.

# resilient.py — a failed retriever degrades the result, it does not remove it
async def gather_rankings(query: str, depth: int) -> dict[str, list[str]]:
    async def safe(name, coro):
        try:
            return name, [h["id"] for h in await asyncio.wait_for(coro, timeout=0.15)]
        except (asyncio.TimeoutError, Exception):
            metrics.incr("hybrid.retriever_failed", tags={"retriever": name})
            return name, []                    # empty list, not an exception
    pairs = await asyncio.gather(
        safe("lexical", search_lexical(query, size=depth)),
        safe("vector", search_vector(query, size=depth)),
    )
    return {name: ids for name, ids in pairs if ids}

3. Set weights from a measured sweep

Weights multiply each retriever’s contribution. Sweep the ratio rather than both values, since only the ratio matters.

# sweep.py — find the weight ratio that maximises NDCG on the judgment set
for ratio in (0.25, 0.5, 0.75, 1.0, 1.5, 2.0, 4.0):
    weights = {"lexical": 1.0, "vector": ratio}
    print(ratio, evaluate(lambda q: rrf(gather(q), weights=weights)))

4. Route identifier queries away from fusion

Fusion is the wrong tool when one retriever is known to be wrong for a query class. A query that looks like a part number, an ISBN, or an order id should go to lexical alone, because vector similarity on identifiers actively promotes near-misses.

# route.py — skip the vector retriever for identifier-shaped queries
import re

IDENTIFIER = re.compile(r"^[A-Z0-9][A-Z0-9\-_/]{3,}$", re.I)

def retrievers_for(query: str) -> list[str]:
    if IDENTIFIER.match(query.strip()) and not query.strip().isalpha():
        return ["lexical"]                 # exactness matters more than semantics
    return ["lexical", "vector"]

5. Log the fused result at a sampling rate

The provenance is only useful if it is recorded. Sampling keeps the volume manageable while guaranteeing you have examples when a complaint arrives.

if random.random() < 0.01:
    log.info("hybrid.result", extra={
        "query_hash": sha256(query), "k": K, "weights": WEIGHTS,
        "top": [{"id": r["id"], "from": r["from"]} for r in fused[:5]],
    })

Verification

Confirm agreement beats a single strong placement, which is the property k = 60 is chosen for:

python3 -c "
from fusion import rrf
r = rrf({'lex': ['A','X','Y','B','C'], 'vec': ['Z','W','A','V','B']}, k=60)
print([(d['id'], d['score']) for d in r[:3]])"
# => [('A', 0.032...), ('B', 0.031...), ('X', 0.016...)]
# A appears at 1 and 3; B at 4 and 5; both beat X which appears once at 2.

Confirm a failed retriever degrades rather than breaks:

VECTOR_ENDPOINT=http://127.0.0.1:1 python3 search.py --query "trail runner"
# WARN hybrid.retriever_failed retriever=vector
# => 10 results returned, all lexical

Confirm determinism, which the CI harness depends on:

for i in 1 2 3; do python3 search.py --query "waterproof jacket" --ids-only; done | uniq | wc -l
# => 1

Common Pitfalls

Tuning k against your judgment set

The conventional k = 60 comes from the original paper’s experiments on TREC collections and is a reasonable prior, not a law. Sweeping it takes minutes once a harness exists, and the optimum tells you something about your retrievers.

A low optimum — under about 20 — means one retriever is substantially more trustworthy than the other, and its top placements deserve to dominate. That is worth knowing independently: it usually means the weaker retriever needs work rather than a larger share of the blend. A high optimum means the two are comparably good and their agreement is the strongest available signal, which is the situation fusion is designed for.

for k in 0 5 10 20 40 60 100; do
  echo -n "k=$k "; python3 evaluate.py --fusion rrf --k "$k" --metric ndcg@10
done
# k=0   0.641
# k=5   0.668
# k=20  0.689
# k=60  0.694   <- flat plateau from about 40 upward
# k=100 0.693

A flat plateau, as above, is the common result and means the parameter is not critical for your corpus — which is itself a useful finding, because it lets you stop thinking about it and spend the attention on candidate depth, where the returns are larger.

Operational notes

Expose k and the weights as configuration rather than constants, and record them alongside every logged result. When someone asks why a document ranked where it did six weeks ago, the answer depends on the parameters in force at the time, and reconstructing them from a git history is considerably worse than reading them from the log line.

Watch the overlap rate between candidate lists as an ongoing health metric. It should sit in a stable band — typically 20–50% for genuinely complementary retrievers. A drift toward zero means the retrievers have diverged, usually because an embedding model changed or a mapping altered what lexical matches; a drift toward one means they have converged and the vector side is no longer contributing anything worth its latency.

Related

• Hybrid search and fusion strategies — the parent guide covering candidate depth, latency and retriever design.
• Weighting lexical and vector scores — the score-fusion alternative and when magnitude information is worth the fragility.
• Vector search integration strategies — where the vector retriever’s recall, which bounds fusion quality, is tuned.