Weighting Lexical and Vector Scores

Rank fusion is the safer default, but it throws away information: it cannot tell a decisive first place from a marginal one. When that magnitude genuinely matters — typically when one retriever is confident on a minority of queries and you want that confidence to win — score fusion is the right tool, provided the normalisation is stable. This page implements it and, more importantly, shows how to detect when the normalisation has silently stopped working. It sits under hybrid search and fusion strategies in search engine selection and architecture.

Prerequisites

  1. A working rank-fusion baseline to compare against — score fusion should be adopted only if it beats RRF on your judgment set.
  2. Raw scores exposed by both retrievers, not just ordered ids.
  3. A judgment set large enough to distinguish differences of roughly 0.01 NDCG, since the gain over RRF is usually small.
  4. Monitoring for score distributions, because the failure mode of this approach is drift rather than error.

Diagnosis: why the two scores cannot simply be added

BM25 scores are unbounded, depend on corpus statistics, and vary with query length: a one-word query might produce top scores near 8, a four-word query near 30. Cosine similarities are bounded in a narrow band — anything plausible sits between about 0.55 and 0.95, and the difference between an excellent match and a mediocre one is often 0.08.

Adding them means the lexical score dominates completely, and multiplying a small weight onto it means the weight is doing the normalisation implicitly and badly. The observed distributions make this obvious:

                     p5     p50     p95    max
BM25 (1-word query)  2.1    6.4    11.8   14.2
BM25 (4-word query)  7.9   19.3    31.6   38.0
cosine               0.58   0.71    0.86   0.93
# A fixed weight cannot serve both BM25 rows, let alone reconcile them with cosine.

Score fusion therefore requires mapping both onto a common scale before combining, and every choice of mapping has different failure characteristics.

Incomparable score ranges from two retrievers BM25 scores span a wide, query-dependent range while cosine similarities occupy a narrow band, so the two cannot be combined without normalisation. BM25 1-word 2 – 14 BM25 4-word 8 – 38 cosine 0.58 – 0.93 One fixed weight cannot reconcile three distributions — which is why the mapping, not the weight, is the design.
The lexical range moves with query length while the vector range barely moves at all. Any fixed combination is therefore query-length dependent.

Solution Steps

1. Choose a normalisation with known failure characteristics

Three options are common and they fail differently. Min-max within the result set maps the best result to 1 and the worst to 0; simple, and entirely dependent on which documents happened to match, so the same document scores differently depending on its company. Z-score centres on the mean and scales by standard deviation; more robust to a single outlier, still result-set dependent. Fixed reference points map against constants derived from a historical sample; result-set independent, and requires re-derivation when the corpus changes materially.

For production, fixed reference points are the only variant that does not change a document’s score based on what else matched.

# normalise.py — fixed reference points, derived once from a sample and pinned
REFERENCE = {
    "lexical": {"p05": 3.2, "p95": 28.7},     # re-derive after a major corpus change
    "vector":  {"p05": 0.58, "p95": 0.91},
}

def norm(value: float, retriever: str) -> float:
    ref = REFERENCE[retriever]
    scaled = (value - ref["p05"]) / (ref["p95"] - ref["p05"])
    return max(0.0, min(1.0, scaled))          # clamp; outliers do not distort others

2. Blend with an explicit weight

# blend.py — convex combination on the normalised scale
ALPHA = 0.65        # weight on lexical; 1 - ALPHA on vector

def blended(lex: dict[str, float], vec: dict[str, float]) -> dict[str, float]:
    out = {}
    for doc in set(lex) | set(vec):
        l = norm(lex.get(doc, REFERENCE["lexical"]["p05"]), "lexical")
        v = norm(vec.get(doc, REFERENCE["vector"]["p05"]), "vector")
        out[doc] = ALPHA * l + (1 - ALPHA) * v
    return out

Note the default for a missing document: the low reference point rather than zero. A document found by only one retriever should be penalised for the absence, not annihilated by it.

3. Fit the weight on a judgment set

for a in 0.3 0.4 0.5 0.6 0.7 0.8; do
  echo -n "alpha=$a "; python3 evaluate.py --fusion score --alpha "$a" --metric ndcg@10
done
# alpha=0.3  0.671
# alpha=0.5  0.688
# alpha=0.65 0.697   <- optimum
# alpha=0.8  0.690

4. Compare honestly against rank fusion

Score fusion is more fragile, so it must earn its place. Run both against the same judgment set and adopt it only if the margin exceeds your measurement noise.

python3 evaluate.py --fusion rrf   --k 60      # 0.694
python3 evaluate.py --fusion score --alpha 0.65 # 0.697
# +0.003 against a noise floor of ~0.012 — not a real difference; keep RRF.

That outcome is common and is the correct reason to stop. The extra complexity is only worth carrying when the margin is unambiguous.

5. Monitor the score distributions

The failure mode of score fusion is that the reference points stop describing reality — after a reindex, a mapping change, or an embedding model update. Nothing errors; the blend simply becomes wrong.

# drift.py — alert when observed percentiles diverge from the pinned references
def check_drift(observed: dict, reference: dict, tolerance=0.15) -> list[str]:
    drifted = []
    for retriever, ref in reference.items():
        for point in ("p05", "p95"):
            obs = observed[retriever][point]
            if abs(obs - ref[point]) / max(abs(ref[point]), 1e-6) > tolerance:
                drifted.append(f"{retriever}.{point}: ref {ref[point]} observed {obs}")
    return drifted

Verification

Confirm the normalisation maps a typical strong match near the top of the scale:

python3 -c "
from normalise import norm
print(round(norm(26.1, 'lexical'), 3), round(norm(0.88, 'vector'), 3))"
# => 0.898 0.909

Confirm a document found by only one retriever still ranks sensibly:

python3 -c "
from blend import blended
b = blended({'A': 27.0, 'B': 9.0}, {'B': 0.89, 'C': 0.87})
print(sorted(b.items(), key=lambda kv: -kv[1]))"
# => [('B', 0.61...), ('A', 0.59...), ('C', 0.30...)]
# B, found by both, wins; A's strong lexical score still beats C's vector-only match.

Confirm drift detection fires when references go stale:

python3 drift.py --sample 5000
# lexical.p95: ref 28.7 observed 41.2
# => references stale after the mapping change; re-derive before trusting scores

Common Pitfalls

Min-max normalisation in production

It is the first thing everyone implements and it makes a document’s score depend on which other documents matched. Two nearly identical queries can therefore rank the same document very differently, and the effect is impossible to explain to anyone. Use fixed reference points, and keep min-max for exploratory analysis only.

Treating a missing document as score zero

A document the vector retriever did not return is not maximally dissimilar; it is simply outside the candidate depth. Scoring it zero means anything found by one retriever alone is effectively eliminated, which discards the complementary coverage that motivated hybrid search. Impute the low reference point instead.

Re-fitting the weight without re-deriving the references

When results degrade, the instinct is to re-sweep the blend weight. If the underlying cause is drifted references, the sweep finds a weight that partially compensates and hides the real problem, which then resurfaces after the next reindex. Check drift first; re-fit the weight only against current references.

When score fusion is genuinely the right choice

There is one situation where the extra fragility pays reliably: when one retriever is confidently right on a minority of queries and you want that confidence to override the other. The clearest example is a corpus containing exact-match targets — model numbers, canonical names — where a BM25 score far above the rest of the result set is near-certain evidence, and rank fusion cannot express “far above”.

The test is straightforward. Compute, on your judgment set, the correlation between a retriever’s score margin (top score minus second score) and whether the top result is relevant. If a large margin reliably predicts correctness, that magnitude carries information rank fusion is discarding, and score fusion can exploit it. If the correlation is weak, the magnitude is noise and RRF is strictly better.

python3 margin_analysis.py --retriever lexical
# margin decile 1 (smallest):  top-1 relevant 41%
# margin decile 5:             top-1 relevant 58%
# margin decile 10 (largest):  top-1 relevant 94%
# => strong signal; score magnitude is worth preserving on this corpus

That single measurement decides the question far more reliably than comparing aggregate NDCG, because it identifies why score fusion would help rather than whether it did on one sample.

Score margin against top-result correctness Queries where the top score far exceeds the second are much more likely to have a correct top result, showing that magnitude carries information. score margin decile → correct 94% 41%
A steep curve like this is the licence to use score fusion. A flat one means the magnitude is noise and rank fusion loses nothing by discarding it.

Operational notes

Pin the reference points in configuration alongside the weight, and version them together. A blend weight is only meaningful relative to the normalisation it was fitted against, and storing them separately guarantees that someone eventually updates one without the other.

Re-derive references on a schedule tied to corpus events rather than the calendar: after a reindex that changes analysis, after an embedding model change, and after any bulk import that materially shifts corpus statistics. Between those events the distributions are stable enough that monthly checks are ample.

Events that invalidate pinned score references Analyzer changes, embedding model updates and bulk imports all shift the score distributions and require references to be re-derived. analyzer change shifts BM25 range new embedding model shifts cosine range large bulk import shifts corpus statistics Tie re-derivation to these events; a calendar schedule will miss them and a drift alarm will catch them late.
Three routine operations invalidate the normalisation silently. Tying re-derivation to the events rather than to a date is what keeps score fusion trustworthy.

Above all, keep the rank-fusion implementation in place behind a flag. It is the fastest possible mitigation when references drift during an incident, and having it available turns a subtle scoring problem into a one-line configuration change while you investigate.