Building a Judgment List from Click Logs

Manual grading produces excellent labels at a rate of roughly a hundred pairs an hour, which does not scale to a catalog. Click logs produce millions of interactions for free, and they are systematically biased in ways that will train a ranker to reproduce whatever it already does. This page extracts usable judgments from interactions while correcting for the bias, and anchors the result against a small human-graded set. It sits under relevance evaluation and offline testing in the ranking algorithms and relevance tuning area.

Prerequisites

1. Impression logging: every result shown, with its position, tied to a request identifier — not just the clicks.
2. Click events carrying the same request identifier, so a click can be attributed to the exact result list that produced it.
3. A downstream success signal where one exists — add to cart, subscription, download — because clicks alone conflate “looked promising” with “was correct”.
4. A few hundred human-graded pairs to anchor and validate the derived labels.

Diagnosis: clicks measure attention, not relevance

The core problem is that users click what they are shown, and they are shown what the current ranker chose. A document at position one gets clicked far more often than an equally relevant document at position eight — not because it is better, but because it was seen first. Deriving labels naively therefore encodes the current ranking as ground truth, and any model trained on those labels converges toward the status quo.

The magnitude is easy to underestimate. Position-one click-through is typically five to ten times position-five on the same query, and the ratio persists even when the results are deliberately shuffled:

position   impressions   clicks    CTR
1              412,880    74,318   18.0%
2              412,880    35,108    8.5%
3              412,880    20,644    5.0%
5              412,880     9,909    2.4%
10             412,880     3,303    0.8%
# The same documents rotated through these positions retain the same curve.

That curve is the examination probability, and it is what has to be divided out before a click count means anything about relevance.

Solution Steps

1. Log impressions, not just clicks

Without impressions there is no denominator, and a click count alone cannot be converted into a rate. This is the step teams discover they skipped six months into the project.

// impression.js — log what was shown, with positions, once per result set
function logImpressions(requestId, query, results) {
  analytics.track("search.impression", {
    request_id: requestId,
    query_hash: sha256(query),          // hash, not the raw query
    shown: results.map((r, i) => ({ id: r.id, position: i + 1 })),
  });
}

2. Filter clicks that do not indicate relevance

A click followed by an immediate return to the results is evidence against relevance — the user looked and rejected it. Dwell time separates the two, and the threshold is domain-specific rather than universal.

# filter.py — keep clicks that were followed by engagement
DWELL_THRESHOLD_S = 20      # calibrate per domain; a recipe is not a product page

def satisfied_clicks(events):
    for click in events.clicks:
        nxt = events.next_event_after(click)
        if nxt is None or (nxt.at - click.at).total_seconds() >= DWELL_THRESHOLD_S:
            yield click            # long dwell, or session ended here
        # short dwell followed by a return to results: a rejection, not a match

3. Divide out the position bias

Weight each click by the inverse of the probability that its position was examined. This is inverse-propensity weighting, and the propensities come either from a randomisation experiment or from a click model fitted to the logs.

# propensity.py — inverse propensity weighting of click evidence
POSITION_PROPENSITY = {1: 1.00, 2: 0.62, 3: 0.44, 4: 0.35, 5: 0.29,
                       6: 0.24, 7: 0.21, 8: 0.18, 9: 0.16, 10: 0.15}

def weighted_relevance(clicks_by_position: dict[int, int]) -> float:
    return sum(n / POSITION_PROPENSITY.get(pos, 0.15)
               for pos, n in clicks_by_position.items())

4. Convert weighted evidence into graded labels

Raw weighted counts are unbounded and query-dependent, so normalise within each query before bucketing into grades.

# grade.py — per-query normalisation, then bucket into 0–3
def grade_query(doc_weights: dict[str, float]) -> dict[str, int]:
    if not doc_weights:
        return {}
    top = max(doc_weights.values())
    out = {}
    for doc, w in doc_weights.items():
        ratio = w / top
        out[doc] = 3 if ratio >= 0.75 else 2 if ratio >= 0.40 else 1 if ratio >= 0.15 else 0
    return out

5. Anchor against human grades

Derived labels need a reference point, or a systematic error in the click model propagates silently into every metric built on them. Compare against a human-graded sample and measure agreement.

# anchor.py — how well do derived labels match human judgement?
def agreement(derived: dict, human: dict) -> float:
    shared = set(derived) & set(human)
    if not shared:
        return 0.0
    exact = sum(1 for k in shared if derived[k] == human[k])
    close = sum(1 for k in shared if abs(derived[k] - human[k]) <= 1)
    print(f"exact {exact/len(shared):.2f}  within-one {close/len(shared):.2f}")
    return close / len(shared)

6. Add controlled randomisation for the tail

The parts of the ranking users never see produce no clicks and therefore no labels, which permanently blinds the judgment set to retrieval failures. Swapping two adjacent results on a small slice of traffic generates the counterfactual evidence, at negligible cost to the user experience.

# explore.py — 2% of sessions get one adjacent swap in the top ten
def maybe_swap(results, session_id, rate=0.02):
    if hash(session_id) % 100 >= rate * 100:
        return results, None
    i = hash(session_id) % (min(len(results), 10) - 1)
    results[i], results[i + 1] = results[i + 1], results[i]
    return results, ("swap", i, i + 1)      # log it so the analysis can use it

Verification

Confirm the derived labels rank a known-good document above a known-bad one:

python3 derive.py --query "trail runner" --top 3
# doc-9912  grade 3   (weighted 41.2)   <- the exact product
# doc-1044  grade 2   (weighted 18.7)
# doc-7781  grade 1   (weighted  6.1)

Confirm agreement with the human anchor set is acceptable before using the labels:

python3 anchor.py --derived judgments/derived-v3.jsonl --human judgments/human-core.jsonl
# exact 0.61  within-one 0.89
# => within-one above 0.85 is usable; below 0.75 the click model needs work

Confirm the position curve flattens after weighting, which is the direct test of whether the correction worked:

python3 propensity.py --report
# raw CTR by position:      18.0  8.5  5.0  3.1  2.4  ...
# weighted relevance:        1.00 1.04 0.97 1.02 0.99 ...   <- bias removed

Common Pitfalls

What the derived labels cannot tell you

Two limits are structural rather than fixable, and knowing them prevents a class of wasted effort.

The first is that click-derived labels can only grade documents that were shown. A relevant document the retrieval stage never returns produces no evidence of any kind, so the judgment set records it as absent rather than as a failure — and a metric computed against those labels will report that retrieval is working perfectly. This is why recall against a human-graded set matters: it is the only measurement that can see a document the system never surfaced.

The second is that clicks measure appeal, not correctness. A result with a striking image and a low price attracts clicks whether or not it matches the query, and derived labels will grade it highly. Where a downstream success signal exists — purchase, subscription, completion — weighting clicks by it corrects much of this. Where none exists, the labels are measuring what users chose to look at, and it is worth being explicit about that rather than calling it relevance.

Operational notes

Refresh derived labels on a schedule and keep the human anchor set stable. The derived portion tracks the catalog and user behaviour, both of which drift; the anchor exists to detect when that drift has become distortion. If agreement with the anchor falls over successive refreshes, the click model has stopped describing your users — which is information worth having, and invisible without the anchor.

Store judgments as an append-only versioned artefact rather than a mutable table. Every metric anyone quotes is relative to a specific label set, and being able to recompute an old score exactly is what makes historical comparisons meaningful. A judgment file in the repository, versioned alongside the code, gives that property for free and costs nothing at the scale these sets reach.

Related

• Relevance evaluation and offline testing — the parent guide covering what to do with the labels once they exist.
• Measuring NDCG and MRR in CI — the harness that consumes this judgment set on every commit.
• Learning-to-rank — where position bias in the labels does the most damage, because it becomes the model’s objective.