Logging Features for LTR Training Data

The most expensive failure in learning-to-rank is not a bad model but a model trained on features that differ from the ones it is served. Offline metrics stay excellent, online behaviour degrades, and the cause is invisible because nothing errors. The fix is structural: compute features once, log the exact vector used to serve each request, and train on that. This page implements it. It sits under learning-to-rank in ranking algorithms and relevance tuning.

Prerequisites

1. A feature set defined in one module, imported by both the serving path and any training job.
2. A request identifier threaded through search, impressions and clicks, per search analytics and click tracking.
3. Somewhere to write a few kilobytes per search — a warehouse table or object storage partitioned by date.
4. A judgment or interaction source to join labels against, from building a judgment list from click logs.

Diagnosis: reconstructed features are different features

The tempting approach is to reconstruct feature vectors at training time by re-running queries against the current index. It is much less work than logging, and it produces a training set that describes a system nobody served.

Three things differ between then and now. Corpus statistics move — a term’s idf changes as documents are added, so the same query-document pair scores differently this week than last. Mutable attributes move — popularity, stock level, price are all different values by the time the training job runs. And the index itself moves — documents are updated and deleted, so some pairs cannot be reconstructed at all.

feature                served 2026-06-14   reconstructed 2026-07-20   delta
bm25_title                     12.41                     11.88       -4%
popularity_log                  3.20                      4.85       +52%
days_since_update              14.0                      50.0       +257%
in_stock                        1.0                       0.0       flipped
# The model would learn from the right-hand column and be served the left.

The days_since_update row is the clearest illustration: a feature computed relative to “now” is guaranteed wrong when recomputed later, and it is a feature almost every LTR implementation includes.

Solution Steps

1. Define features in one place, used by both paths

# features.py — the single definition, imported by serving and by evaluation
FEATURE_ORDER = (
    "bm25_title", "bm25_description", "category_match",
    "popularity_log", "days_since_update", "in_stock",
)

def extract(query: str, hit: dict, now_ts: float) -> list[float]:
    src = hit["_source"]
    return [
        hit["fields"]["bm25_title"][0],
        hit["fields"]["bm25_desc"][0],
        1.0 if src.get("category") in query_categories(query) else 0.0,
        math.log1p(src.get("popularity", 0)),
        (now_ts - src["updated_at_ts"]) / 86400.0,
        1.0 if src.get("in_stock") else 0.0,
    ]

Passing now_ts in rather than calling the clock inside the function is what makes the extraction reproducible: replaying a logged request with its original timestamp yields the original vector.

2. Log the vector for the candidates you scored

Log the candidate set, not the whole corpus. The model only ever sees the top N from the first-pass retrieval, so that is the population it should be trained on.

# serve.py — log exactly what was scored, once per request
def rank(query: str, request_id: str):
    now = time.time()
    candidates = engine.search(query, size=200, docvalue_fields=["bm25_title", "bm25_desc"])
    vectors = {h["_id"]: extract(query, h, now) for h in candidates["hits"]["hits"]}
    scores = model.predict(list(vectors.values()))
    feature_log.write({
        "request_id": request_id,
        "ts": now,
        "query_hash": sha256(query),
        "model_version": MODEL_VERSION,
        "feature_order": list(FEATURE_ORDER),
        "candidates": [{"doc_id": d, "features": v, "score": s}
                       for (d, v), s in zip(vectors.items(), scores)],
    })
    return order_by_score(vectors, scores)

3. Write to storage that will not slow the request

The write must not be on the critical path. Buffer in memory and flush asynchronously, and accept losing the tail of a buffer on a crash — a fraction of a per cent of training data is a much better trade than added latency on every search.

# feature_log.py — batched, fire-and-forget, never blocks the response
class FeatureLog:
    def __init__(self, sink, batch=200, flush_s=5.0):
        self._buf, self._sink = [], sink
        self._batch, self._flush_s = batch, flush_s
        threading.Thread(target=self._loop, daemon=True).start()

    def write(self, record):
        self._buf.append(record)              # never blocks; bounded by the loop below

    def _loop(self):
        while True:
            time.sleep(self._flush_s)
            batch, self._buf = self._buf[: self._batch], self._buf[self._batch :]
            if batch:
                try: self._sink.put_ndjson(batch)
                except Exception: metrics.incr("ltr.feature_log.dropped", len(batch))

4. Include the model version and feature order in every record

Both change over time, and a training set that cannot say which feature was in which position is unusable. Recording the order explicitly means a later feature addition does not silently shift every column.

5. Join to labels by request identifier

-- One training row per candidate, with the label from the interaction stream.
SELECT f.request_id, f.query_hash, c.doc_id, c.features,
       CASE WHEN s.doc_id IS NOT NULL THEN 2      -- downstream success
            WHEN k.doc_id IS NOT NULL THEN 1      -- click only
            ELSE 0 END                    AS label
FROM   feature_log f
CROSS  JOIN UNNEST(f.candidates) AS c
LEFT   JOIN search_click   k ON k.request_id = f.request_id AND k.doc_id = c.doc_id
LEFT   JOIN search_success s ON s.request_id = f.request_id AND s.doc_id = c.doc_id
WHERE  f.ts BETWEEN @from AND @to;

Graded labels drawn from click versus downstream success are considerably better than binary click labels, and they cost nothing extra once both events share the request identifier.

6. Assert on skew before every training run

# skew.py — compare logged distributions against a live sample
def check_skew(logged: dict, live: dict, tol=0.10) -> list[str]:
    bad = []
    for name in FEATURE_ORDER:
        lm, sm = logged[name]["mean"], live[name]["mean"]
        if abs(lm - sm) / max(abs(lm), 1e-9) > tol:
            bad.append(f"{name}: logged mean {lm:.3f} vs live {sm:.3f}")
    return bad

Verification

Confirm a logged vector reproduces exactly when replayed with its original timestamp:

python3 replay.py --request-id 4f1c-...-91 --doc sku-4471
# logged:   [12.41, 3.02, 1.0, 3.20, 14.0, 1.0]
# replayed: [12.41, 3.02, 1.0, 3.20, 14.0, 1.0]
# => identical; the extraction is deterministic given the timestamp

Confirm the log covers the traffic it should:

SELECT date(ts) AS d,
       count(*)                                     AS logged_searches,
       (SELECT count(*) FROM search_executed
        WHERE date(ts) = date(f.ts) AND ranking_version LIKE 'ltr%') AS ltr_searches
FROM   feature_log f GROUP BY 1 ORDER BY 1;
-- logged_searches should be within ~1% of ltr_searches

Confirm feature logging did not slow the request path:

python3 report.py --metric search_latency_p99 --compare-deploy ltr-logging
# before 214 ms · after 216 ms
# => within noise; the write is off the critical path

Versioning the feature set

Features change: one is added, another is found to be broken and removed, a third has its definition adjusted. Each of those makes older logged vectors incompatible with newer ones, and a training job that silently concatenates both produces a model trained on columns that mean different things in different rows.

The defence is a feature-set version recorded on every row, incremented whenever the set or any definition changes, and asserted at training time. Training then either uses one version or explicitly migrates older rows — and the migration is only possible because the version makes the difference visible.

FEATURE_SET_VERSION = 4      # bump on any add, remove, or definition change

def load_training_rows(since, until, version=FEATURE_SET_VERSION):
    rows = warehouse.query(FEATURE_SQL, since=since, until=until)
    mismatched = [r for r in rows if r["feature_set_version"] != version]
    if mismatched:
        raise ValueError(f"{len(mismatched)} rows from an older feature set — "
                         f"migrate or narrow the window")
    return rows

The practical consequence is that a feature change costs a training-data window: rows logged before the change cannot be used with rows logged after. Planning feature changes to land just before a natural retraining boundary avoids discarding weeks of data for nothing.

Common Pitfalls

Operational notes

Storage is the objection everyone raises and it is smaller than expected. Two hundred candidates with six float features is about 5 KB per search uncompressed and under 1 KB compressed; at 100,000 searches a day that is roughly 100 MB a day, or 36 GB a year. Partition by date, keep ninety days hot and archive the rest, and the cost is negligible against the value of a training set that matches production.

Sample if the volume genuinely is a problem, but sample by request rather than by candidate. Logging a random 10% of searches in full preserves the structure of each candidate set, which the training process depends on; logging a random 10% of candidates across all searches destroys it, because the model learns from the relationship between documents competing for the same query.

Related

• Learning-to-rank — the parent guide covering model choice, serving and the two failure paths.
• Training an LTR model with XGBoost — the training step that consumes this log.
• Building a judgment list from click logs — where the labels joined to these vectors come from, and how to debias them.