Instrumenting Click-Through Rate for Search

Click-through rate is the metric every search team reaches for first and the one most often computed in a way that cannot support a decision. The problems are not conceptual — a ratio of clicks to opportunities — but definitional: which clicks, which opportunities, and compared against what. This page defines each precisely and shows what the resulting number can and cannot be used for. It sits under search analytics and click tracking in search frontend and UX patterns.

Prerequisites

  1. Impression events emitted on viewport entry, with positions, sharing a request identifier with the search event.
  2. Click events carrying the same identifier and the clicked document’s position.
  3. A query-class label on the search event, because CTR is not comparable across classes.
  4. Server-side search counts to reconcile against, since client-side events are lossy.

Diagnosis: three denominators, three different metrics

“Click-through rate” names at least three distinct quantities, and teams routinely compare one against another without noticing.

Per-search CTR — the fraction of searches that produced at least one click — answers “did search work for this person?”. It is the metric that tracks user satisfaction most directly and the one to put on a dashboard.

Per-impression CTR — clicks divided by results shown — answers “how appealing was an average result?”. It falls when you show more results, which makes it treacherous: a change that renders twenty results instead of ten will lower it without anything getting worse.

Per-position CTR — clicks at position n divided by impressions at position n — answers “how good is the result we put here?”. It is the only one of the three that supports comparing two rankings, because it controls for the positional advantage that otherwise dominates.

change: results per page 10 -> 20

per-search CTR      48.1%  ->  49.6%   (slightly better: more chances to find something)
per-impression CTR   6.2%  ->   3.4%   (much "worse": denominator doubled)
per-position CTR@1  18.0%  ->  18.1%   (unchanged: the top result is the same)
# Three metrics, one change, three different stories. Only one of them is misleading.
Three click-through denominators and what each answers Per-search measures user success, per-impression measures result appeal and is distorted by page size, and per-position supports ranking comparison. per search "did search work?" dashboard metric stable under page size per impression "how appealing?" moves with page size easy to misread per position "is this slot good?" compares rankings needs position logging name the denominator in the metric's title, always — "CTR" alone starts arguments and never compare a number computed one way against one computed another
All three are legitimate and they answer different questions. The failure is calling all of them "CTR" and comparing across them.
The same data yields different rates depending on the denominator One hundred searches showing ten results each with sixty clicks gives a forty-eight per cent per-search rate and a six per cent per-impression rate. 48 of 100 searches got a click per-search CTR = 48% 60 clicks across 1,000 impressions per-impression CTR = 6% Identical underlying data, an eightfold difference in the headline number. Whichever you use, state it in the metric's name and never mix the two in one comparison.
The same week of data supports both figures. Neither is wrong; quoting one while another team quotes the other is what causes the argument.

Solution Steps

1. Compute all three from one event stream

-- Three metrics, one query, per day.
WITH s AS (SELECT request_id, ts FROM search_executed),
     i AS (SELECT request_id, count(*) AS shown FROM search_impression GROUP BY 1),
     c AS (SELECT request_id, count(*) AS clicks FROM search_click GROUP BY 1)
SELECT date_trunc('day', s.ts)                                    AS day,
       avg(CASE WHEN coalesce(c.clicks,0) > 0 THEN 1 ELSE 0 END)  AS ctr_per_search,
       sum(coalesce(c.clicks,0))::float / nullif(sum(i.shown),0)  AS ctr_per_impression
FROM   s LEFT JOIN i USING (request_id) LEFT JOIN c USING (request_id)
GROUP  BY 1 ORDER BY 1;

2. Break out per-position rates

SELECT i.position,
       count(DISTINCT i.request_id)                               AS impressions,
       count(DISTINCT c.request_id)                               AS clicks,
       count(DISTINCT c.request_id)::float
         / nullif(count(DISTINCT i.request_id), 0)                AS ctr
FROM   search_impression i
LEFT   JOIN search_click c
       ON c.request_id = i.request_id AND c.position = i.position
WHERE  i.ts > now() - interval '7 days'
GROUP  BY 1 ORDER BY 1;

3. Segment by query class before drawing conclusions

A navigational query has a very different natural CTR from an exploratory one, and an aggregate blends them into a number that moves whenever the query mix moves — including for seasonal reasons unrelated to search quality.

SELECT s.query_class,
       avg(CASE WHEN c.request_id IS NOT NULL THEN 1 ELSE 0 END) AS ctr_per_search,
       count(*)                                                  AS searches
FROM   search_executed s
LEFT   JOIN LATERAL (SELECT 1 FROM search_click WHERE request_id = s.request_id LIMIT 1) c ON true
GROUP  BY 1 ORDER BY searches DESC;

4. Reconcile client events against server counts

Client-side collection loses events to ad blockers, closed tabs and network failures. Knowing the loss rate is the difference between a metric and a guess.

SELECT date_trunc('day', ts) AS day,
       count(*) FILTER (WHERE source = 'server') AS server_searches,
       count(*) FILTER (WHERE source = 'client') AS client_impressions_sessions,
       1 - count(*) FILTER (WHERE source = 'client')::float
           / nullif(count(*) FILTER (WHERE source = 'server'), 0) AS client_loss_rate
FROM   search_events GROUP BY 1 ORDER BY 1;
-- A stable 8–12% loss rate is normal. A step change is an instrumentation break.

5. Pair CTR with a success metric before acting on it

CTR alone rewards results that look appealing. Any ranking decision taken on CTA without a downstream success measure is one bad campaign away from optimising for clickbait.

SELECT s.ranking_version,
       avg(CASE WHEN c.request_id IS NOT NULL THEN 1 ELSE 0 END) AS ctr,
       avg(CASE WHEN x.request_id IS NOT NULL THEN 1 ELSE 0 END) AS success_rate
FROM   search_executed s
LEFT   JOIN LATERAL (SELECT 1 FROM search_click   WHERE request_id = s.request_id LIMIT 1) c ON true
LEFT   JOIN LATERAL (SELECT 1 FROM search_success WHERE request_id = s.request_id LIMIT 1) x ON true
GROUP  BY 1;

Verification

Confirm the per-position curve has the expected shape — a steep decline. A flat curve means positions are not being recorded correctly:

python3 report.py --metric ctr_by_position
# pos 1  18.2%   pos 2  8.6%   pos 3  5.1%   pos 5  2.4%   pos 10  0.8%

Confirm click events all join to an impression:

python3 report.py --check orphan_clicks
# clicks without a matching impression: 0.4%
# => under 1% is acceptable (fast clicks before the observer fires); above 5% is a bug

Confirm the metric is stable when nothing changed:

python3 report.py --metric ctr_per_search --days 30 --stddev
# mean 47.8%  weekly stddev 0.9pp
# => a change must exceed roughly 2pp to be distinguishable from normal variation

Using CTR to compare two rankings

The per-position breakdown is what makes ranking comparison possible, and the technique is worth stating explicitly because the naive comparison is so tempting.

Comparing overall CTR between two rankings conflates two effects: whether the new ranking put better documents in each slot, and whether it changed how many documents users examined. A ranking that surfaces one excellent result and nine poor ones can produce the same aggregate CTR as one with ten decent results, while being a much worse experience.

Comparing per-position CTR isolates the first effect. If the new ranking’s position-one CTR is higher and its position-two through five rates are unchanged, it has genuinely improved the top slot. If every position improved slightly, the improvement is real and broad. If position one improved and the rest fell, the ranking has become top-heavy — sometimes desirable, usually not.

                pos1    pos2    pos3    pos4    pos5
ranking A      18.0%    8.6%    5.1%    3.5%    2.4%
ranking B      21.4%    8.4%    5.0%    3.4%    2.4%
# B is better at position one and identical elsewhere: a clean improvement.

That said, per-position CTR still cannot tell you whether users were satisfied — only whether they clicked. For a genuine ranking comparison on live traffic, interleaving is both more sensitive and more direct, and it needs far less traffic to reach a conclusion.

Common Pitfalls

Comparing CTR across a page-size or layout change

Per-impression CTR moves mechanically when the number of results shown changes, and layout changes — bigger cards, more images, a different grid — move it too by changing how many results are examined. Any comparison spanning such a change is invalid on that denominator. Use per-search or per-position, both of which are robust to layout.

Treating a CTR drop as a relevance regression

CTR falls for many reasons that are not relevance: a seasonal shift in query mix, a new surface bringing lower-intent traffic, a checkout change that reduces browsing. Before investigating ranking, check whether the query-class mix moved, because that alone explains a large share of unexplained CTR movements.

Computing a rate from two different collectors

A numerator from client-side clicks and a denominator from server-side searches produces a rate depressed by the client loss rate, and the depression varies by platform and over time. Keep both sides of any ratio from the same collector, and use the reconciliation query separately to understand loss.

Operational notes

Treat CTR as a monitoring metric rather than an optimisation target. It is excellent at detecting that something changed and poor at telling you whether the change was good, which makes it the right thing to alert on and the wrong thing to maximise.

Publish the metric definitions alongside the dashboard. “CTR” on a chart with no denominator stated is the single most common cause of two teams reaching opposite conclusions from the same data, and a one-line description under each panel prevents it permanently.

Set alert thresholds from measured variance rather than from round numbers. A metric with a weekly standard deviation of 0.9 percentage points should not alert on a one-point move, and computing that figure once — as in the verification step — turns alerting from a source of noise into a source of signal.

Things that move click-through rate other than relevance Query mix, layout, traffic source and page size all move the metric independently of search quality. query mix seasonal layout card size, grid traffic source intent differs page size per-impression only Check these four before opening a relevance investigation — they explain most unexplained movements.
Four confounders, all cheap to rule out and all more likely than a ranking regression on any given week.