Deduplicating Documents Before Indexing

Two feeds deliver the same product, one with a trailing space in the title and one without, and search now returns it twice — three times after the next import. Duplicates are the fastest way to make good relevance look broken, because the user sees the failure in the top three results. This page removes duplicates at ingestion time, where it is cheap and permanent, rather than at query time with collapse, where it is a per-request cost that hides the underlying problem. It sits under data normalization and cleaning in the data ingestion and synchronization pipelines area.

Prerequisites

  1. A written definition of what makes two records the same thing in your domain — this is a product decision, not a technical one.
  2. Access to every feed that writes into the index, including the ones nobody remembers.
  3. A key-value store for fingerprint lookups if the corpus is too large to hold in process memory.
  4. A version-guarded writer, so a merge result cannot be overwritten by a stale copy of one of its inputs.

Diagnosis: duplicates come from ids, not from data

Almost every duplicate in a search index traces back to a document id derived from something feed-specific — a row id, a feed-local sku, a hash of the whole payload. Two feeds carrying the same real-world entity produce two ids, so the engine correctly stores two documents.

The pattern is obvious once you group by a normalised title:

_id                        title                     source_feed   updated_at
prod-88213                 "Trail Runner 3"          erp           2026-07-24
sku-TR3-EU                 "Trail Runner 3 "         supplier_eu   2026-07-25
c1f0a9e4b7d2               "trail runner 3"          scraper       2026-07-25
# Three ids, three documents, one product. The index is behaving exactly as instructed.

Query-time collapsing hides this from the result list but not from relevance: term frequencies are inflated, facet counts are wrong, and any learning-to-rank feature computed over the corpus is trained on triple-counted evidence. The fix belongs upstream.

Deduplication pipeline from multiple feeds to one canonical document Three feeds are normalised, assigned a canonical key, checked against a fingerprint store, merged by a field precedence rule and written once. ERP feed supplier feed scraper normalise case, space, unicode canonical key GTIN or fingerprint merge rule field precedence one document stable _id the document id is derived from the canonical key, never from the feed so the same entity from any feed addresses the same document
Deduplication is really id assignment. Once the id comes from the entity rather than the feed, the engine deduplicates for free on every write.

Solution Steps

1. Normalise before comparing anything

Comparison is only as good as the normalisation in front of it. Apply the same function everywhere, and keep it boring: Unicode form, case, whitespace, punctuation.

# normalise.py — deterministic, dependency-light, applied identically on every feed
import re, unicodedata

_WS = re.compile(r"\s+")
_PUNCT = re.compile(r"[^\w\s]", re.UNICODE)

def norm_text(value: str) -> str:
    v = unicodedata.normalize("NFKC", value or "")   # fi -> fi, full-width -> ASCII
    v = v.casefold().strip()                         # casefold beats lower() for ß, İ
    v = _PUNCT.sub(" ", v)
    return _WS.sub(" ", v).strip()

2. Prefer a real identifier over a computed fingerprint

If the domain has a natural key — GTIN, ISBN, MPN plus brand — use it. A computed fingerprint is a fallback for records that lack one, not a replacement for one that exists.

# canonical.py — natural key first, fingerprint second
import hashlib

def canonical_key(rec: dict) -> str:
    if rec.get("gtin"):
        return f"gtin:{rec['gtin'].strip()}"
    if rec.get("brand") and rec.get("mpn"):
        return f"mpn:{norm_text(rec['brand'])}|{norm_text(rec['mpn'])}"
    basis = "|".join(norm_text(rec.get(f, "")) for f in ("brand", "title", "size", "colour"))
    return "fp:" + hashlib.blake2b(basis.encode(), digest_size=16).hexdigest()

def document_id(rec: dict) -> str:
    return canonical_key(rec)      # the id IS the canonical key

3. Define field precedence instead of last-write-wins

When two feeds disagree, the newest write should not automatically win — the more authoritative source should. Encode that as a table so it is reviewable.

# merge.py — per-field source precedence, highest priority first
PRECEDENCE = {
    "price":       ["erp", "supplier_eu", "scraper"],
    "title":       ["supplier_eu", "erp", "scraper"],
    "description": ["supplier_eu", "scraper", "erp"],
    "stock":       ["erp"],                       # only the ERP may set stock
}

def merge(existing: dict, incoming: dict, field_sources: dict) -> dict:
    out = dict(existing)
    for field, value in incoming.items():
        order = PRECEDENCE.get(field)
        if not order:
            out[field] = value                     # unlisted fields: last write wins
            continue
        cur_src, new_src = field_sources.get(field), incoming["_source_feed"]
        if cur_src is None or order.index(new_src) <= order.index(cur_src):
            out[field] = value
            field_sources[field] = new_src
    return out

4. Catch near-duplicates that no key will match

Records that differ only by a word or two need similarity, not equality. MinHash over character shingles finds them at a cost that scales linearly rather than quadratically.

# nearlydup.py — MinHash signature, banded for candidate lookup
import hashlib

def shingles(text: str, k: int = 5) -> set[str]:
    t = norm_text(text)
    return {t[i:i + k] for i in range(max(1, len(t) - k + 1))}

def minhash(text: str, perms: int = 64) -> list[int]:
    sh = shingles(text)
    sig = []
    for seed in range(perms):
        sig.append(min(int(hashlib.blake2b(s.encode(), digest_size=8,
                                           salt=str(seed).encode()[:16]).hexdigest(), 16)
                       for s in sh))
    return sig

def similarity(a: list[int], b: list[int]) -> float:
    return sum(x == y for x, y in zip(a, b)) / len(a)   # estimates Jaccard

5. Route uncertain matches to review, not to a merge

Above a high threshold, merge automatically. Below a low one, keep both. In between, park the pair for a human — silently merging two genuinely different products is worse than showing two similar ones.

AUTO_MERGE, REVIEW = 0.92, 0.78

def decide(sim: float) -> str:
    if sim >= AUTO_MERGE:
        return "merge"
    if sim >= REVIEW:
        return "review"          # queue for a human; index both meanwhile
    return "distinct"
Similarity bands and the action taken in each Similarity below 0.78 keeps records distinct, 0.78 to 0.92 routes to human review, and above 0.92 merges automatically. distinct — keep both review queue merge automatically 0.0 0.78 0.92 1.0 Move the thresholds using a labelled sample, not intuition — the middle band's width is a staffing decision.
Three bands, three actions. The review band exists because a false merge destroys data while a false split merely looks untidy.

Verification

Confirm that the same entity from two feeds produces one document:

python3 - <<'PY'
from canonical import document_id
a = {"gtin": "0712345678905", "title": "Trail Runner 3 ", "_source_feed": "erp"}
b = {"gtin": "0712345678905", "title": "trail runner 3",  "_source_feed": "supplier_eu"}
print(document_id(a), document_id(b), document_id(a) == document_id(b))
PY
# => gtin:0712345678905 gtin:0712345678905 True

Confirm the index actually holds one copy after both feeds run:

curl -s 'localhost:9200/products/_search' -H 'Content-Type: application/json' -d '{
  "size": 0,
  "aggs": {"dupes": {"terms": {"field": "gtin", "min_doc_count": 2, "size": 10}}}
}' | jq '.aggregations.dupes.buckets'
# => []      (no GTIN appears on more than one document)

Confirm precedence held rather than last-write-wins:

curl -s 'localhost:9200/products/_doc/gtin:0712345678905?filter_path=_source.price,_source.title'
# => {"_source":{"price":129.0,"title":"Trail Runner 3"}}
#     price from the ERP feed, title from the supplier feed — as the table specifies

Common Pitfalls

Hashing the whole payload as the document id

A payload hash changes whenever any field changes, so an updated record becomes a new document and the old one lingers. This produces a slow, steady accumulation of orphans that looks like a duplication bug but is really an id-stability bug. The id must derive from identity fields only — the ones that define which entity this is — never from mutable content.

Normalising on read instead of on write

Applying normalisation at query time makes results look deduplicated while the index still holds duplicates, so facet counts, aggregations, and any corpus statistic remain wrong. Worse, the two code paths drift: the query-side normaliser gains a rule the ingestion side lacks, and matches quietly stop working. One normaliser, applied at ingestion, used by everything.

Merging without version guards

A merge reads the existing document, combines it with the incoming record, and writes the result — a read-modify-write that two workers can perform concurrently, each overwriting the other’s contribution. Guard the write with the sequence number read during the merge and retry on conflict, or the merged document will intermittently be missing fields that were definitely present in both inputs.

Operational notes

Deduplication is easiest to introduce on a rebuild rather than in place, because changing the id derivation changes every document’s address. The practical sequence is to build the deduplicating writer, use it to populate a shadow index, compare document counts and a sample of merged records against the live index, and then swap. Attempting the change in place leaves the old feed-keyed documents behind as orphans, which then have to be identified and deleted — more work than the rebuild would have been.

Track the duplicate rate per feed as an ingestion metric. A feed whose records collapse into existing documents 40% of the time is telling you something structural — usually that it overlaps another feed entirely and could be retired, or that its records lack the identifier everyone assumed they had. This number is also the cheapest possible regression test for a normalisation change: if a deploy moves it by more than a point or two, the normaliser changed behaviour.

Keep an audit trail of merges. Recording which source records contributed to each document turns “why does this product say 129 when the supplier says 119?” from an investigation into a lookup, and it is the only practical way to unwind a bad merge rule after the fact. A small _merged_from array of source ids and feed names on each document costs almost nothing in storage and repays itself the first time a merchandiser asks.

Budget for the fingerprint store the same way you budget for the index. A 50-million-document corpus with 64-permutation MinHash signatures needs roughly 25 GB before compression, which is a real infrastructure decision rather than an implementation detail. If that is too much, band the signature and store only the bands — candidate recall drops slightly and memory drops by an order of magnitude, which is usually the right trade for a catalog where the natural key already handles the majority of matches.

Which mechanism catches which share of duplicates Natural keys catch most duplicates, normalised fingerprints catch a smaller share, near-duplicate similarity catches the remainder, and a residue always needs human review. natural key (GTIN, MPN) — 71% fingerprint — 22% MinHash review Build the cheap mechanisms first — similarity search earns its cost only on the last few percent.
A measured breakdown from a retail catalog. Reaching for similarity search before exhausting natural keys is the classic sequencing mistake.