Verifying Webhook Signatures for Search Sync

A webhook endpoint that writes straight into your search index is a public, unauthenticated write API unless you verify signatures — and most implementations verify them incorrectly in one of four specific ways. This page implements HMAC verification the way it survives production: over the raw body, with a timestamp window, constant-time comparison, and a rotation path that does not require a synchronized deploy. It is part of webhook-driven sync patterns within the data ingestion and synchronization pipelines area.

Prerequisites

  1. A shared secret from the sending system, stored in a secret manager and never in the repository.
  2. A web framework from which you can read the raw request body before any JSON parsing.
  3. Somewhere to record recently seen signatures for replay defence — Redis with a TTL is enough.
  4. A sender that includes both a signature and a timestamp header; if it does not, replay defence is impossible and you need a different transport.

Diagnosis: the four ways verification is usually wrong

Signature verification looks like ten lines of code, and each of the four common mistakes leaves the endpoint fully exploitable while appearing to work in tests.

Signing the re-serialized body is the most common. The framework parses JSON, your handler re-encodes it to compute the HMAC, and key order or whitespace differs from what the sender signed — so verification fails for legitimate requests. The usual “fix” is to skip verification for requests that fail, which removes it entirely.

Comparing with == leaks timing information; over enough requests an attacker can recover a valid signature byte by byte. Omitting the timestamp check means a captured request stays valid forever, so an attacker who observes one legitimate delete can replay it indefinitely. And accepting the signature header without pinning the algorithm allows an attacker to specify a weaker one.

A rejected-request log makes the raw-body problem obvious once you know to look for it:

WARN  webhook signature mismatch  path=/hooks/products
      expected=sha256=6f1c…  computed=sha256=9ab3…
      raw_len=412  parsed_reserialized_len=398
#                  ^^^ 14 bytes of whitespace the sender signed and we discarded
Signature verification pipeline for an inbound webhook The raw body and timestamp are checked for freshness, the HMAC is recomputed and compared in constant time, the signature is recorded against replay, and only then is the payload parsed and indexed. raw request bytes, unparsed timestamp ±5 min window HMAC compare constant time replay check seen-signature set parse + index trusted now any failure → 401, log, and drop — never fall through to the handler parsing before verification is the bug that makes all of this decorative
Verification happens on bytes, before parsing. Every stage to the left of "parse" operates on data you do not yet trust.

Solution Steps

1. Capture the raw body before anything parses it

This is framework-specific and it is the step that makes the rest correct.

# app.py — Flask: read raw bytes, verify, then parse yourself
from flask import Flask, request, abort
import json

app = Flask(__name__)

@app.post("/hooks/products")
def products_hook():
    raw = request.get_data()                      # bytes, exactly as received
    sig = request.headers.get("X-Signature-256", "")
    ts = request.headers.get("X-Signature-Timestamp", "")
    if not verify(raw, sig, ts):
        abort(401)
    payload = json.loads(raw)                     # parse only after verifying
    enqueue_index_update(payload)
    return "", 202                                # accept fast; index asynchronously

2. Bind the timestamp into the signed material

Signing the body alone allows replay. Signing timestamp.body and checking the window makes a captured request expire.

# verify.py — HMAC over "<timestamp>.<raw body>" with a freshness window
import hashlib, hmac, time

MAX_SKEW_S = 300     # 5 minutes; tighten if sender and receiver clocks are NTP-synced

def expected_signature(secret: bytes, raw: bytes, ts: str) -> str:
    signed = ts.encode() + b"." + raw
    return "sha256=" + hmac.new(secret, signed, hashlib.sha256).hexdigest()

def fresh(ts: str) -> bool:
    try:
        return abs(time.time() - int(ts)) <= MAX_SKEW_S
    except (TypeError, ValueError):
        return False

3. Compare in constant time, and pin the algorithm

hmac.compare_digest takes the same time whether the first byte differs or the last. Refuse any prefix other than the one you expect.

def verify(raw: bytes, sig: str, ts: str) -> bool:
    if not sig.startswith("sha256="):     # pin the algorithm; never read it from the header
        return False
    if not fresh(ts):
        return False
    for secret in active_secrets():       # supports rotation — see step 5
        if hmac.compare_digest(sig, expected_signature(secret, raw, ts)):
            return True
    return False

4. Reject replays with a short-lived seen-set

The timestamp window bounds replay to five minutes; a seen-set closes it entirely within that window.

# replay.py — Redis SET NX with a TTL slightly longer than the skew window
def first_time_seen(redis, sig: str, ttl_s: int = 600) -> bool:
    # Returns True only for the first request carrying this signature.
    return bool(redis.set(f"hook:sig:{sig}", "1", nx=True, ex=ttl_s))

5. Rotate secrets without a synchronized deploy

Accept a list of active secrets. Rotation becomes: add the new secret, wait for the sender to switch, remove the old one — three independent deploys, no coordinated cutover.

# secrets.py — the ordering matters: newest first, so the common case verifies fastest
def active_secrets() -> list[bytes]:
    return [s.encode() for s in (
        os.environ["WEBHOOK_SECRET_CURRENT"],
        *( [os.environ["WEBHOOK_SECRET_PREVIOUS"]]
           if os.environ.get("WEBHOOK_SECRET_PREVIOUS") else [] ),
    )]
Three-step secret rotation with an overlap window The receiver accepts both secrets, then the sender switches to the new secret, then the old secret is removed, with no synchronized deploy required. 1. receiver accepts both old + new secret 2. sender switches signs with new secret 3. drop the old after zero old-key hits Instrument which secret verified each request, so step 3 is driven by evidence rather than by a guess. A single-secret receiver forces steps 1–3 into one simultaneous deploy, which is where outages come from.
Accepting a list rather than a single secret turns rotation from a coordinated cutover into three ordinary deploys.

Verification

Send a correctly signed request and confirm it is accepted:

SECRET='test-secret'; TS=$(date +%s); BODY='{"id":"sku-1","op":"upsert"}'
SIG="sha256=$(printf '%s.%s' "$TS" "$BODY" | openssl dgst -sha256 -hmac "$SECRET" -r | cut -d' ' -f1)"
curl -s -o /dev/null -w '%{http_code}\n' -X POST localhost:5000/hooks/products \
  -H "X-Signature-256: $SIG" -H "X-Signature-Timestamp: $TS" \
  -H 'Content-Type: application/json' -d "$BODY"
# => 202

Confirm a stale timestamp is rejected even with a valid signature:

TS=$(( $(date +%s) - 3600 ))
SIG="sha256=$(printf '%s.%s' "$TS" "$BODY" | openssl dgst -sha256 -hmac "$SECRET" -r | cut -d' ' -f1)"
curl -s -o /dev/null -w '%{http_code}\n' -X POST localhost:5000/hooks/products \
  -H "X-Signature-256: $SIG" -H "X-Signature-Timestamp: $TS" -d "$BODY"
# => 401   (signature is valid; the request is simply too old)

Confirm a replay of a fresh, valid request is rejected the second time:

for i in 1 2; do
  curl -s -o /dev/null -w "%{http_code} " -X POST localhost:5000/hooks/products \
    -H "X-Signature-256: $SIG" -H "X-Signature-Timestamp: $TS" -d "$BODY"
done; echo
# => 202 401

Common Pitfalls

Body parsers that consume the stream before your handler runs

Many frameworks read and discard the raw body as middleware, so request.body in the handler is already a parsed object. Express needs express.raw({type: 'application/json'}) on the webhook route specifically, and ASGI apps often need the body cached explicitly. Test this by signing a payload with unusual whitespace: if verification passes for compact JSON and fails for pretty-printed JSON, you are hashing a re-serialization.

Doing the index write inside the request

Webhook senders retry aggressively on timeouts, so a handler that performs a bulk index write inline turns a slow index into a flood of duplicate deliveries. Verify, enqueue, and return 202 in single-digit milliseconds; let the queue absorb the indexing work and the retries covered in handling webhook retries in search sync.

Logging the signature or the raw body at info level

Signatures are not secrets, but bodies frequently contain personal data, and a log line holding both makes replay trivial for anyone with log access. Log the truncated signature prefix and a payload hash instead — enough to correlate a delivery, not enough to reconstruct or replay it.

Operational notes

Treat the verification helper as security-critical code: one implementation, one set of tests, imported by every endpoint that accepts a webhook. The failure mode this prevents is subtle — a second endpoint added months later copies eighty per cent of the original code and omits the timestamp check, and nothing about the copy looks wrong in review. A shared helper with a test that asserts each of the four defences independently makes the omission impossible rather than merely unlikely.

Emit a counter for every rejection reason — bad signature, stale timestamp, replay, unknown secret — because the shape of the failures tells you what is happening. A burst of stale-timestamp rejections is almost always clock drift on your own hosts rather than an attack, and it resolves with NTP rather than with an incident. A burst of bad-signature rejections immediately after a deploy usually means the secret was rotated on one side only.

Keep an eye on the split between secrets during a rotation window. Once the counter for the previous secret has been zero for a full retry cycle of the sender — typically a day, because failed deliveries retry over hours — removing it is safe. Removing it earlier drops exactly the deliveries that were queued for retry under the old key, which is the one case rotation is supposed to protect.

It is also worth deciding, explicitly, what an unverifiable request should do to your monitoring. Returning 401 is correct, but a sender that has been misconfigured for a week will generate a steady stream of them and the search index will be silently stale the whole time. Pair the rejection counter with a positive one — successful deliveries per sender per hour — and alert on the absence of successes rather than only on the presence of failures. Absence-of-signal alerts are the ones that catch the failure mode where nothing is obviously broken and nothing is arriving either.

Rejection reasons and what each one usually means Stale timestamps usually indicate clock drift, bad signatures usually indicate a rotation mismatch, and replays usually indicate sender retries. stale timestamp → check NTP first bad signature → rotation mismatch replay rejected → normal sender retry Label the counter by reason: the label, not the count, is what shortens the investigation. Also alert on zero successful deliveries — silence is the failure nobody notices.
Three rejection reasons with three different owners. Collapsing them into one "webhook errors" counter throws away the diagnosis.

Verification is also the natural place to enforce a payload size limit. An endpoint that reads an unbounded body before checking anything can be exhausted by a single large request from an unauthenticated caller, and the signature check does not help because it happens after the read. Cap the body at a size comfortably above the largest legitimate payload, reject anything larger before hashing, and the endpoint stops being a denial-of-service surface as well as an authentication one.