Signed HTTP callbacks for extraction events. Headers, signature verification, retries, and idempotency.

A webhook is an HTTPS endpoint you control that the API calls when something happens, most commonly when an extraction completes. Webhooks replace polling for any integration that wants real-time results without a busy loop.

Event types

Three event types ship today:

Subscribe to one or more at registration time:

{
  "url":    "https://api.example.com/dd-webhook",
  "events": ["extraction.completed", "extraction.failed"]
}

If you only want one event class, subscribe to one event. If you want "anything terminal," subscribe to artifact.processed.

How a webhook is selected for a job

At submit time, the platform picks the webhook to call by walking this priority list, top to bottom:

1. webhook_id on the request. A pre-registered webhook by ID. Highest precedence.
2. webhook_url on the request. A one-shot HTTPS URL. The signing secret is auto-generated and returned once in the submit response (webhook_signing_secret).
3. Pipeline default. A webhook bound to the pipeline that the API key resolves to.
4. API-key default. A webhook bound directly to the API key.
5. None. No callback is made; you must poll.

Most teams register one production webhook against the API key, then override per-request only for one-off destinations (e.g. a customer's tenant-specific endpoint).

Signing

Every delivery includes an HMAC-SHA256 signature in the X-Datadistillers-Signature header. Verify it on every inbound request; an unverified webhook is an open RCE channel.

The signed payload is t={timestamp}.{raw_body}, where timestamp is the Unix epoch seconds (also sent as X-Timestamp).

Headers on every delivery:

A complete verifier in Python:

import hmac, hashlib, time

def verify(secret: str, signature_header: str, body: str,
           tolerance: int = 300) -> bool:
    parts = dict(p.split('=', 1) for p in signature_header.split(','))
    ts, sig = int(parts['t']), parts['v1']

    if abs(time.time() - ts) > tolerance:
        return False                                      # replay window

    msg = f't={ts}.{body}'
    expected = hmac.new(secret.encode(), msg.encode(), hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, sig)

The same logic in Node:

import crypto from 'node:crypto';

export function verify(secret, signatureHeader, body, tolerance = 300) {
  const parts = Object.fromEntries(
    signatureHeader.split(',').map((p) => {
      const i = p.indexOf('=');
      return [p.slice(0, i), p.slice(i + 1)];
    }),
  );
  const ts = parseInt(parts.t, 10);
  const sig = parts.v1;

  if (Math.abs(Date.now() / 1000 - ts) > tolerance) return false;

  const msg = `t=${ts}.${body}`;
  const expected = crypto.createHmac('sha256', secret)
    .update(msg).digest('hex');

  return crypto.timingSafeEqual(Buffer.from(expected), Buffer.from(sig));
}

Secret rotation

POST /webhooks/{id}/rotate-secret issues a new signing secret and keeps the previous one valid for a 24-hour overlap window.

                                            ┌─ rotate ─┐
                  old secret  ─────────────▶│          │
                                            │  24h     │  new secret  ─────────▶
                                            └──────────┘

During the overlap, your verifier should accept either secret. After 24 hours the old secret stops working without warning. A safe verification loop:

def verify_either(current, previous, header, body):
    if verify(current, header, body):
        return True
    if previous and verify(previous, header, body):
        return True
    return False

The new secret is shown once in the rotate response, same as creation. Store it somewhere safe before discarding the response.

Retries and idempotency

A delivery is "successful" if your endpoint returns a 2xx within the configured timeout. Anything else (non-2xx, network error, timeout) is a failure and triggers a retry.

Schedule:

After attempt 4, delivery is abandoned. Inspect failures via GET /webhooks/{id}/deliveries or GET /job/{id}/webhook-logs.

The X-Webhook-Id header is the same on every retry. Use it as an idempotency key:

key = request.headers['X-Webhook-Id']
if seen.contains(key):
    return 200, ''                # already processed; ack
seen.add(key)
process(payload)

URL requirements

The platform refuses to register a webhook if the URL:

• Doesn't use HTTPS; plaintext is rejected outright.
• Resolves to a private IP, loopback, or cloud-metadata range; SSRF guard.
• Is longer than 2083 characters.

These checks run at registration and every URL change via PATCH /webhooks/{id}. If your endpoint sits behind a private VPC, expose a reverse proxy with a public DNS name; don't try to register the internal address.