Batch. DataDistill Docs

reference

Batch #

POST /batch: submit up to 50 documents in one request, with shared template, schema, and webhook config.

POST /batch submits up to 50 files in a single request. Each file gets an independent presigned upload URL; push them concurrently for maximum throughput. All files share the same template, schema, and webhook configuration.

Like the single-file POST /extract flow, submitting only provisions the upload URLs. The backend does not listen for S3 events, so after every PUT has returned you must call POST /artifacts/confirm-batch-upload/{batch_id} to enqueue the jobs.

Method	Path	Auth	Returns
`POST`	`/api/v1/batch`	HTTP Basic	`202 Accepted` + `BatchSubmitResponse`

Request body

json

{
  "files": [
    { "filename": "invoice-001.pdf", "artifact_type": "application/pdf", "artifact_size": 184320 },
    { "filename": "invoice-002.pdf", "artifact_type": "application/pdf", "artifact_size": 192011 }
  ],
  "template_id":      "tpl_invoice_v3",
  "extraction_schema": null,
  "webhook_id":       null,
  "webhook_url":      null
}

Field	Type	Required	Notes
`files`	array	yes	1–50 entries. Each must be unique by `filename`.
`template_id`	string	no	Saved template. Mutually exclusive with `extraction_schema`.
`extraction_schema`	object	no	Inline schema. Mutually exclusive with `template_id`.
`webhook_id`	string	no	Pre-registered webhook for all files.
`webhook_url`	string	no	One-shot HTTPS URL. Signing secret returned once.

Per-file descriptor (BatchFileDescriptor):

Field	Type	Required	Notes
`filename`	string	yes	Must be unique within the batch.
`artifact_type`	string	yes	MIME type. Signed into the upload URL.
`artifact_size`	integer	yes	File size in bytes.
`retention_policy`	enum	no	Per-file. Defaults to `30d`.

Duplicate filenames within a single batch are rejected with 400.

Response (`202 Accepted`)

json

{
  "batch_id": "bat_z9y8x7",
  "total":    2,
  "files": [
    {
      "filename":    "invoice-001.pdf",
      "job_id":      "job_aaaa",
      "artifact_id": "art_aaaa",
      "upload_url":  "https://s3.amazonaws.com/…?X-Amz-Signature=…",
      "fields":      {}
    },
    {
      "filename":    "invoice-002.pdf",
      "job_id":      "job_bbbb",
      "artifact_id": "art_bbbb",
      "upload_url":  "https://s3.amazonaws.com/…?X-Amz-Signature=…",
      "fields":      {}
    }
  ],
  "webhook_signing_secret": null
}

Field	Notes
`batch_id`	Groups all files in the dashboard and webhook payloads.
`total`	Echo of `files.length`.
`files[]`	Per-file IDs and presigned URL.
`webhook_signing_secret`	Set only if `webhook_url` was passed. Shown once.

Uploading the files

Each upload_url is independent. Push them in parallel.

bash

cat batch.json | jq -r '.files[] | "\(.upload_url)\t\(.filename)"' | \
  xargs -P 8 -n 1 sh -c '
    URL=$(echo "$0" | cut -f1)
    FILE=$(echo "$0" | cut -f2)
    curl -X PUT "$URL" -H "Content-Type: application/pdf" --upload-file "$FILE"
  '

For Python and JavaScript variants, see Batch processing.

Confirming the batch

After the PUTs finish, queue every uploaded file in one call:

bash

curl -u "$DD_KEY:$DD_SECRET" \
  -X POST https://api.datadistillers.com/api/v1/artifacts/confirm-batch-upload/bat_z9y8x7

The platform flips each successfully uploaded artifact to uploaded, sets its job to queued_for_processing, and dispatches the workers. Artifacts that never uploaded remain in pending_upload. Confirm is idempotent — a second call after a partial upload picks up the stragglers without re-queueing earlier successes. Until this call, every job stays in pending and the platform takes no further action.

Tracking completion

Two patterns:

Webhook: register one webhook and let it deliver per-job events. The payload includes batch_id for grouping.

Polling: iterate over the per-file job_id values and poll /job/{id} (or /artifacts/{id}/status for a lighter response) until each is terminal. See Monitoring jobs.

Errors

Status	Cause
`400`	Empty `files`, > 50 files, duplicate filenames, both `template_id` and `extraction_schema`.
`401`	Auth missing or invalid.
`402`	Wallet frozen or `spendable` insufficient for the estimated batch hold.
`404`	`template_id` or `webhook_id` not found.
`422`	Body shape didn't validate.

A 402 is "all-or-nothing"; the platform won't queue partial batches. Either the entire batch is accepted or none of it is. Reduce the file count or add wallet funds.

← PreviousArtifacts Next →Templates