Evaluating your corpus

The repo ships an evaluation harness (scripts/eval_extraction.py) that measures the platform per stage against your own ground truth: OCR recall in isolation (no LLM cost), then extraction accuracy end-to-end. It is the referee for every pipeline or template change.

Prerequisites

• A running stack (the harness talks to the API for OCR and to your LLM endpoint for extraction).
• A corpus directory with one *_truth.json per document, e.g. data/in/invoice-001.pdf + data/in/invoice-001_truth.json.
• Host-side environment: DATABASE_URL pointing at the stack’s Postgres, and CONNECTION_ENCRYPTION_KEYS from your .env if you use --from-infra.

1. Fetch — OCR the corpus once and cache canonicals

python scripts/eval_extraction.py fetch \
  --api-url http://localhost:8080 --api-key rk_... \
  --data-dir data/in --cache-dir data/eval_cache

Submits one canonical-mode batch, polls to terminal, and caches every CanonicalOcrDocument. Re-attach to an interrupted fetch with --batch-id. After a pipeline upgrade, re-OCR the same corpus with POST /v1/batches/{batch_id}/reprocess (or the Reprocess all button on the batch detail page) and re-run fetch --batch-id to refresh the cache — no resubmission, no dedup interference.

2. OCR audit — is the value even in the canonical?

python scripts/eval_extraction.py ocr-audit --data-dir data/in --cache-dir data/eval_cache

Checks every populated ground-truth value for presence in the cached canonicals with recall-biased matching (European number formats, date separators and prose months, separator-tolerant identifiers). No LLM involved — a MISSING here is an OCR-stage problem, and no template iteration can fix it. Fix OCR first; everything downstream is capped by this table.

3. Run — extraction accuracy against ground truth

python scripts/eval_extraction.py run \
  --data-dir data/in --cache-dir data/eval_cache \
  --template my_templates/my-invoice.json \
  --from-infra --max-concurrent 4 \
  --report-out eval/reports/run-001.json

--from-infra resolves the LLM URL, served model, backend, credentials, and max_model_len from the platform infra_endpoints row — the same source of truth the worker uses, so the harness can never silently disagree with production about context-window budgets. (Without it, pass --llm-url, --llm-model, --backend, --llm-api-key, and export LLM_CONTEXT_WINDOW to match your model.)

The report scores every field with schema-aware rules and prints per-field pct_ok, the document PASS/PARTIAL/FAIL split, and the decision distribution (accept vs review).

For production invoice evals, leave the 2.4.1 extraction-quality defaults on: EXTRACTION_POSTPROCESSING_ENABLED=true, EXTRACTION_EVIDENCE_PLANNING_ENABLED=true, EXTRACTION_CHUNK_MERGE_STRATEGY=field_aware, and TAX_ID_COUNTRY_PREFIX_MODE=infer. These match the worker path: long/noisy documents use focused evidence pages, chunked runs merge totals/tax recap fields with field-aware rules, and normalized values keep raw-value audit metadata in the extraction payload.

4. Iterating a template without overfitting (split protocol)

Tuning a template against the same documents you score on will overfit — the numbers go up while real-world accuracy doesn’t. Use a train/holdout split:

1. Split the corpus deterministically (e.g. sort filenames, alternate halves).
2. Iterate the template only against the train half — by hand, or with the template forge (give the forge only the train files + their ground truth).
3. Score each candidate on both halves with --only:

python scripts/eval_extraction.py run ... --only "inv-001*,inv-003*,inv-005*" --report-out train.json
python scripts/eval_extraction.py run ... --only "inv-002*,inv-004*,inv-006*" --report-out holdout.json

4. Accept a candidate only when the holdout improves too. A train-only improvement is memorisation, not a better template.

Reading the results

• ocr-audit recall low → OCR stage: check layout policy, page ranges, source quality. Template work cannot help.
• run conversion low on OCR-present values → extraction stage: template semantics, schema, prompt overlay.
• decisions all review → calibration: look at flag_kinds in the report (confidence thresholds, validation rules, judge flags).