# Evaluating your corpus

The repo ships an evaluation harness (`scripts/eval_extraction.py`) that
measures the platform **per stage** against your own ground truth: OCR recall
in isolation (no LLM cost), then extraction accuracy end-to-end. It is the
referee for every pipeline or template change.

## Prerequisites

- A running stack (the harness talks to the API for OCR and to your LLM
  endpoint for extraction).
- A corpus directory with one `*_truth.json` per document, e.g.
  `data/in/invoice-001.pdf` + `data/in/invoice-001_truth.json`.
- Host-side environment: `DATABASE_URL` pointing at the stack's Postgres, and
  `CONNECTION_ENCRYPTION_KEYS` from your `.env` if you use `--from-infra`.

## 1. Fetch — OCR the corpus once and cache canonicals

```bash
python scripts/eval_extraction.py fetch \
  --api-url http://localhost:8080 --api-key rk_... \
  --data-dir data/in --cache-dir data/eval_cache
```

Submits one canonical-mode batch, polls to terminal, and caches every
`CanonicalOcrDocument`. Re-attach to an interrupted fetch with `--batch-id`.
After a pipeline upgrade, re-OCR the same corpus with
`POST /v1/batches/{batch_id}/reprocess` (or the **Reprocess all** button on
the batch detail page) and re-run `fetch --batch-id` to refresh the cache —
no resubmission, no dedup interference.

## 2. OCR audit — is the value even in the canonical?

```bash
python scripts/eval_extraction.py ocr-audit --data-dir data/in --cache-dir data/eval_cache
```

Checks every populated ground-truth value for presence in the cached
canonicals with recall-biased matching (European number formats, date
separators and prose months, separator-tolerant identifiers). **No LLM
involved** — a `MISSING` here is an OCR-stage problem, and no template
iteration can fix it. Fix OCR first; everything downstream is capped by this
table.

## 3. Run — extraction accuracy against ground truth

```bash
python scripts/eval_extraction.py run \
  --data-dir data/in --cache-dir data/eval_cache \
  --template my_templates/my-invoice.json \
  --from-infra --max-concurrent 4 \
  --report-out eval/reports/run-001.json
```

`--from-infra` resolves the LLM URL, served model, backend, credentials, and
`max_model_len` from the platform `infra_endpoints` row — the same source of
truth the worker uses, so the harness can never silently disagree with
production about context-window budgets. (Without it, pass `--llm-url`,
`--llm-model`, `--backend`, `--llm-api-key`, and export
`LLM_CONTEXT_WINDOW` to match your model.)

The report scores every field with schema-aware rules and prints per-field
`pct_ok`, the document PASS/PARTIAL/FAIL split, and the decision
distribution (`accept` vs `review`).

For production invoice evals, leave the 2.4.1 extraction-quality defaults on:
`EXTRACTION_POSTPROCESSING_ENABLED=true`,
`EXTRACTION_EVIDENCE_PLANNING_ENABLED=true`,
`EXTRACTION_CHUNK_MERGE_STRATEGY=field_aware`, and
`TAX_ID_COUNTRY_PREFIX_MODE=infer`. These match the worker path: long/noisy
documents use focused evidence pages, chunked runs merge totals/tax recap fields
with field-aware rules, and normalized values keep raw-value audit metadata in
the extraction payload.

## 4. Iterating a template without overfitting (split protocol)

Tuning a template against the same documents you score on will overfit —
the numbers go up while real-world accuracy doesn't. Use a train/holdout
split:

1. Split the corpus deterministically (e.g. sort filenames, alternate
   halves).
2. Iterate the template **only against the train half** — by hand, or with
   the template forge (give the forge only the train files + their ground
   truth).
3. Score each candidate on both halves with `--only`:

```bash
python scripts/eval_extraction.py run ... --only "inv-001*,inv-003*,inv-005*" --report-out train.json
python scripts/eval_extraction.py run ... --only "inv-002*,inv-004*,inv-006*" --report-out holdout.json
```

4. Accept a candidate **only when the holdout improves too**. A
   train-only improvement is memorisation, not a better template.

## Reading the results

- **`ocr-audit` recall low** → OCR stage: check layout policy, page ranges,
  source quality. Template work cannot help.
- **`run` conversion low on OCR-present values** → extraction stage: template
  semantics, schema, prompt overlay.
- **decisions all `review`** → calibration: look at `flag_kinds` in the
  report (confidence thresholds, validation rules, judge flags).