# Scientific evaluation and promotion gates

This directory is the evidence boundary for changing DocStack's visual OCR
profiles. A candidate is never promoted from a global average or an ad-hoc
benchmark. It must process the same frozen bytes and RenderBundle surfaces as
the serving baseline, pass paired statistical gates overall and in every
declared corpus stratum, and carry an executable rollback contract.

The tooling is provider-neutral. It imports no OCR engine and consumes only
immutable JSON artifacts.

## Current baseline registry

`manifests/legacy-dual-vlm-v1.json` freezes the evidence that exists for the
current 62-document corpus:

- 62 source-artifact SHA-256 hashes and byte counts;
- 62 ground-truth-source SHA-256 hashes and byte counts;
- deterministic, stratum- and leakage-group-based train/dev/held-out membership
  (37/13/12);
- the profile name `legacy-dual-vlm-v1`.

It deliberately does **not** invent missing evidence. Language, layout,
difficulty, and security annotations are marked pending; leakage groups use
exact source hashes until reviewed source-family metadata exists; renderer,
model, prompt, and codec revisions are pending; no canonical baseline artifact
or standardized visual golden was present to hash. The registry therefore
passes structural validation but fails promotion-readiness validation. That is
the correct result—not a reason to weaken the gate.

Corpus bytes remain outside Git. The committed registry proves which bytes are
intended without redistributing documents or labels.

## Contracts

The versioned corpus schema is `schemas/corpus-manifest-v1.schema.json`.
Every document declares:

- a stable ID, split, leakage group, and the basis for that group;
- format, language(s), layout tag(s), difficulty, and security tag(s);
- annotation status and available ground-truth surfaces;
- hashed source, golden, canonical, or other artifacts.

Promotion-ready manifests require all three splits, reviewed strata, complete
annotations, a standardized `evaluation_golden` artifact per document, and a
complete baseline profile pin. Validation also rejects:

- duplicate document IDs;
- a leakage group in more than one split;
- identical role-specific artifact hashes in more than one split;
- registry-digest drift;
- path traversal, missing files, wrong byte counts, or wrong hashes when
  artifact verification is enabled.

Exact-hash grouping is a safe registration fallback, not a claim that
near-duplicate families have been reviewed. Supply reviewed metadata before a
held-out experiment.

Standardized goldens use `docstack.evaluation-golden/v1` and may contain
transcript, surfaces/assets, labeled boxes, reading order, table grids/spans,
formulas, template values, critical and numeric-critical field paths,
source-grounding IDs, and an adjudicated `accept`/`needs_review` decision.
Unannotated sections are omitted and therefore not silently scored as perfect.
An annotated section missing from a candidate is scored as a failure/omission.

Prediction runs use `docstack.prediction-run/v1`. Each document carries the
frozen source hash, RenderBundle hash, prediction, operational measurements,
and an explicit failure. A run must cover the complete requested split; failed
or partial documents cannot be dropped from the denominator.

## Metrics

`evaluation.metrics` computes document-paired measurements for:

- surface and asset retention;
- character error rate (CER), word error rate (WER), omission, and unsupported
  or hallucinated tokens;
- label-aware layout F1/IoU and normalized reading-order accuracy;
- table cell and topology accuracy plus normalized formula exactness;
- schema validity, extraction precision/recall/F1/exact accuracy, critical
  fields, critical numeric accuracy, and grounding overlap;
- review-decision accuracy plus review, false-accept, and false-review rates;
- failure/retry rate, latency, throughput, CPU/GPU memory, GPU-seconds, tokens,
  pixels, and cost.

Review metrics use the fixed split as their denominator. A false accept is a
document adjudicated `needs_review` that the candidate marks `accept`; a false
review is an adjudicated `accept` marked `needs_review`. Missing or unknown
review decisions are contract errors and cannot disappear from either rate.

`evaluation.statistics` reports paired baseline and candidate means, medians,
p95 values, oriented improvement (positive is always better), deterministic
paired-bootstrap confidence intervals, and paired sign-randomization p-values.
The seed, confidence level, and sample counts are policy inputs and appear in
the immutable policy artifact.
Metric runs pin `docstack-scientific-evaluator/1` and a hash of every metric's
direction, unit, and bounds. Baseline and candidate reports produced by a
different scoring contract cannot be compared.

## Promotion policy

`policies/primary-lane-promotion-v1.json` is the predeclared initial replacement
policy. It requires shadow operation, identical RenderBundles, complete held-out
coverage, no critical-field/omission/hallucination regression, quality floors,
false-accept and false-review ceilings, and bounded latency/resource/cost
regressions.

Every metric rule runs once overall and once for every observed value of the
requested format, languages, layouts, difficulty, and security dimensions.
One failed stratum blocks promotion even when the overall mean passes. The
decision JSON records every check, confidence interval, p-value, artifact/run
digest, protocol violation, rollback target, and a final `promote` or `block`.
The policy also declares mandatory held-out strata; a missing PDF/PPTX/XLSX,
target-language, hard/adversarial, layout, or security cohort blocks the
protocol before averages are considered. The production policy requires at
least two paired held-out documents in every mandatory stratum; larger corpora
can raise that floor without changing the evaluator.

Parallel or cascade policies should set `require_superiority: true` on their
predeclared quality-gain metric and retain explicit overhead limits. This makes
the extra latency/GPU/reconciliation cost earn its place rather than relying on
an untested architecture preference.

## Rollback

Promotion policy also declares production triggers. Telemetry uses
`docstack.shadow-telemetry/v1`, with paired candidate/baseline values overall
and by stratum. Triggers support absolute candidate thresholds and absolute or
relative paired regression. A stratum trigger must breach in the **same**
stratum for the declared number of trailing windows. Missing critical
telemetry triggers rollback when `on_missing` is `trigger`.

The rollback decision identifies the profile to restore and preserves the
paired evidence for incident analysis.

## CLI

Show all commands and examples:

```bash
python scripts/eval_promotion.py --help
```

Freeze only available corpus evidence:

```bash
python scripts/eval_promotion.py register \
  --corpus-dir data/in \
  --manifest-id legacy-dual-vlm-v1-62 \
  --baseline-profile legacy-dual-vlm-v1 \
  --expected-count 62 \
  --split-seed legacy-dual-vlm-v1-62 \
  --created-at 2026-08-03T00:00:00+00:00 \
  --output eval/manifests/legacy-dual-vlm-v1.json
```

With the same mounted bytes, that command reproduces the committed manifest
byte-for-byte. A new corpus revision must use a new manifest ID, seed, and
timestamp instead of rewriting this registry.

Reviewed metadata can be supplied with `--metadata`. It is keyed by exact
source filename; omitted entries remain pending:

```json
{
  "documents": {
    "sample.pdf": {
      "leakage_group": "issuer-family-17",
      "strata": {
        "format": "pdf",
        "languages": ["ar", "en"],
        "layouts": ["mixed-direction", "table-heavy"],
        "difficulty": "hard",
        "security": ["sensitive"]
      },
      "annotations": {
        "status": "complete",
        "available": ["transcription", "layout", "tables", "extraction", "grounding", "critical-numeric", "review"],
        "reviewed_by": "evaluation-board-2026-08"
      },
      "evaluation_golden": "evidence/goldens/sample.golden.json"
    }
  }
}
```

Registration uses iterative multi-label stratification while keeping each
reviewed leakage group indivisible. The requested 60/20/20 document quotas are
exact when group sizes permit; label distributions are balanced as closely as
the group boundary permits.
Goldens can instead be discovered in bulk with `--golden-dir`; accepted names
are `<source filename>.golden.json` and `<source stem>.golden.json`.

Validate the committed registry structure without requiring local corpus
bytes:

```bash
python scripts/eval_promotion.py validate-manifest \
  --manifest eval/manifests/legacy-dual-vlm-v1.json \
  --expected-count 62
```

On the controlled evaluation host, verify every byte and demand readiness:

```bash
python scripts/eval_promotion.py validate-manifest \
  --manifest eval/manifests/legacy-dual-vlm-v1.json \
  --expected-count 62 --verify-artifacts --require-labels
```

Score serving baseline and shadow candidate predictions:

```bash
python scripts/eval_promotion.py score \
  --manifest eval/manifests/corpus-v2.json \
  --predictions /evidence/baseline-shadow.json \
  --artifact-root /evidence/root \
  --output /evidence/baseline-metrics.json

python scripts/eval_promotion.py score \
  --manifest eval/manifests/corpus-v2.json \
  --predictions /evidence/candidate-shadow.json \
  --artifact-root /evidence/root \
  --output /evidence/candidate-metrics.json
```

Create the promotion decision (exit 1 means blocked):

```bash
python scripts/eval_promotion.py compare \
  --manifest eval/manifests/corpus-v2.json \
  --baseline-run /evidence/baseline-metrics.json \
  --candidate-run /evidence/candidate-metrics.json \
  --policy eval/policies/primary-lane-promotion-v1.json \
  --output /evidence/promotion-decision.json
```

Evaluate production rollback windows:

```bash
python scripts/eval_promotion.py rollback \
  --telemetry /evidence/shadow-windows.json \
  --policy eval/policies/primary-lane-promotion-v1.json \
  --output /evidence/rollback-decision.json
```

Run the deterministic end-to-end fixture used by CI:

```bash
python scripts/eval_promotion.py smoke
```

Never tune a threshold, prompt, renderer, or route using the held-out result.
Create a new versioned manifest/policy when the corpus or predeclared decision
contract changes; do not rewrite an artifact that already supported a decision.