# Vision execution operations runbook

# Vision execution operations runbook

This runbook is the production companion to
[`docstack-next-generation-architecture-research.md`](/docstack-next-generation-architecture-research/)
§23. It covers scrape discovery, telemetry acceptance, alert response, safe
diagnostics, and rollback for the render-first visual execution plane.

## Operating boundary

DocStack has two independent Prometheus registries because the API and SAQ
worker are separate processes:

| Registry | Default endpoint | Owns | Exposure rule |
|---|---|---|---|
| API | `api:8080/metrics` | HTTP, queue, provider-health, and API-owned pipeline metrics | Cluster-internal scrape; preserve the deployment's existing gateway policy |
| Worker | `worker:9090/metrics` | Worker pipeline and `docstack_vision_*` metrics | Cluster-internal scrape only; never add it to public Ingress or Gateway routes |

The worker listener is unauthenticated by design and contains aggregate,
bounded-cardinality operational data. It starts before runtime initialization,
fails worker startup if its configured address cannot bind, and is shut down
and joined during worker termination. Set `WORKER_METRICS_ENABLED=false` only
when another in-process collector owns the same registry; disabling the only
worker scrape path removes the production vision telemetry contract.

Metric labels must never contain tenant identifiers, job identifiers, document
content, extracted values, filenames, prompts, responses, presigned URLs, or
free-form errors. Per-job content and provenance belong in tenant-authorized,
retention-controlled diagnostics—not Prometheus.

## Configure and hot-reload model providers

DocStack fixes the semantic model contracts used by its load-bearing lanes but
does not fix the infrastructure vendor that serves them. The active source of
truth is the revisioned `infra_endpoints` table, not process environment:

| Role | Required behavior | Supported binding adapters |
|---|---|---|
| `paddleocr_vl` | Paddle full lane using the accepted PaddleOCR-VL-1.6 contract | `vllm`, `openai` |
| `granite_docling_vl` | Docling full lane using Granite Docling, DocTags, and the declared prompt/codec contract | `local`, `vllm`, `ollama`, `openai` |
| `glmocr_vl` | task-bounded GLM-OCR text/table/formula recognition | `vllm`, `ollama`, `openai` |
| `llm` | template-driven extraction, drafting, judging, and characterization | `vllm`, `ollama`, `openai`, `gemini`, `anthropic` |

Environment values seed any missing platform-default rows at first boot. They
do not overwrite a row on restart. A platform administrator manages defaults
at `/admin/platform/infrastructure`; an enabled tenant administrator manages
overrides at `/admin/infrastructure`. Tenant resolution is override first and
platform default second. Credentials are encrypted with
`CONNECTION_ENCRYPTION_KEYS`; API responses show only whether a credential is
present and its non-secret header name.

For each role:

1. Select the protocol adapter, base URL, provider deployment/routing alias,
   credential-header name and value, concurrency, and timeout.
2. Choose **Test draft**. This tests the unsaved form through the same
   validation path and has no upsert, commit, audit-state, or invalidation
   effect. A successful connectivity/shape probe proves the advertised alias
   is reachable; it is not independent proof that a vendor has mapped that
   alias to the claimed immutable model weights.
3. Choose **Apply**. Apply compares the displayed revision with active state,
   validates again, commits and audits the next revision, and publishes an
   eager invalidation. A concurrent edit returns HTTP 409; refresh rather than
   overwriting it.
4. Submit a canary and inspect its execution diagnostics. The plan must show
   the expected binding ID, source, provider alias, public fingerprint, and
   configuration revision. It must not contain the credential value.

The consistency boundary is plan creation. Every new job performs a fresh
effective-binding resolution and freezes the four results for the complete
attempt. A worker rebuilds affected clients when a binding fingerprint changes,
including after a missed Redis notification. An already created or in-flight
plan deliberately continues on its original revision. “Immediate” therefore
means **the next plan**, not mutation of work already executing.

The equivalent API operations are:

```text
POST /v1/admin/platform/infra/{role}/probe
PUT  /v1/admin/platform/infra/{role}
POST /v1/admin/tenants/{tenant_id}/infra/{role}/probe
PUT  /v1/admin/tenants/{tenant_id}/infra/{role}
```

The body carries `backend_kind`, `base_url`, `served_model`, optional
`auth_header_name`, optional write-only `auth_header`, `max_concurrent`,
`timeout_seconds`, and `expected_revision`. `local://baked` is valid only for
the Granite/Docling local adapter. Remote OpenAI-wire URLs may be entered with
or without a trailing `/v1`; non-`/v1` proxy path prefixes are preserved. URL
credentials, query strings, parameters, fragments, whitespace, and control
characters are rejected. Credential values are printable ASCII and baked-local
bindings cannot carry one.

### Azure AI Foundry and self-hosted providers

For Azure's OpenAI-compatible v1 surface, use protocol `openai`, the exact
resource base ending in `/openai` or `/openai/v1`, the Azure deployment name
as `served_model`, and either `api-key` with the key value or `Authorization`
with a complete bearer value. Self-hosted vLLM and Ollama use the same fields,
typically with `Authorization` absent or set to a bearer token. Never put a
secret into the URL.

Docling and the direct GLM adapter forward the approved configured header.
Paddle's supported remote-pipeline integration exposes a single API-key input
that behaves as bearer authorization; it cannot directly emit Azure's
`api-key` header. For Paddle on Azure, use Microsoft Entra bearer
authorization or an approved internal header-normalizing gateway. Do not work
around this limitation by embedding the key in a query string.

Provider flexibility does not authorize model substitution. The visual
service rejects an ephemeral binding whose required model contract/revision
does not match its declared service configuration. Before a hosted binding gains
conditional or primary authority, retain provider-side deployment evidence and
run the same adapter conformance, stored-surface replay, corpus, shadow,
latency, resource, and rollback gates used for baked models.

### Controlled provider egress

Vision pods are egress-denied except for DNS by default. For Kubernetes,
allow only stable provider or internal-gateway CIDRs and required ports:

```yaml
vision:
  networkPolicy:
    providerEgress:
      enabled: true
      cidrs:
        - 203.0.113.16/32  # replace with the approved provider/gateway CIDR
      ports: [443]
```

The chart refuses `enabled: true` with no CIDR or port. When the provider uses
changing public addresses, prefer an internal egress gateway or a CNI FQDN
policy expressed through `vision.networkPolicy.extraEgress`; do not widen the
CIDR to the Internet.

The base Compose vision network is internal. After the host firewall or egress
gateway allowlist is installed, opt in with:

```bash
docker compose \
  -f docker/docker-compose.yml \
  -f docker/docker-compose.vision.yaml \
  -f docker/docker-compose.vision-provider-egress.yaml up -d
```

Platform and tenant destinations share one DNS/IP/CIDR policy. It runs before
draft Probe, Apply/CLI writes, active Test, and every API/worker cache retrieval;
every current A/AAAA answer must be public or explicitly covered by
`CONNECTION_ALLOWED_INTERNAL_CIDRS`. The runtime re-check prevents a stored
hostname from silently rebinding to loopback, link-local metadata, or private
infrastructure between jobs. Direct HTTPX, OpenAI, Gemini, Anthropic, and GLM
transports disable redirects and ambient proxy discovery, and probe/audit
errors never retain an upstream response body.

Keep egress restricted even with these controls. Some isolated Paddle/Docling
provider adapters own their final socket resolution, leaving a narrow DNS
change interval after the worker's validation; a default-deny network policy,
stable egress gateway, or CNI FQDN policy is the final containment boundary.
An intentional internal endpoint requires both its narrow CIDR in
`CONNECTION_ALLOWED_INTERNAL_CIDRS` on API/workers and a matching egress rule.

## Enable and verify collection

### Kubernetes and Helm

The chart creates a `ClusterIP` worker metrics Service with a named `metrics`
port and standard Prometheus scrape annotations whenever
`worker.metrics.enabled=true` (the default). For a Prometheus Operator install,
enable the optional `ServiceMonitor` and supply any labels required by the
cluster's Prometheus selector:

```yaml
worker:
  metrics:
    enabled: true
    bindHost: 0.0.0.0
    port: 9090
    serviceMonitor:
      enabled: true
      interval: 30s
      scrapeTimeout: 10s
      additionalLabels:
        release: kube-prometheus-stack
```

The `ServiceMonitor` relabels every discovered target with
`docstack_component="worker"`; the target-down alert and dashboard rely on
that stable label. If a different discovery mechanism consumes the Service
annotations, add the same target label in Prometheus relabeling.

Verify the rendered configuration and the in-cluster endpoint before enabling
alerts:

```bash
helm template docstack ./charts/docstack -f production-values.yaml > /tmp/docstack-rendered.yaml
kubectl -n <namespace> get service -l app.kubernetes.io/component=worker
kubectl -n <namespace> port-forward service/<rendered-fullname>-worker-metrics 9090:9090
curl --fail --silent http://127.0.0.1:9090/metrics | grep '^docstack_vision_'
```

Use the actual rendered Service name if the release name is truncated. A
successful `curl` that contains only Python process metrics proves reachability,
not vision instrumentation; complete the canary gate below.

### Docker Compose

Compose passes `WORKER_METRICS_*` to the worker and exposes port 9090 only to
containers on the Compose network. Configure Prometheus with
`worker:9090/metrics`. The port is deliberately not published on the host. If
temporary host access is needed for local diagnosis, use a narrowly bound
override such as `127.0.0.1:9090:9090` and remove it afterward.

### Rules and dashboard

- Load [`monitoring/vision/prometheus-rules.yaml`](../monitoring/vision/prometheus-rules.yaml)
  into the Prometheus rule loader or the platform's rule-management system.
- Import
  [`monitoring/vision/grafana-dashboard.json`](../monitoring/vision/grafana-dashboard.json)
  and select the Prometheus datasource that scrapes both registries.
- Run `promtool check rules monitoring/vision/prometheus-rules.yaml` in the
  monitoring delivery pipeline before applying a rule revision.
- Tune thresholds only from a documented baseline window. Keep the minimum
  sample clauses unless a replacement accounts for low-volume noise.

The supplied rate and latency alerts are deliberately volume-gated. The
target-down alert only fires for discovered worker targets. There is no generic
"vision metric absent" alert because quiet traffic and missing instrumentation
are indistinguishable without a scheduled canary. **No data is unknown, not
healthy**; the acceptance canary is the control that resolves that ambiguity.

## Production telemetry acceptance gate

Run this gate in staging and after every production deployment that changes
the worker, a vision adapter, the active execution profile, or metric wiring.
Use synthetic, non-sensitive fixtures representing the enabled paths:

1. Submit a multi-surface document that executes the active full-lane profile.
2. Submit a fixture designed to trigger targeted recognition if crop rescue is
   enabled by that profile.
3. Wait for both jobs to reach a terminal decision.
4. Confirm the worker target stayed up and each exercised family increased in
   the canary window.

PromQL acceptance checks for a ten-minute window:

```text
min(up{docstack_component="worker"}) == 1

sum(increase(docstack_vision_render_duration_seconds_count[10m])) > 0
sum(increase(docstack_vision_render_surfaces_total[10m])) > 0
sum(increase(docstack_vision_render_pixels_total[10m])) > 0
sum(increase(docstack_vision_lane_duration_seconds_count[10m])) > 0
sum(increase(docstack_vision_engine_results_total[10m])) > 0
sum(increase(docstack_vision_quality_signals_total[10m])) > 0
sum(increase(docstack_vision_reconciliation_decisions_total[10m])) > 0
sum(increase(docstack_vision_review_decisions_total[10m])) > 0
```

For the targeted-recognition fixture:

```text
sum(increase(docstack_vision_crop_rescues_total[10m])) > 0
sum(increase(docstack_vision_crop_area_pixels_count[10m])) > 0
```

Resource measurements are provider-reported and therefore conditional on the
adapter returning a supported numeric key. For every configured provider that
claims resource telemetry, verify:

```text
sum by (engine, resource, unit) (
  increase(docstack_vision_resource_measurement_count[10m])
) > 0
```

Render warnings are expected to remain absent for a clean fixture; verify the
family with a deterministic warning fixture during release qualification, not
by forcing a production failure. A zero result can be a valid outcome. An empty
query after its path was deliberately exercised is a telemetry failure and
blocks promotion even when job outputs look correct.

## Metrics contract and interpretation

| Research §23 requirement | Production signal | Interpretation |
|---|---|---|
| Render time, pages/surfaces, pixels, warnings | `docstack_vision_render_duration_seconds`, `docstack_vision_render_surfaces_total`, `docstack_vision_render_pixels_total`, `docstack_vision_render_warnings_total` | Group only by bounded `format`, `profile`, surface `kind`, and warning `code` |
| Primary/secondary lane latency | `docstack_vision_lane_duration_seconds{lane,role,status}` | Compare role and status; do not average successful and failed calls without labeling |
| Engine failure and partial success | `docstack_vision_engine_results_total{engine,role,status}` | `failed` and recognition `rejected` are failures; `partial` is reported separately |
| Crop rescue count and area | `docstack_vision_crop_rescues_total`, `docstack_vision_crop_area_pixels` | A high rescue rate can mean effective recovery or degraded upstream quality; pair it with outcomes |
| Coverage, truncation, repetition, structure | `docstack_vision_quality_signals_total{phase,kind,severity,outcome,action}` | The bounded `outcome=passed|failed` label provides the denominator for failure ratios |
| Disagreement and unresolved evidence | `docstack_vision_reconciliation_decisions_total{kind,state}` | `unresolved` is operationally unsafe; retained alternatives are visible but not equivalent to unresolved |
| Review rate | `docstack_vision_review_decisions_total{decision,reason}` | Worker `review` is normalized to `needs_review`; reason is a bounded category |
| Queue, CPU/GPU time, memory, utilization, pixels, tokens, cost | `docstack_vision_resource_measurement{engine,resource,unit}` | Provider-reported histograms; missing values are unknown and must not be converted to zero |
| Omission and hallucination on audited samples | Scientific evaluator artifacts in [`eval/`](../eval/README.md) | Never infer semantic omission or hallucination from a live runtime counter |
| Extraction schema, grounding, judge outcomes | Evaluation reports plus tenant-authorized job diagnostics | Promotion uses frozen goldens and paired reports; job diagnosis uses persisted contracts |

The Prometheus metric name `docstack_vision_resource_measurement` is a
histogram family, so query its `_bucket`, `_sum`, and `_count` series. Units are
explicit bounded labels and must not be combined across units.

## Alert response

### Worker metrics target down

Alert: `DocStackWorkerMetricsTargetDown`.

1. Confirm the worker pod is running and ready enough to process jobs. SAQ has
   no application HTTP liveness endpoint; metrics reachability is a monitoring
   signal, not a replacement for container-exit liveness and the Redis worker
   heartbeat.
2. Inspect worker startup logs for `could not bind` or a listener-thread error.
   A bind conflict is fatal by design. Check that no sidecar or duplicate
   process uses `WORKER_METRICS_PORT`.
3. Verify Service selectors, EndpointSlices, NetworkPolicies, Prometheus
   namespace access, and the ServiceMonitor selector labels.
4. Compare `WORKER_METRICS_BIND_HOST` with the deployment topology. Kubernetes
   Services require a pod-reachable address such as `0.0.0.0`, not loopback.
5. Restore scrape reachability before interpreting any other vision panel.

If the worker is healthy but all targets disappeared, treat this as monitoring
control-plane impact. If the worker is crash-looping because the listener
cannot bind, restore the previous known-good port/configuration or resolve the
port conflict; do not disable monitoring as a shortcut.

### Engine failures or partial results

Alerts: `DocStackVisionEngineFailureRateHigh`,
`DocStackVisionEnginePartialRateHigh`, and
`DocStackVisionCropRescueFailureRateHigh`.

1. Break down `docstack_vision_engine_results_total` by bounded `engine`,
   `role`, and `status`; identify whether the issue is a full lane or targeted
   recognition.
2. Correlate lane p95, queue p95, provider health, pod restarts, accelerator
   pressure, timeout/error codes in execution attempts, profile changes, and
   the provider-binding revision/fingerprint carried by affected plans.
3. Open one tenant-authorized failed and one partial job. Compare attempt
   request/response contracts, persisted raw artifacts, warnings, and selected
   evidence. Do not copy document content into tickets or logs.
4. If the candidate path is failing and the baseline remains within its
   contract, stop promotion or activate the declared rollback profile.

### Render warnings or latency

Alerts: `DocStackVisionRenderWarningRateHigh` and
`DocStackVisionRenderP95LatencyHigh`.

1. Split warnings by `format`, `profile`, and bounded warning `code`; split
   latency by `format` and `profile`.
2. Check surface and pixel throughput. A sudden pixel increase often explains
   latency or memory pressure and can indicate a render-profile change.
3. Inspect persisted RenderBundle manifests for affected authorized jobs,
   including surface order, dimensions, source-plane metadata, hashes, and
   renderer warnings.
4. Compare against the authoritative Office reference rendering for Office
   regressions. Do not attribute renderer loss to OCR without that comparison.

### Latency, queue, or resource pressure

Alerts: `DocStackVisionLaneP95LatencyHigh` and
`DocStackVisionQueueP95High`.

1. Compare p95 by engine/lane and role with throughput. A queue increase with
   flat execution time indicates capacity pressure; both increasing suggests
   provider or accelerator degradation.
2. Review bounded `cpu_memory`, `gpu_memory`, `gpu_utilization`, `cpu_time`,
   `gpu_time`, `input_pixels`, `output_tokens`, and `estimated_cost` series.
3. Check pod limits, throttling, OOM kills, node accelerator health, provider
   concurrency, and upstream rate limits.
4. Do not increase worker concurrency independently of provider slots, GPU
   memory, and the execution profile's resource budget.

### Quality, reconciliation, or review pressure

Alerts: `DocStackVisionHighSeverityQualityFailureRateHigh`,
`DocStackVisionUnresolvedReconciliationRateHigh`, and
`DocStackVisionReviewRateHigh`.

1. Break quality signals down by `phase`, `kind`, `severity`, `outcome`, and
   `action`; compare failed signals with their passed denominator.
2. Break reconciliation down by object `kind` and `state`. Open the diagnostics
   evidence candidates for representative authorized jobs to see selected,
   rejected, retained-alternative, and unresolved evidence.
3. Split review routing by `decision` and bounded `reason`. Check review-queue
   age and staffing before changing quality thresholds.
4. Never reduce a threshold merely to clear the alert. Any threshold change is
   a new profile/policy revision and requires the scientific promotion gate.

## Trace and operator diagnostics

For a tenant-authorized job, `GET /v1/documents/{job_id}` returns
`vision_diagnostics` after the immutable RenderBundle and EvidenceLedger both
exist. Review responses carry the same diagnostic envelope. The document and
review detail interfaces render the execution trace, visual surface, and
evidence layers.

The persisted envelope supplies the §23 trace chain:

- source and artifact SHA-256/byte metadata identify the received bytes;
- RenderBundle manifests describe rendered surfaces, order, profiles, hashes,
  and warnings;
- attempts identify stage, lane, deployment, surface group, bounded request,
  response, timings, resources, and raw artifact IDs;
- EvidenceLedger candidates and selections explain normalization,
  alternatives, rejection, selection, and unresolved evidence;
- quality assessments/signals explain secondary actions and review routing;
- canonical result/evidence links support Markdown and structured-field
  grounding.

Operator overlays cover native evidence, Paddle, Docling, targeted-recognition
results, canonical objects, reading order, and selected/rejected candidates
when those observations exist. Signed artifact locations are resolved
separately so immutable provenance payloads are not mutated.

Treat diagnostics as sensitive content. Use application authorization, avoid
screenshots or exports outside the incident boundary, and follow artifact
retention/deletion policy. Prometheus and alert annotations must refer only to
aggregate bounded labels, never a job or its contents.

## Rollback and recovery

### Rotate deployment Secrets

Every externally managed Kubernetes Secret is coupled to the Helm release by a
non-secret revision. Use `secrets.rotationToken` for runtime/DB/Redis secrets
and `vision.secretRotationToken` for internal-auth, mTLS, and HPD upstream
secrets; increment both when a coordinated rotation spans both trust domains.
Prefer creating a new Secret, updating its chart reference and corresponding
token in the same promotion values revision, and deploying through the
protected platform acceptance workflow. That workflow fails before mutation
if a referenced Secret or required key is missing, then uses Helm 4's
rollback-on-failure rollout. Never reuse a token for changed Secret data: Helm
cannot otherwise detect same-name Secret mutations and environment-backed
credentials remain stale in running processes.

After rotation, verify API readiness, worker target health, enabled vision
readiness, and the canary telemetry gate below. Preserve the old Secret until
the rollback window closes. Session-key rotation signs users out; connection
encryption rotation must keep older decrypt keys available until ciphertext
has migrated. A rollback must restore both the prior Secret references and its
matching scoped rotation token/values revision.

Every promoted vision profile must name a tested rollback profile and carry
paired shadow telemetry under the policy in [`eval/`](../eval/README.md). Roll
back when a predeclared production trigger fires, when the telemetry acceptance
gate fails, or when trace evidence shows an unsafe unresolved/grounding path.

1. Preserve the alert window, deployment/profile fingerprints, evaluation
   decision, and aggregate PromQL evidence.
2. Change `vision.rollout.activeProfile` to the declared
   `vision.rollout.rollbackProfile` through the normal deployment mechanism.
3. Confirm the chart's fail-closed validation accepts the rollback profile and
   its required services.
4. Roll out API and worker configuration together, then run the production
   telemetry acceptance gate against the restored profile.
5. Confirm failure/partial/unresolved/review rates and latency return to the
   baseline envelope. Keep the faulty profile out of authority until a new
   frozen evaluation and promotion decision passes.

Do not delete execution artifacts or rewrite immutable deployment/profile
records during response. They are the audit evidence needed for diagnosis and
future promotion decisions.

For a provider-only incident, leave the execution profile and semantic model
contract unchanged. Re-enter the last known-good endpoint/alias/header/capacity
values in the appropriate platform or tenant console, Test draft, and Apply
them as a new monotonic revision. Do not decrement or edit history in place.
New plans use the rollback binding; in-flight plans either finish on their
pinned revision or are cancelled and resubmitted according to incident policy.
Run a canary and confirm the new plan lineage before declaring recovery.