Vision execution operations runbook

Vision execution operations runbook

This runbook is the production companion to docstack-next-generation-architecture-research.md §23. It covers scrape discovery, telemetry acceptance, alert response, safe diagnostics, and rollback for the render-first visual execution plane.

Operating boundary

DocStack has two independent Prometheus registries because the API and SAQ worker are separate processes:

Registry	Default endpoint	Owns	Exposure rule
API	api:8080/metrics	HTTP, queue, provider-health, and API-owned pipeline metrics	Cluster-internal scrape; preserve the deployment’s existing gateway policy
Worker	worker:9090/metrics	Worker pipeline and docstack_vision_* metrics	Cluster-internal scrape only; never add it to public Ingress or Gateway routes

The worker listener is unauthenticated by design and contains aggregate, bounded-cardinality operational data. It starts before runtime initialization, fails worker startup if its configured address cannot bind, and is shut down and joined during worker termination. Set WORKER_METRICS_ENABLED=false only when another in-process collector owns the same registry; disabling the only worker scrape path removes the production vision telemetry contract.

Metric labels must never contain tenant identifiers, job identifiers, document content, extracted values, filenames, prompts, responses, presigned URLs, or free-form errors. Per-job content and provenance belong in tenant-authorized, retention-controlled diagnostics—not Prometheus.

Configure and hot-reload model providers

DocStack fixes the semantic model contracts used by its load-bearing lanes but does not fix the infrastructure vendor that serves them. The active source of truth is the revisioned infra_endpoints table, not process environment:

Role	Required behavior	Supported binding adapters
paddleocr_vl	Paddle full lane using the accepted PaddleOCR-VL-1.6 contract	vllm, openai
granite_docling_vl	Docling full lane using Granite Docling, DocTags, and the declared prompt/codec contract	local, vllm, ollama, openai
glmocr_vl	task-bounded GLM-OCR text/table/formula recognition	vllm, ollama, openai
llm	template-driven extraction, drafting, judging, and characterization	vllm, ollama, openai, gemini, anthropic

Environment values seed any missing platform-default rows at first boot. They do not overwrite a row on restart. A platform administrator manages defaults at /admin/platform/infrastructure; an enabled tenant administrator manages overrides at /admin/infrastructure. Tenant resolution is override first and platform default second. Credentials are encrypted with CONNECTION_ENCRYPTION_KEYS; API responses show only whether a credential is present and its non-secret header name.

For each role:

1. Select the protocol adapter, base URL, provider deployment/routing alias, credential-header name and value, concurrency, and timeout.
2. Choose Test draft. This tests the unsaved form through the same validation path and has no upsert, commit, audit-state, or invalidation effect. A successful connectivity/shape probe proves the advertised alias is reachable; it is not independent proof that a vendor has mapped that alias to the claimed immutable model weights.
3. Choose Apply. Apply compares the displayed revision with active state, validates again, commits and audits the next revision, and publishes an eager invalidation. A concurrent edit returns HTTP 409; refresh rather than overwriting it.
4. Submit a canary and inspect its execution diagnostics. The plan must show the expected binding ID, source, provider alias, public fingerprint, and configuration revision. It must not contain the credential value.

The consistency boundary is plan creation. Every new job performs a fresh effective-binding resolution and freezes the four results for the complete attempt. A worker rebuilds affected clients when a binding fingerprint changes, including after a missed Redis notification. An already created or in-flight plan deliberately continues on its original revision. “Immediate” therefore means the next plan, not mutation of work already executing.

The equivalent API operations are:

POST /v1/admin/platform/infra/{role}/probe
PUT  /v1/admin/platform/infra/{role}
POST /v1/admin/tenants/{tenant_id}/infra/{role}/probe
PUT  /v1/admin/tenants/{tenant_id}/infra/{role}

The body carries backend_kind, base_url, served_model, optional auth_header_name, optional write-only auth_header, max_concurrent, timeout_seconds, and expected_revision. local://baked is valid only for the Granite/Docling local adapter. Remote OpenAI-wire URLs may be entered with or without a trailing /v1; non-/v1 proxy path prefixes are preserved. URL credentials, query strings, parameters, fragments, whitespace, and control characters are rejected. Credential values are printable ASCII and baked-local bindings cannot carry one.

Azure AI Foundry and self-hosted providers

For Azure’s OpenAI-compatible v1 surface, use protocol openai, the exact resource base ending in /openai or /openai/v1, the Azure deployment name as served_model, and either api-key with the key value or Authorization with a complete bearer value. Self-hosted vLLM and Ollama use the same fields, typically with Authorization absent or set to a bearer token. Never put a secret into the URL.

Docling and the direct GLM adapter forward the approved configured header. Paddle’s supported remote-pipeline integration exposes a single API-key input that behaves as bearer authorization; it cannot directly emit Azure’s api-key header. For Paddle on Azure, use Microsoft Entra bearer authorization or an approved internal header-normalizing gateway. Do not work around this limitation by embedding the key in a query string.

Provider flexibility does not authorize model substitution. The visual service rejects an ephemeral binding whose required model contract/revision does not match its declared service configuration. Before a hosted binding gains conditional or primary authority, retain provider-side deployment evidence and run the same adapter conformance, stored-surface replay, corpus, shadow, latency, resource, and rollback gates used for baked models.

Controlled provider egress

Vision pods are egress-denied except for DNS by default. For Kubernetes, allow only stable provider or internal-gateway CIDRs and required ports:

vision:
  networkPolicy:
    providerEgress:
      enabled: true
      cidrs:
        - 203.0.113.16/32  # replace with the approved provider/gateway CIDR
      ports: [443]

The chart refuses enabled: true with no CIDR or port. When the provider uses changing public addresses, prefer an internal egress gateway or a CNI FQDN policy expressed through vision.networkPolicy.extraEgress; do not widen the CIDR to the Internet.

The base Compose vision network is internal. After the host firewall or egress gateway allowlist is installed, opt in with:

docker compose \
  -f docker/docker-compose.yml \
  -f docker/docker-compose.vision.yaml \
  -f docker/docker-compose.vision-provider-egress.yaml up -d

Platform and tenant destinations share one DNS/IP/CIDR policy. It runs before draft Probe, Apply/CLI writes, active Test, and every API/worker cache retrieval; every current A/AAAA answer must be public or explicitly covered by CONNECTION_ALLOWED_INTERNAL_CIDRS. The runtime re-check prevents a stored hostname from silently rebinding to loopback, link-local metadata, or private infrastructure between jobs. Direct HTTPX, OpenAI, Gemini, Anthropic, and GLM transports disable redirects and ambient proxy discovery, and probe/audit errors never retain an upstream response body.

Keep egress restricted even with these controls. Some isolated Paddle/Docling provider adapters own their final socket resolution, leaving a narrow DNS change interval after the worker’s validation; a default-deny network policy, stable egress gateway, or CNI FQDN policy is the final containment boundary. An intentional internal endpoint requires both its narrow CIDR in CONNECTION_ALLOWED_INTERNAL_CIDRS on API/workers and a matching egress rule.

Enable and verify collection

Kubernetes and Helm

The chart creates a ClusterIP worker metrics Service with a named metrics port and standard Prometheus scrape annotations whenever worker.metrics.enabled=true (the default). For a Prometheus Operator install, enable the optional ServiceMonitor and supply any labels required by the cluster’s Prometheus selector:

worker:
  metrics:
    enabled: true
    bindHost: 0.0.0.0
    port: 9090
    serviceMonitor:
      enabled: true
      interval: 30s
      scrapeTimeout: 10s
      additionalLabels:
        release: kube-prometheus-stack

The ServiceMonitor relabels every discovered target with docstack_component="worker"; the target-down alert and dashboard rely on that stable label. If a different discovery mechanism consumes the Service annotations, add the same target label in Prometheus relabeling.

Verify the rendered configuration and the in-cluster endpoint before enabling alerts:

helm template docstack ./charts/docstack -f production-values.yaml > /tmp/docstack-rendered.yaml
kubectl -n <namespace> get service -l app.kubernetes.io/component=worker
kubectl -n <namespace> port-forward service/<rendered-fullname>-worker-metrics 9090:9090
curl --fail --silent http://127.0.0.1:9090/metrics | grep '^docstack_vision_'

Use the actual rendered Service name if the release name is truncated. A successful curl that contains only Python process metrics proves reachability, not vision instrumentation; complete the canary gate below.

Docker Compose

Compose passes WORKER_METRICS_* to the worker and exposes port 9090 only to containers on the Compose network. Configure Prometheus with worker:9090/metrics. The port is deliberately not published on the host. If temporary host access is needed for local diagnosis, use a narrowly bound override such as 127.0.0.1:9090:9090 and remove it afterward.

Rules and dashboard

• Load monitoring/vision/prometheus-rules.yaml into the Prometheus rule loader or the platform’s rule-management system.
• Import monitoring/vision/grafana-dashboard.json and select the Prometheus datasource that scrapes both registries.
• Run promtool check rules monitoring/vision/prometheus-rules.yaml in the monitoring delivery pipeline before applying a rule revision.
• Tune thresholds only from a documented baseline window. Keep the minimum sample clauses unless a replacement accounts for low-volume noise.

The supplied rate and latency alerts are deliberately volume-gated. The target-down alert only fires for discovered worker targets. There is no generic “vision metric absent” alert because quiet traffic and missing instrumentation are indistinguishable without a scheduled canary. No data is unknown, not healthy; the acceptance canary is the control that resolves that ambiguity.

Production telemetry acceptance gate

Run this gate in staging and after every production deployment that changes the worker, a vision adapter, the active execution profile, or metric wiring. Use synthetic, non-sensitive fixtures representing the enabled paths:

1. Submit a multi-surface document that executes the active full-lane profile.
2. Submit a fixture designed to trigger targeted recognition if crop rescue is enabled by that profile.
3. Wait for both jobs to reach a terminal decision.
4. Confirm the worker target stayed up and each exercised family increased in the canary window.

PromQL acceptance checks for a ten-minute window:

min(up{docstack_component="worker"}) == 1

sum(increase(docstack_vision_render_duration_seconds_count[10m])) > 0
sum(increase(docstack_vision_render_surfaces_total[10m])) > 0
sum(increase(docstack_vision_render_pixels_total[10m])) > 0
sum(increase(docstack_vision_lane_duration_seconds_count[10m])) > 0
sum(increase(docstack_vision_engine_results_total[10m])) > 0
sum(increase(docstack_vision_quality_signals_total[10m])) > 0
sum(increase(docstack_vision_reconciliation_decisions_total[10m])) > 0
sum(increase(docstack_vision_review_decisions_total[10m])) > 0

For the targeted-recognition fixture:

sum(increase(docstack_vision_crop_rescues_total[10m])) > 0
sum(increase(docstack_vision_crop_area_pixels_count[10m])) > 0

Resource measurements are provider-reported and therefore conditional on the adapter returning a supported numeric key. For every configured provider that claims resource telemetry, verify:

sum by (engine, resource, unit) (
  increase(docstack_vision_resource_measurement_count[10m])
) > 0

Render warnings are expected to remain absent for a clean fixture; verify the family with a deterministic warning fixture during release qualification, not by forcing a production failure. A zero result can be a valid outcome. An empty query after its path was deliberately exercised is a telemetry failure and blocks promotion even when job outputs look correct.

Metrics contract and interpretation

Research §23 requirement	Production signal	Interpretation
Render time, pages/surfaces, pixels, warnings	docstack_vision_render_duration_seconds, docstack_vision_render_surfaces_total, docstack_vision_render_pixels_total, docstack_vision_render_warnings_total	Group only by bounded format, profile, surface kind, and warning code
Primary/secondary lane latency	docstack_vision_lane_duration_seconds{lane,role,status}	Compare role and status; do not average successful and failed calls without labeling
Engine failure and partial success	docstack_vision_engine_results_total{engine,role,status}	failed and recognition rejected are failures; partial is reported separately
Crop rescue count and area	docstack_vision_crop_rescues_total, docstack_vision_crop_area_pixels	A high rescue rate can mean effective recovery or degraded upstream quality; pair it with outcomes
Coverage, truncation, repetition, structure	docstack_vision_quality_signals_total{phase,kind,severity,outcome,action}	The bounded `outcome=passed
Disagreement and unresolved evidence	docstack_vision_reconciliation_decisions_total{kind,state}	unresolved is operationally unsafe; retained alternatives are visible but not equivalent to unresolved
Review rate	docstack_vision_review_decisions_total{decision,reason}	Worker review is normalized to needs_review; reason is a bounded category
Queue, CPU/GPU time, memory, utilization, pixels, tokens, cost	docstack_vision_resource_measurement{engine,resource,unit}	Provider-reported histograms; missing values are unknown and must not be converted to zero
Omission and hallucination on audited samples	Scientific evaluator artifacts in eval/	Never infer semantic omission or hallucination from a live runtime counter
Extraction schema, grounding, judge outcomes	Evaluation reports plus tenant-authorized job diagnostics	Promotion uses frozen goldens and paired reports; job diagnosis uses persisted contracts

The Prometheus metric name docstack_vision_resource_measurement is a histogram family, so query its _bucket, _sum, and _count series. Units are explicit bounded labels and must not be combined across units.

Alert response

Worker metrics target down

Alert: DocStackWorkerMetricsTargetDown.

1. Confirm the worker pod is running and ready enough to process jobs. SAQ has no application HTTP liveness endpoint; metrics reachability is a monitoring signal, not a replacement for container-exit liveness and the Redis worker heartbeat.
2. Inspect worker startup logs for could not bind or a listener-thread error. A bind conflict is fatal by design. Check that no sidecar or duplicate process uses WORKER_METRICS_PORT.
3. Verify Service selectors, EndpointSlices, NetworkPolicies, Prometheus namespace access, and the ServiceMonitor selector labels.
4. Compare WORKER_METRICS_BIND_HOST with the deployment topology. Kubernetes Services require a pod-reachable address such as 0.0.0.0, not loopback.
5. Restore scrape reachability before interpreting any other vision panel.

If the worker is healthy but all targets disappeared, treat this as monitoring control-plane impact. If the worker is crash-looping because the listener cannot bind, restore the previous known-good port/configuration or resolve the port conflict; do not disable monitoring as a shortcut.

Engine failures or partial results

Alerts: DocStackVisionEngineFailureRateHigh, DocStackVisionEnginePartialRateHigh, and DocStackVisionCropRescueFailureRateHigh.

1. Break down docstack_vision_engine_results_total by bounded engine, role, and status; identify whether the issue is a full lane or targeted recognition.
2. Correlate lane p95, queue p95, provider health, pod restarts, accelerator pressure, timeout/error codes in execution attempts, profile changes, and the provider-binding revision/fingerprint carried by affected plans.
3. Open one tenant-authorized failed and one partial job. Compare attempt request/response contracts, persisted raw artifacts, warnings, and selected evidence. Do not copy document content into tickets or logs.
4. If the candidate path is failing and the baseline remains within its contract, stop promotion or activate the declared rollback profile.

Render warnings or latency

Alerts: DocStackVisionRenderWarningRateHigh and DocStackVisionRenderP95LatencyHigh.

1. Split warnings by format, profile, and bounded warning code; split latency by format and profile.
2. Check surface and pixel throughput. A sudden pixel increase often explains latency or memory pressure and can indicate a render-profile change.
3. Inspect persisted RenderBundle manifests for affected authorized jobs, including surface order, dimensions, source-plane metadata, hashes, and renderer warnings.
4. Compare against the authoritative Office reference rendering for Office regressions. Do not attribute renderer loss to OCR without that comparison.

Latency, queue, or resource pressure

Alerts: DocStackVisionLaneP95LatencyHigh and DocStackVisionQueueP95High.

1. Compare p95 by engine/lane and role with throughput. A queue increase with flat execution time indicates capacity pressure; both increasing suggests provider or accelerator degradation.
2. Review bounded cpu_memory, gpu_memory, gpu_utilization, cpu_time, gpu_time, input_pixels, output_tokens, and estimated_cost series.
3. Check pod limits, throttling, OOM kills, node accelerator health, provider concurrency, and upstream rate limits.
4. Do not increase worker concurrency independently of provider slots, GPU memory, and the execution profile’s resource budget.

Quality, reconciliation, or review pressure

Alerts: DocStackVisionHighSeverityQualityFailureRateHigh, DocStackVisionUnresolvedReconciliationRateHigh, and DocStackVisionReviewRateHigh.

1. Break quality signals down by phase, kind, severity, outcome, and action; compare failed signals with their passed denominator.
2. Break reconciliation down by object kind and state. Open the diagnostics evidence candidates for representative authorized jobs to see selected, rejected, retained-alternative, and unresolved evidence.
3. Split review routing by decision and bounded reason. Check review-queue age and staffing before changing quality thresholds.
4. Never reduce a threshold merely to clear the alert. Any threshold change is a new profile/policy revision and requires the scientific promotion gate.

Trace and operator diagnostics

For a tenant-authorized job, GET /v1/documents/{job_id} returns vision_diagnostics after the immutable RenderBundle and EvidenceLedger both exist. Review responses carry the same diagnostic envelope. The document and review detail interfaces render the execution trace, visual surface, and evidence layers.

The persisted envelope supplies the §23 trace chain:

• source and artifact SHA-256/byte metadata identify the received bytes;
• RenderBundle manifests describe rendered surfaces, order, profiles, hashes, and warnings;
• attempts identify stage, lane, deployment, surface group, bounded request, response, timings, resources, and raw artifact IDs;
• EvidenceLedger candidates and selections explain normalization, alternatives, rejection, selection, and unresolved evidence;
• quality assessments/signals explain secondary actions and review routing;
• canonical result/evidence links support Markdown and structured-field grounding.

Operator overlays cover native evidence, Paddle, Docling, targeted-recognition results, canonical objects, reading order, and selected/rejected candidates when those observations exist. Signed artifact locations are resolved separately so immutable provenance payloads are not mutated.

Treat diagnostics as sensitive content. Use application authorization, avoid screenshots or exports outside the incident boundary, and follow artifact retention/deletion policy. Prometheus and alert annotations must refer only to aggregate bounded labels, never a job or its contents.

Rollback and recovery

Rotate deployment Secrets

Every externally managed Kubernetes Secret is coupled to the Helm release by a non-secret revision. Use secrets.rotationToken for runtime/DB/Redis secrets and vision.secretRotationToken for internal-auth, mTLS, and HPD upstream secrets; increment both when a coordinated rotation spans both trust domains. Prefer creating a new Secret, updating its chart reference and corresponding token in the same promotion values revision, and deploying through the protected platform acceptance workflow. That workflow fails before mutation if a referenced Secret or required key is missing, then uses Helm 4’s rollback-on-failure rollout. Never reuse a token for changed Secret data: Helm cannot otherwise detect same-name Secret mutations and environment-backed credentials remain stale in running processes.

After rotation, verify API readiness, worker target health, enabled vision readiness, and the canary telemetry gate below. Preserve the old Secret until the rollback window closes. Session-key rotation signs users out; connection encryption rotation must keep older decrypt keys available until ciphertext has migrated. A rollback must restore both the prior Secret references and its matching scoped rotation token/values revision.

Every promoted vision profile must name a tested rollback profile and carry paired shadow telemetry under the policy in eval/. Roll back when a predeclared production trigger fires, when the telemetry acceptance gate fails, or when trace evidence shows an unsafe unresolved/grounding path.

1. Preserve the alert window, deployment/profile fingerprints, evaluation decision, and aggregate PromQL evidence.
2. Change vision.rollout.activeProfile to the declared vision.rollout.rollbackProfile through the normal deployment mechanism.
3. Confirm the chart’s fail-closed validation accepts the rollback profile and its required services.
4. Roll out API and worker configuration together, then run the production telemetry acceptance gate against the restored profile.
5. Confirm failure/partial/unresolved/review rates and latency return to the baseline envelope. Keep the faulty profile out of authority until a new frozen evaluation and promotion decision passes.

Do not delete execution artifacts or rewrite immutable deployment/profile records during response. They are the audit evidence needed for diagnosis and future promotion decisions.

For a provider-only incident, leave the execution profile and semantic model contract unchanged. Re-enter the last known-good endpoint/alias/header/capacity values in the appropriate platform or tenant console, Test draft, and Apply them as a new monotonic revision. Do not decrement or edit history in place. New plans use the rollback binding; in-flight plans either finish on their pinned revision or are cancelled and resubmitted according to incident policy. Run a canary and confirm the new plan lineage before declaring recovery.