Skip to content
docstack-ocr

Languages & internationalization

The platform is multilingual by virtue of its models — PaddleOCR-VL-1.6 (≈109 languages), GLM-OCR, and whichever extraction LLM you configure. The orchestration prompts are written in English, but every stage is made intelligent about the document’s language: the platform detects it once and threads the verdict into the OCR-trust gate, the extraction and judge prompts, and grounding.

Detection

Section titled “Detection”

At canonical assembly (so single- and multi-page documents are covered) the worker computes a detected_language from two signals:

  • Script + reading direction — a deterministic Unicode histogram over the OCR text. Zero dependency; this is what drives right-to-left handling and a script-appropriate language candidate set.
  • Language — py3langid (offline, model bundled in the wheel) disambiguates same-script languages (es/fr/pt/de within Latin, ru/sr/bg within Cyrillic) with a 0–1 confidence.

On multi-page documents the characteriser also names the primary_language; it refines a close same-script sibling that py3langid confuses on flat text (e.g. Galician vs Spanish), never flipping script.

The verdict — language, script, reading direction, confidence, source — rides on the canonical and shows as a chip on the document detail view. Metric: docstack_document_language_total{language,script,source}.

What the language signal drives

Section titled “What the language signal drives”
  • OCR trust — Arabic is a day-one language and GLM Arabic output is trusted by default (the other risk gates still apply). Opt out per tenant with glm_risk_policy.glm_arabic_supported=false.
  • Extraction / judge / classifier prompts — a generic English === Document Language === preamble names the language, script, and reading direction, and adds a one-line guidance fragment for languages with a known pitfall (RTL + Eastern-Arabic digits, German compounding). The model is told to interpret foreign labels by meaning and output values verbatim — it never translates extracted values.
  • Grounding — the per-field confidence check recognises printed prose dates across the day-one + campaign languages (including year-first orderings like Hungarian 2030 március 15), so a correctly-extracted foreign date is not penalised.

NER is English-primary

Section titled “NER is English-primary”

The optional entities processing mode (GLiNER2) and its Universal Entity Taxonomy are English. Entity quality on non-English documents is best-effort and not a v1 guarantee — it is intentionally not swapped for a multilingual model. Multilingual support lives in the template-extraction path described above. When the entities stage runs on a detected non-English document it logs an advisory and bumps docstack_entities_non_english_total{language}.

Configuring

Section titled “Configuring”

language_policy (platform or tenant scope, editable in the admin UI):

  • detection_enabled (default true) — detect and thread the language.
  • language_hints_enabled (default true) — inject the prompt preamble.
  • language_allowlist (default empty = all 97) — restrict py3langid’s candidate ISO-639-1 codes. Narrowing to your corpus’s languages cleanly separates close same-script siblings (set ["en","es","fr","de"] and a Spanish invoice stops being labelled Galician).