Infrastructure setup

DocStack has three model endpoints:

Endpoint type	Purpose	Usual backend
paddleocr_vl	PaddleOCR-VL OCR path	vllm
glmocr_vl	GLM-OCR OCR path	vllm
llm	extraction, characterisation, judge, drafter, classifier	vllm, ollama, openai, or gemini

The OCR-VLM endpoints are straightforward: run OpenAI-compatible chat-completions servers and point the rows at them. The LLM endpoint is vendor-pluggable and needs the correct backend_kind, base URL, model id, and credential shape.

Source of truth

LLM_URL, LLM_MODEL, LLM_BACKEND_KIND, PADDLEOCR_VL_URL, and GLMOCR_VL_URL are first-boot seeds only. On the first boot of a fresh database, the API writes them into the infra_endpoints table. After any row exists, changing those env vars does not change runtime behavior.

Use one of these runtime surfaces after first boot:

• Platform UI: /admin/platform/infrastructure
• Tenant UI: /admin/infrastructure, when tenant overrides are enabled
• CLI: docstack-admin infra set ...
• API: PUT /v1/admin/platform/infra/{endpoint_type} or tenant equivalent

Tenant overrides replace the platform default only for that tenant. A tenant that overrides only llm still inherits the platform paddleocr_vl and glmocr_vl rows.

Field meanings

Field	Meaning
backend_kind	Wire dialect. vllm, ollama, and openai use OpenAI-compatible chat completions. gemini uses Google’s native Gemini SDK and should be used only for llm.
base_url	For OpenAI-wire backends, the server root. The app appends /v1; a URL that already ends in /v1 is normalized. For Gemini, this is a Gemini host override. Use https://generativelanguage.googleapis.com for the public Developer API.
served_model	Exact model id returned by the vendor’s model listing. Validation rejects ids that are not listed.
auth_header / API key	For vllm, ollama, and openai, this is the full Authorization header value, for example Bearer sk-.... For gemini, this is the raw Google API key with no Bearer prefix.
max_concurrent	Client-side semaphore for this endpoint. Match upstream capacity, such as vLLM --max-num-seqs or Ollama OLLAMA_NUM_PARALLEL.
timeout_seconds	Per-request model call timeout.
max_model_len	Optional LLM context-window hint when /v1/models does not expose one, commonly with Ollama.

Storing any credential requires CONNECTION_ENCRYPTION_KEYS on both API and worker processes:

python -c "import base64,secrets; print('v1:' + base64.urlsafe_b64encode(secrets.token_bytes(32)).decode())"

Keyless local vLLM and local Ollama rows can leave the credential field blank.

LLM vendor matrix

Vendor	backend_kind	base_url	served_model	Credential
vLLM	vllm	http://host:8000	vLLM --served-model-name	blank, or Bearer <key> if vLLM auth is enabled
Local Ollama	ollama	http://host:11434	pulled tag, for example qwen3.6:35b-a3b-q4_K_M	blank
Ollama Cloud	ollama	https://ollama.com/v1	model id from /v1/models, for example gpt-oss:120b	Bearer <OLLAMA_API_KEY>
OpenAI	openai	https://api.openai.com	model id from /v1/models, for example gpt-5.4-mini	Bearer <OPENAI_API_KEY>
Gemini	gemini	https://generativelanguage.googleapis.com	Gemini model id, for example gemini-3.1-flash-lite	raw Gemini API key

Use the model id that the provider lists. For example, Ollama Cloud can accept aliases in a chat call, but DocStack validation checks /v1/models; use gpt-oss:120b if that is what the model list returns, not an unlisted alias such as gpt-oss:120b-cloud.

First-boot examples

For local Ollama:

LLM_BACKEND_KIND=ollama
LLM_URL=http://ollama:11434
LLM_MODEL=qwen3.6:35b-a3b-q4_K_M
LLM_REASONING_EFFORT=none

For Gemini:

CONNECTION_ENCRYPTION_KEYS=v1:<base64-32-byte-key>
LLM_BACKEND_KIND=gemini
LLM_URL=https://generativelanguage.googleapis.com
LLM_MODEL=gemini-3.1-flash-lite
LLM_API_KEY=<google-api-key>
LLM_REASONING_EFFORT=none
LLM_GEMINI_SAFETY=block_none

LLM_API_KEY is the env var DocStack reads for Gemini first-boot seeding. GEMINI_API_KEY and GEMINI_MODEL are not read by the runtime config loader unless you map them yourself before starting the API.

If the database already has infra_endpoints rows, use the runtime UI or CLI instead of changing env:

docstack-admin infra set llm \
  --backend gemini \
  --url https://generativelanguage.googleapis.com \
  --model gemini-3.1-flash-lite \
  --auth-header "$(cat /run/secrets/gemini_api_key)" \
  --timeout 300

docstack-admin infra set llm \
  --backend ollama \
  --url https://ollama.com/v1 \
  --model gpt-oss:120b \
  --auth-header "Bearer $(cat /run/secrets/ollama_api_key)" \
  --timeout 300

OCR-VLM setup

Keep paddleocr_vl and glmocr_vl on OpenAI-compatible chat-completions servers. The bundled model-server path uses vLLM:

Endpoint	backend_kind	base_url	served_model
paddleocr_vl	vllm	http://paddle:8000	PaddleOCR-VL-1.6
glmocr_vl	vllm	http://glm:8000	glm-ocr

Do not use gemini for OCR-VLM rows. Gemini support is for the text LLM call sites only.

Validation behavior

Every save and test runs a server-side probe:

1. Reachability
2. Served-model presence
3. Shape probe

For vllm, ollama, and openai, reachability and model listing use /v1/models. The shape probe uses /v1/chat/completions. For gemini, the same stages run through the native SDK: model listing plus a small generate_content JSON request.

With INFRA_VALIDATION_HARD_GATE=true, a failed probe rejects the save. This is intentional: it prevents a bad endpoint row from being published to workers.

Common failures

Symptom	Fix
served_model not in /v1/models response	Copy the exact id from the provider’s model list.
Gemini tries to call an Ollama/vLLM URL	Set base_url to https://generativelanguage.googleapis.com when backend_kind=gemini.
OpenAI returns Unrecognized request argument supplied: reasoning_effort	Use a model that accepts the configured reasoning field, or change the app config/code so the field is omitted for that model family.
Ollama reasoning model returns empty JSON content	Set LLM_REASONING_EFFORT=none and keep backend_kind=ollama so the app sends max_tokens.
Credential save returns 503	Set CONNECTION_ENCRYPTION_KEYS on API and worker, then save the endpoint credential again.
Changing .env does nothing	The row already exists. Use the infrastructure UI or docstack-admin infra set.