# Infrastructure setup

DocStack has three model endpoints:

| Endpoint type | Purpose | Usual backend |
|---|---|---|
| `paddleocr_vl` | PaddleOCR-VL OCR path | `vllm` |
| `glmocr_vl` | GLM-OCR OCR path | `vllm` |
| `llm` | extraction, characterisation, judge, drafter, classifier | `vllm`, `ollama`, `openai`, or `gemini` |

The OCR-VLM endpoints are straightforward: run OpenAI-compatible chat-completions servers and point the rows at them. The LLM endpoint is vendor-pluggable and needs the correct `backend_kind`, base URL, model id, and credential shape.

## Source of truth

`LLM_URL`, `LLM_MODEL`, `LLM_BACKEND_KIND`, `PADDLEOCR_VL_URL`, and `GLMOCR_VL_URL` are first-boot seeds only. On the first boot of a fresh database, the API writes them into the `infra_endpoints` table. After any row exists, changing those env vars does not change runtime behavior.

Use one of these runtime surfaces after first boot:

- Platform UI: `/admin/platform/infrastructure`
- Tenant UI: `/admin/infrastructure`, when tenant overrides are enabled
- CLI: `docstack-admin infra set ...`
- API: `PUT /v1/admin/platform/infra/{endpoint_type}` or tenant equivalent

Tenant overrides replace the platform default only for that tenant. A tenant that overrides only `llm` still inherits the platform `paddleocr_vl` and `glmocr_vl` rows.

## Field meanings

| Field | Meaning |
|---|---|
| `backend_kind` | Wire dialect. `vllm`, `ollama`, and `openai` use OpenAI-compatible chat completions. `gemini` uses Google's native Gemini SDK and should be used only for `llm`. |
| `base_url` | For OpenAI-wire backends, the server root. The app appends `/v1`; a URL that already ends in `/v1` is normalized. For Gemini, this is a Gemini host override. Use `https://generativelanguage.googleapis.com` for the public Developer API. |
| `served_model` | Exact model id returned by the vendor's model listing. Validation rejects ids that are not listed. |
| `auth_header` / API key | For `vllm`, `ollama`, and `openai`, this is the full Authorization header value, for example `Bearer sk-...`. For `gemini`, this is the raw Google API key with no `Bearer` prefix. |
| `max_concurrent` | Client-side semaphore for this endpoint. Match upstream capacity, such as vLLM `--max-num-seqs` or Ollama `OLLAMA_NUM_PARALLEL`. |
| `timeout_seconds` | Per-request model call timeout. |
| `max_model_len` | Optional LLM context-window hint when `/v1/models` does not expose one, commonly with Ollama. |

Storing any credential requires `CONNECTION_ENCRYPTION_KEYS` on both API and worker processes:

```bash
python -c "import base64,secrets; print('v1:' + base64.urlsafe_b64encode(secrets.token_bytes(32)).decode())"
```

Keyless local vLLM and local Ollama rows can leave the credential field blank.

## LLM vendor matrix

| Vendor | `backend_kind` | `base_url` | `served_model` | Credential |
|---|---|---|---|---|
| vLLM | `vllm` | `http://host:8000` | vLLM `--served-model-name` | blank, or `Bearer <key>` if vLLM auth is enabled |
| Local Ollama | `ollama` | `http://host:11434` | pulled tag, for example `qwen3.6:35b-a3b-q4_K_M` | blank |
| Ollama Cloud | `ollama` | `https://ollama.com/v1` | model id from `/v1/models`, for example `gpt-oss:120b` | `Bearer <OLLAMA_API_KEY>` |
| OpenAI | `openai` | `https://api.openai.com` | model id from `/v1/models`, for example `gpt-5.4-mini` | `Bearer <OPENAI_API_KEY>` |
| Gemini | `gemini` | `https://generativelanguage.googleapis.com` | Gemini model id, for example `gemini-3.1-flash-lite` | raw Gemini API key |

Use the model id that the provider lists. For example, Ollama Cloud can accept aliases in a chat call, but DocStack validation checks `/v1/models`; use `gpt-oss:120b` if that is what the model list returns, not an unlisted alias such as `gpt-oss:120b-cloud`.

## First-boot examples

For local Ollama:

```bash
LLM_BACKEND_KIND=ollama
LLM_URL=http://ollama:11434
LLM_MODEL=qwen3.6:35b-a3b-q4_K_M
LLM_REASONING_EFFORT=none
```

For Gemini:

```bash
CONNECTION_ENCRYPTION_KEYS=v1:<base64-32-byte-key>
LLM_BACKEND_KIND=gemini
LLM_URL=https://generativelanguage.googleapis.com
LLM_MODEL=gemini-3.1-flash-lite
LLM_API_KEY=<google-api-key>
LLM_REASONING_EFFORT=none
LLM_GEMINI_SAFETY=block_none
```

`LLM_API_KEY` is the env var DocStack reads for Gemini first-boot seeding. `GEMINI_API_KEY` and `GEMINI_MODEL` are not read by the runtime config loader unless you map them yourself before starting the API.

If the database already has `infra_endpoints` rows, use the runtime UI or CLI instead of changing env:

```bash
docstack-admin infra set llm \
  --backend gemini \
  --url https://generativelanguage.googleapis.com \
  --model gemini-3.1-flash-lite \
  --auth-header "$(cat /run/secrets/gemini_api_key)" \
  --timeout 300
```

```bash
docstack-admin infra set llm \
  --backend ollama \
  --url https://ollama.com/v1 \
  --model gpt-oss:120b \
  --auth-header "Bearer $(cat /run/secrets/ollama_api_key)" \
  --timeout 300
```

## OCR-VLM setup

Keep `paddleocr_vl` and `glmocr_vl` on OpenAI-compatible chat-completions servers. The bundled model-server path uses vLLM:

| Endpoint | `backend_kind` | `base_url` | `served_model` |
|---|---|---|---|
| `paddleocr_vl` | `vllm` | `http://paddle:8000` | `PaddleOCR-VL-1.6` |
| `glmocr_vl` | `vllm` | `http://glm:8000` | `glm-ocr` |

Do not use `gemini` for OCR-VLM rows. Gemini support is for the text LLM call sites only.

## Validation behavior

Every save and test runs a server-side probe:

1. Reachability
2. Served-model presence
3. Shape probe

For `vllm`, `ollama`, and `openai`, reachability and model listing use `/v1/models`. The shape probe uses `/v1/chat/completions`. For `gemini`, the same stages run through the native SDK: model listing plus a small `generate_content` JSON request.

With `INFRA_VALIDATION_HARD_GATE=true`, a failed probe rejects the save. This is intentional: it prevents a bad endpoint row from being published to workers.

## Common failures

| Symptom | Fix |
|---|---|
| `served_model not in /v1/models response` | Copy the exact id from the provider's model list. |
| Gemini tries to call an Ollama/vLLM URL | Set `base_url` to `https://generativelanguage.googleapis.com` when `backend_kind=gemini`. |
| OpenAI returns `Unrecognized request argument supplied: reasoning_effort` | Use a model that accepts the configured reasoning field, or change the app config/code so the field is omitted for that model family. |
| Ollama reasoning model returns empty JSON content | Set `LLM_REASONING_EFFORT=none` and keep `backend_kind=ollama` so the app sends `max_tokens`. |
| Credential save returns 503 | Set `CONNECTION_ENCRYPTION_KEYS` on API and worker, then save the endpoint credential again. |
| Changing `.env` does nothing | The row already exists. Use the infrastructure UI or `docstack-admin infra set`. |