Contract testing

The OpenAPI spec at /openapi.json is auto-generated by FastAPI from the route signatures and Pydantic models. Contract testing runs the spec back against the live API to verify that:

• Every documented endpoint actually exists and accepts the documented payloads.
• Every response shape matches the declared schema.
• No undocumented status codes appear.
• Headers, query params, and path params behave as advertised.

We use schemathesis — a property-based fuzz tester driven by an OpenAPI spec. Other tools work too (Dredd, Postman / Newman, Pact); schemathesis is the most lightweight option for a Python project.

One-shot run

With the stack up:

pip install schemathesis    # one time
schemathesis run \
    --base-url http://localhost:8080 \
    http://localhost:8080/openapi.json \
    --checks all \
    --report

--checks all enables every built-in check (status_code_conformance, response_schema_conformance, content_type_conformance, not_a_server_error, negative_data_rejection, ignored_auth, etc.).

Authenticated tests

Most endpoints need an API key. Pass it with -H:

KEY="rk_<public>_<secret>"
schemathesis run \
    --base-url http://localhost:8080 \
    -H "Authorization: Bearer $KEY" \
    http://localhost:8080/openapi.json \
    --checks all

Schemathesis will test reads (GET / list endpoints) thoroughly. For mutating endpoints it generates random payloads — point it at a dedicated test tenant so it doesn’t pollute production data.

Integrating into CI

Add to pyproject.toml:

[project.optional-dependencies]
test-contract = ["schemathesis>=3.36"]

Run as a separate CI job after the docker compose stack is healthy:

- name: Boot stack
  run: docker compose -f docker/docker-compose.yml up -d
- name: Wait for API
  run: |
    until curl -sf http://localhost:8080/v1/health; do sleep 2; done
- name: Mint test API key
  run: ./scripts/mint-test-key.sh > /tmp/key
- name: Contract test
  run: |
    schemathesis run \
      --base-url http://localhost:8080 \
      -H "Authorization: Bearer $(cat /tmp/key)" \
      http://localhost:8080/openapi.json \
      --checks all \
      --workers 4 \
      --hypothesis-deadline=10000

--workers 4 parallelises requests. --hypothesis-deadline=10000 (ms) gives the OCR pipeline room to breathe on multipart upload tests.

What schemathesis catches in practice

Common drift this surfaces, in order of frequency:

1. Missing response_model on a route — FastAPI emits no schema; the response includes fields the client can’t predict.
2. Pydantic model out of sync with code — a field was renamed or made optional but the response still uses the old shape.
3. Status codes not declared in responses= — handler raises a 404 but the OpenAPI spec only declares 200 and 422.
4. Path params with insufficient validation — handler accepts tenant_id="../../../etc/passwd" because no regex constraint was declared.
5. Multipart limits — the spec doesn’t declare the upload size cap, so schemathesis happily generates 200 MiB blobs and finds an undocumented 413.

Treat each schemathesis failure as a contract bug — fix the code or fix the spec; never silence the failure.

Other quality checks

For spec quality (independent of behaviour) the standard tools are:

• redocly lint openapi.json — broad ruleset (operation IDs, descriptions, examples).
• spectral lint openapi.json --ruleset spectral:oas — rule engine with a strong OpenAPI ruleset.
• vacuum lint openapi.json — fast, opinionated linter.

Run any one of these in pre-commit if the OpenAPI surface is part of your delivery contract.