> ## Documentation Index
> Fetch the complete documentation index at: https://docs.airmux.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Production operations

> Monitor readiness, preserve state, update safely, and diagnose control-plane or gateway failures.

## Health endpoints

Liveness only answers whether the process can serve HTTP. Readiness answers whether its role can currently serve useful
traffic. Configure process restarts from `/healthz` and traffic admission from `/readyz`.

| Plane         | Endpoint   | Healthy response         | `503` meaning                                                                       |
| ------------- | ---------- | ------------------------ | ----------------------------------------------------------------------------------- |
| Control plane | `/healthz` | `200 {"status":"ok"}`    | Never returned for a dependency failure                                             |
| Control plane | `/readyz`  | `200 {"status":"ready"}` | Postgres is unavailable                                                             |
| Data plane    | `/healthz` | `200 {"status":"ok"}`    | Never returned for a dependency failure                                             |
| Data plane    | `/readyz`  | `200 {"status":"ready"}` | No accepted bundle snapshot is available, or the metering writer cannot accept work |

The all-in-one listener routes `/healthz` to the control plane and `/readyz` to the data plane. This makes the public
liveness check independent of Postgres while readiness covers the inference path. A data plane that rejects a newer
manifest keeps serving its last accepted snapshot and remains ready.

## Metrics

Configure OTLP export, Prometheus scraping, and alerting with the [OpenTelemetry guide](/docs/deployment/opentelemetry).

## Request correlation and logs

Every control-plane and data-plane response carries an `airmux`-generated UUIDv7 in `X-Request-ID`. A caller-supplied
value is replaced. Data-plane usage events use the same request ID, including all attempts in a fallback chain.

Production logs are newline-delimited JSON with `timestamp`, `level`, `logger`, `event`, and `message`. Logs emitted in
a request context also contain `request_id`; event-specific bounded fields appear at the top level. Start an
investigation from the response header, then search application logs and usage events for that value. Development mode
uses human-readable logs instead.

## CLI diagnostics

```bash theme={null}
airmux status
airmux doctor
airmux gateways list --all
airmux events tail
```

Use `doctor` first to distinguish local profile, control-plane authentication, and gateway readiness failures.
`gateways list` needs instance-scoped `data-planes.read` permission, so run it with an instance-scoped profile or
management key.

## Backups

Back up these as one recovery point:

1. Postgres
2. Runtime management credential files
3. Provider secret files
4. Each gateway's identity, cached bundles, and SQLite usage outbox

Restrict backup access because the default file secret store contains plaintext provider credentials. Test restoration on isolated infrastructure.

## Updates

Compose completes its migration and catalog jobs before starting the control plane. A direct control-plane restart does not
run either job. The process refuses to start unless the database is at the expected schema revision. A split gateway can keep
serving an accepted cached bundle while the control plane is unavailable. Because pre-1.0 releases do not promise cross-version
compatibility between planes, deploy one image digest across all components. Follow
[Upgrade and rollback](/docs/deployment/upgrades) to restore the database and state together if verification fails.

## Common failures

| Symptom                                 | Check                                                                                                                                                               |
| --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `bundle_unavailable`                    | Bundle poll authentication, control-plane URL, and cached state mount                                                                                               |
| Data-plane `/readyz` returns `503`      | Bundle poll authentication, control-plane reachability, and cached state mount                                                                                      |
| `bundle_manifest_rejected` is `1`       | Gateway logs for an invalid provider, model, credential, or policy reference                                                                                        |
| `credential_backend_unavailable`        | Secret-store mount, permissions, and availability                                                                                                                   |
| Usage missing from webapp               | Metering outbox metrics, export failures, and control-plane event-ingest metrics                                                                                    |
| New key returns `invalid_token`         | Wait for adoption, then inspect gateway readiness and logs                                                                                                          |
| Streams pause or buffer                 | Reverse-proxy buffering and read timeout                                                                                                                            |
| Migration or catalog job fails          | Run `docker compose logs migrate taxonomy` with the same Compose file and project options used to start the stack, then check database reachability and permissions |
| Control plane reports a schema mismatch | Run `airmux control-plane migrate` before starting it                                                                                                               |