> ## Documentation Index
> Fetch the complete documentation index at: https://docs.airmux.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# OpenTelemetry

> Export airmux metrics through OTLP or scrape Prometheus endpoints, and choose useful alerts.

`airmux` records operational metrics through OpenTelemetry. Each process defaults its `service.name` to
`airmux-control-plane` or `airmux-data-plane`; set `OTEL_SERVICE_NAME` to override it.

## Export metrics with OTLP

Set these variables on each process to send metrics to an OpenTelemetry Collector or another OTLP-compatible backend:

```bash theme={null}
OTEL_METRICS_EXPORTER=otlp
OTEL_EXPORTER_OTLP_METRICS_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=http://otel-collector:4318/v1/metrics
```

OTLP export is disabled unless `OTEL_METRICS_EXPORTER` includes `otlp`. The exporter also honors standard OTLP headers,
certificate, timeout, compression, and periodic export interval variables.

## Scrape with Prometheus

Each standalone control-plane and data-plane listener exposes unauthenticated Prometheus metrics at `/metrics`. The
all-in-one public listener does not proxy this path. Scrape the internal listeners from a monitoring network:

```yaml theme={null}
scrape_configs:
  - job_name: airmux-control-plane
    static_configs:
      - targets: ["control-plane:8000"]
  - job_name: airmux-data-plane
    static_configs:
      - targets: ["data-plane:8081"]
```

Do not expose either metrics endpoint to the public internet. Metrics omit organization, workspace, user, request, bundle,
model, and provider identifiers. Request metrics use bounded dimensions such as route, dialect, stream mode, outcome,
status class, and provider family.

## Monitor the control plane

| Metric                                                       | Meaning                                                              |
| ------------------------------------------------------------ | -------------------------------------------------------------------- |
| `airmux_control_plane_http_requests_total`                   | Completed requests by route, method, outcome, and status class       |
| `airmux_control_plane_http_request_duration_seconds`         | Request latency by route, method, and outcome                        |
| `airmux_control_plane_inflight_requests`                     | Requests currently executing by route                                |
| `airmux_control_plane_throttle_decisions_total`              | Allowed, quota-rejected, and capacity-rejected throttle decisions    |
| `airmux_control_plane_event_ingest_total`                    | Usage-event ingest outcomes                                          |
| `airmux_control_plane_event_ingest_duration_seconds`         | Usage-event ingest latency                                           |
| `airmux_control_plane_bundle_publications_total`             | Bundle publisher outcomes                                            |
| `airmux_control_plane_bundle_publication_pending`            | Organizations with unpublished bundle generations                    |
| `airmux_control_plane_bundle_publication_oldest_age_seconds` | Time this process has continuously observed pending publication work |

The publication-age gauge resets to zero after restart. The pending count comes from durable state and is the authoritative
backlog size.

## Monitor the data plane

| Metric                                                    | Meaning                                                                                     |
| --------------------------------------------------------- | ------------------------------------------------------------------------------------------- |
| `airmux_data_plane_http_requests_total`                   | Completed requests by route, method, caller dialect, stream mode, outcome, and status class |
| `airmux_data_plane_http_request_duration_seconds`         | Request latency by route, caller dialect, stream mode, and outcome                          |
| `airmux_data_plane_inflight_requests`                     | Requests currently executing by route and stream mode                                       |
| `airmux_data_plane_upstream_attempts_total`               | Provider attempts by egress family and outcome                                              |
| `airmux_data_plane_upstream_attempt_duration_seconds`     | Provider-attempt latency by egress family and outcome                                       |
| `airmux_data_plane_credential_cache_requests_total`       | Credential cache hits, misses, negative hits, and backend failures                          |
| `airmux_data_plane_credential_load_duration_seconds`      | Secret-backend load latency by outcome                                                      |
| `airmux_data_plane_bundle_poll_total`                     | Unchanged, adopted, rejected, and failed bundle polls                                       |
| `airmux_data_plane_bundle_snapshots`                      | Accepted bundle snapshots currently held by the worker                                      |
| `airmux_data_plane_bundle_manifest_rejected`              | `1` when the newest manifest was rejected                                                   |
| `airmux_data_plane_bundle_last_adopted_timestamp_seconds` | Unix timestamp of the last successful bundle adoption                                       |
| `airmux_data_plane_metering_writer_queue_depth`           | In-memory metering records waiting for durable admission                                    |
| `airmux_data_plane_metering_writer_queue_capacity`        | Configured in-memory metering queue capacity                                                |
| `airmux_data_plane_metering_admission_total`              | Accepted, full, and closed metering admissions                                              |
| `airmux_data_plane_metering_outbox_pending`               | Durable usage events awaiting export                                                        |
| `airmux_data_plane_metering_outbox_oldest_age_seconds`    | Age of the oldest durable usage event                                                       |
| `airmux_data_plane_metering_exports_total`                | Usage export outcomes                                                                       |
| `airmux_data_plane_metering_export_duration_seconds`      | Usage export latency by outcome                                                             |

## Set alerts

Alert on sustained changes rather than a single failed request. Useful starting signals include:

* Readiness returning `503`
* Growing bundle publication backlog or age
* `bundle_manifest_rejected` remaining `1`, or rejected and failed bundle polls increasing
* `metering_admission_total` increasing for `full` or `closed`
* Growing metering outbox count or oldest-event age
* Failed metering exports or credential backend failures
* Sustained increases in failed HTTP requests or upstream attempts

Choose thresholds based on normal traffic and polling intervals for your deployment.