OpenTelemetry

airmux records operational metrics through OpenTelemetry. Each process defaults its service.name to airmux-control-plane or airmux-data-plane; set OTEL_SERVICE_NAME to override it.

Export metrics with OTLP

Set these variables on each process to send metrics to an OpenTelemetry Collector or another OTLP-compatible backend:

OTEL_METRICS_EXPORTER=otlp
OTEL_EXPORTER_OTLP_METRICS_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=http://otel-collector:4318/v1/metrics

OTLP export is disabled unless OTEL_METRICS_EXPORTER includes otlp. The exporter also honors standard OTLP headers, certificate, timeout, compression, and periodic export interval variables.

Scrape with Prometheus

Each standalone control-plane and data-plane listener exposes unauthenticated Prometheus metrics at /metrics. The all-in-one public listener does not proxy this path. Scrape the internal listeners from a monitoring network:

scrape_configs:
  - job_name: airmux-control-plane
    static_configs:
      - targets: ["control-plane:8000"]
  - job_name: airmux-data-plane
    static_configs:
      - targets: ["data-plane:8081"]

Do not expose either metrics endpoint to the public internet. Metrics omit organization, workspace, user, request, bundle, model, and provider identifiers. Request metrics use bounded dimensions such as route, dialect, stream mode, outcome, status class, and provider family.

Monitor the control plane

Metric	Meaning
`airmux_control_plane_http_requests_total`	Completed requests by route, method, outcome, and status class
`airmux_control_plane_http_request_duration_seconds`	Request latency by route, method, and outcome
`airmux_control_plane_inflight_requests`	Requests currently executing by route
`airmux_control_plane_throttle_decisions_total`	Allowed, quota-rejected, and capacity-rejected throttle decisions
`airmux_control_plane_event_ingest_total`	Usage-event ingest outcomes
`airmux_control_plane_event_ingest_duration_seconds`	Usage-event ingest latency
`airmux_control_plane_bundle_publications_total`	Bundle publisher outcomes
`airmux_control_plane_bundle_publication_pending`	Organizations with unpublished bundle generations
`airmux_control_plane_bundle_publication_oldest_age_seconds`	Time this process has continuously observed pending publication work

The publication-age gauge resets to zero after restart. The pending count comes from durable state and is the authoritative backlog size.

Monitor the data plane

Metric	Meaning
`airmux_data_plane_http_requests_total`	Completed requests by route, method, caller dialect, stream mode, outcome, and status class
`airmux_data_plane_http_request_duration_seconds`	Request latency by route, caller dialect, stream mode, and outcome
`airmux_data_plane_inflight_requests`	Requests currently executing by route and stream mode
`airmux_data_plane_upstream_attempts_total`	Provider attempts by egress family and outcome
`airmux_data_plane_upstream_attempt_duration_seconds`	Provider-attempt latency by egress family and outcome
`airmux_data_plane_credential_cache_requests_total`	Credential cache hits, misses, negative hits, and backend failures
`airmux_data_plane_credential_load_duration_seconds`	Secret-backend load latency by outcome
`airmux_data_plane_bundle_poll_total`	Unchanged, adopted, rejected, and failed bundle polls
`airmux_data_plane_bundle_snapshots`	Accepted bundle snapshots currently held by the worker
`airmux_data_plane_bundle_manifest_rejected`	`1` when the newest manifest was rejected
`airmux_data_plane_bundle_last_adopted_timestamp_seconds`	Unix timestamp of the last successful bundle adoption
`airmux_data_plane_metering_writer_queue_depth`	In-memory metering records waiting for durable admission
`airmux_data_plane_metering_writer_queue_capacity`	Configured in-memory metering queue capacity
`airmux_data_plane_metering_admission_total`	Accepted, full, and closed metering admissions
`airmux_data_plane_metering_outbox_pending`	Durable usage events awaiting export
`airmux_data_plane_metering_outbox_oldest_age_seconds`	Age of the oldest durable usage event
`airmux_data_plane_metering_exports_total`	Usage export outcomes
`airmux_data_plane_metering_export_duration_seconds`	Usage export latency by outcome

Set alerts

Alert on sustained changes rather than a single failed request. Useful starting signals include:

Readiness returning 503
Growing bundle publication backlog or age
bundle_manifest_rejected remaining 1, or rejected and failed bundle polls increasing
metering_admission_total increasing for full or closed
Growing metering outbox count or oldest-event age
Failed metering exports or credential backend failures
Sustained increases in failed HTTP requests or upstream attempts

Choose thresholds based on normal traffic and polling intervals for your deployment.

Get started

Use the platform

Deploy and operate

Concepts

Contributing

Export metrics with OTLP

Scrape with Prometheus

Monitor the control plane

Monitor the data plane

Set alerts

​Export metrics with OTLP

​Scrape with Prometheus

​Monitor the control plane

​Monitor the data plane

​Set alerts

Export metrics with OTLP

Scrape with Prometheus

Monitor the control plane

Monitor the data plane

Set alerts