Skip to main content
Run a standalone gateway when you need an inference endpoint without the full platform. You manage its models, keys, and provider credentials locally.

Install

You need Python 3.13+ and uv. Clone the repository and sync the CLI package:
Run CLI commands from this checkout with uv run airmux.

Start the gateway

Set a provider key, then initialize and start the gateway. The initializer creates a local configuration and inference key:
The gateway runs in the foreground at http://127.0.0.1:8080. Keep the provider key available to the process. To use another provider, export its key before starting the gateway. The initializer prints ready-to-run commands for your setup.

Send a request

In another terminal, copy the inference key created by initialization and send a request using a model ID from the local taxonomy:
Change the model value to another ID in your taxonomy. See the inference reference for other request formats. Verify streaming with the same inference key. --no-buffer prints each event as it arrives; a successful stream ends with data: [DONE]:

Run in a container

Build the image from the checkout after running gateway init. The container reads the same .airmux directory and provider key as the local setup:
Keep the container running and use the requests above against http://127.0.0.1:8080. The host user runs the gateway so it can read the private inference key. Set AIRMUX_CONFIG_DIR to the parent of .airmux if you initialized it outside the checkout. Stop the local Python gateway before using the same host port. Pass the matching provider key with -e if you use a provider other than OpenAI.

Customize local setup

gateway init creates the starter configuration and taxonomy in .airmux. Use --directory to choose another location or --taxonomy to use your own model catalog. See the configuration reference for advanced settings. Standalone mode has no management API, webapp, or managed usage history. Its keys share one local workspace, and provider credentials come from environment variables. Usage events are discarded by default; see the event outbox settings to write them to a local file.

Upgrade and roll back

Back up .airmux/ and record the current release before changing the CLI. Check out the target release tag, run uv sync --package airmux --frozen, and validate the existing configuration:
Before removing the backup, make a buffered and streaming request. If the new release fails, reinstall the previous release and restore its matching .airmux/ directory. Stop the gateway before restoring state.