One API for every model path.

The problem

AI providers are powerful. Their differences are expensive.

Every provider brings its own credentials, limits, model catalog, availability, and failure behavior. Without a routing layer, switching providers becomes application work.

• Provider-specific integrations

  Each SDK, header scheme, and model ID leaks into application code.

• Rate-limited or unavailable keys

  A single throttled key stalls requests that another key could have served.

• Model changes and upstream failures

  Catalogs shift and endpoints fail; your code absorbs the churn.

OctoFlux does not resolve every provider incompatibility. It supports OpenAI-compatible providers and configurable authentication headers.

The solution

Keep the integration stable. Change the infrastructure behind it.

Point an OpenAI-compatible client at OctoFlux once. Configure providers, models, aliases, and keys centrally, then let the gateway resolve each request.

Application → Provider A
Application → Provider B
Application → Provider C

Application → OctoFlux → Provider A
                       → Provider B
                       → Provider C

• One application-facing API
• Centralized provider configuration
• Explicit routing behavior
• Key rotation
• Error-aware failover

How it works

Watch a request find its path.

Quick start

Keep your existing OpenAI client.

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="OctoFlux-dev-key",
)

response = client.chat.completions.create(
    model="auto",
    messages=[
        {"role": "user", "content": "Hello"}
    ],
)

print(response.choices[0].message.content)

curl \
  -H "Authorization: Bearer $OctoFlux_CLIENT_KEY" \
  -H "Content-Type: application/json" \
  -X POST \
  http://localhost:8000/v1/chat/completions \
  -d '{
    "model": "auto",
    "messages": [
      {
        "role": "user",
        "content": "Hello"
      }
    ]
  }'

Model routing

Choose a model directly. Or define the path once.

{
  "model": "openai/gpt-oss-120b"
}

Provider ecosystem

Configured providers and example model IDs.

These are example configured model IDs, not permanent availability guarantees. Provider model catalogs change. OctoFlux can compare configured model IDs against a provider's /models response; NVIDIA NIM may return bare model names, so OctoFlux handles the corresponding publisher-prefixed configuration IDs.

Capabilities

Routing and failover, made explicit.

Health and operations

Know what is available before the request arrives.

Deployment

Run it where your application runs.

docker compose up -d --build

Architecture

Built around explicit infrastructure decisions.

• OpenAI-compatible request surface
• Configuration-driven provider model
• Deterministic routing pipeline
• Error classification before retry
• Health and cooldown state
• Pooled HTTP clients per provider
• Structured JSON logs
• No mandatory database, Redis, or Kafka
• Docker-ready deployment
• Mocked upstream tests with no required real provider calls