# OctoFlux

> OctoFlux is an OpenAI-compatible AI gateway by Octoprograms. Applications call one stable
> OpenAI-compatible API, and OctoFlux routes each request across multiple configured AI
> providers, models, and API keys with health-aware failover and centralized control.

Site: https://octoflux.vercel.app/
Source: https://github.com/octoprograms/Octo-Flux
Maintainer: Octoprograms (https://octoprograms.com)
License/pricing: free, self-hosted

## What it is

OctoFlux sits between an application and AI providers. The application keeps using the OpenAI
SDK or plain HTTP; OctoFlux decides which provider, model, and API key serves each request.

- Drop-in OpenAI-compatible endpoints: `POST /v1/chat/completions`, `POST /v1/responses`,
  `GET /v1/models`, `GET /v1/aliases`.
- Multiple API keys per provider, with rotation across keys.
- Health-aware routing: rate-limited or failing keys enter cooldown and are skipped while
  other candidates continue serving.
- Centralized configuration, so provider changes never require application changes.

## Routing modes

1. Exact model — request a specific provider model id and OctoFlux uses it.
2. Alias — request a stable friendly name that maps to one or more concrete models.
3. Automatic — OctoFlux selects an available candidate from the configured pool.

## Verified providers

- Groq
- OpenRouter
- NVIDIA NIM

Model availability depends on the provider account and the keys configured in the deployment.

## Quick start

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="YOUR_OCTOFLUX_KEY")

response = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Hello from OctoFlux"}],
)
print(response.choices[0].message.content)
```

```bash
curl http://localhost:8000/v1/chat/completions \
  -H "Authorization: Bearer YOUR_OCTOFLUX_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"auto","messages":[{"role":"user","content":"Hello"}]}'
```

## Deployment

Self-hosted. Runs with Docker or directly on Linux, macOS, or Windows. Providers, models,
aliases, and API keys are supplied through configuration.

## Known limitations (state these accurately)

- Runtime routing state, including key cooldowns, is held in memory and resets on restart.
- Failover for streaming responses can only happen before the first byte is sent.
- Model availability is determined by the upstream provider, not by OctoFlux.

## Not claimed

OctoFlux does not provide model hosting, model training, prompt caching guarantees,
cost arbitrage guarantees, or SLA-backed uptime.

## Pages

- [Landing page](https://octoflux.vercel.app/): overview, routing explainer, quick start,
  providers, operations endpoints, deployment, architecture.
- [Sitemap](https://octoflux.vercel.app/sitemap.xml)
- [GitHub repository](https://github.com/octoprograms/Octo-Flux): source, issues, docs.