Skip to content
Keln

Routing & failover

Keln routes every request and monitors it while it runs. It picks the provider with the lowest predicted TTFT for your prompt, starts a second attempt when a provider falls behind that prediction, and resumes the stream on another node if one fails mid-generation. Keln also reroutes stalled generations and never times out a request that is still being served. There are no fallback chains to configure and no providers to choose.

How a provider is picked

Serving capacity reports health continuously, queue depth, memory pressure, measured speed. For each request, the router filters to providers serving your model with real headroom right now, then picks the provider predicted fastest for your request. Two properties of the pick:

  • Predicted TTFT for your prompt size. The router learns each route's expected TTFT from live traffic, as a function of your prompt size. A 200k-token prompt gets a different prediction than a 500-token one, per route.
  • Keln skips slow providers while faster ones exist. A provider whose predicted first token falls outside the routing bar is passed over while faster capacity exists. This is a preference, not a cutoff: when that provider is the only capacity, it still serves.
  • What your request needs is part of the filter. Before speed is weighed, the pool is narrowed to capacity that can actually serve this request: tool calling or JSON mode in the transport you asked for (streaming and non-streaming count separately), image input, the ability to turn reasoning off, whatever parameters you demanded, and enough context window for your prompt. Route health filters too: Keln excludes a route that is failing verification or whose operator is not in good standing.

If your recent requests share a prompt prefix, routing sends them to the capacity that already has it cached. If a request needs something that capacity can't serve, capability wins and the request routes elsewhere.

Speed targets (SLO)

Keln sets two speed targets for every provider: 50 tokens/second sustained decode and first token within 5 seconds (judged on standard-size prompts; very large prompts are excluded, prefill time grows with prompt size). Keln measures every route against these targets continuously, from live probes and live traffic.

The targets drive routing, not just reporting:

  • Keln prefers supply that currently meets the targets, for every request.
  • Keln uses supply below the targets only when nothing better is available, and returns it to normal rotation once it measures healthy again.

Routing works to meet these targets on every request.

Predictive hedging

Each route is held to a predicted first-token window for your prompt size. When a route exceeds its window, a second route starts the same request in parallel. Keln uses whichever streams first and cancels the other. A hedged request bills once.

Predictive hedging: the router learns each route's expected first-token time for your prompt size; when the window passes with no token, a second route races and the winner streams. HEDGED DISPATCH, ONE REQUEST, TIMELINE first-token window, learned for this route × your prompt size ROUTE A fastest predicted predicted TTFT margin cancelled window passes with no token → hedge fires ROUTE B hedge runner first token stream → you t₀ · request arrives routing preference · slow routes lose the pick, not the request

A provider that falls behind its predicted window gets a second attempt in parallel; the slower one is canceled.

Mid-stream failover

If a node fails while streaming, Keln re-prompts another node with the text generated so far and resumes the stream. You receive one continuous response, and the prompt bills once.

Mid-stream failover: a node fails while generating; another node resumes from the text produced so far; the client sees one uninterrupted stream. MID-STREAM FAILOVER, ONE RESPONSE, TWO NODES YOUR STREAM what you see no visible seam NODE A serving fails mid-generation reroute + resume NODE B resumes picks up with the text generated so far

The resumed stream continues from the exact text already delivered.

Stall detection

Keln watches every active stream for stalls. It cuts and reroutes a generation that stops making progress, and leaves alone a legitimately long pause, such as a reasoning model thinking before its first visible token.

No give-up timer

Keln does not time out a request that is still being served. It retries a slow attempt on faster capacity, up to three times, and the wait allowed for a first token scales with your prompt size. An error is returned only when every capable provider was tried and none could serve, a retryable 502/503.

Targets, measured latency, and what failure looks like: Reliability & SLOs ↗