Skip to content

Routing & failover

Every request is routed and monitored; a route that underperforms its own track record is raced or replaced. There are no fallback chains to configure and no providers to choose.

How a route is picked

Serving capacity reports health continuously, queue depth, memory pressure, measured speed. For each request, the router filters to routes serving your model with real headroom right now, then picks the route predicted fastest for your request. Two properties of the pick:

  • Predicted first-token time, not averages. The router learns each route's expected time-to-first-token from live traffic, as a function of your prompt size. A 200k-token prompt gets a different prediction than a 500-token one, per route.
  • Slow routes lose the pick, not the request. A route whose predicted first token falls outside the routing bar is passed over while faster capacity exists. This is a preference, not a give-up point: when that route is the only capacity, it still serves.
  • What your request needs is part of the filter. Before speed is weighed, the pool is narrowed to capacity that can actually serve this request: tool calling or JSON mode in the transport you asked for (streaming and non-streaming count separately), image input, the ability to turn reasoning off, whatever parameters you demanded, and enough context window for your prompt. Route health filters too, a route currently failing verification, or whose operator is not in good standing, is out of the pool for the pick.

If your recent requests share a prompt prefix, routing sends them to the capacity that already has it cached. If a request needs something that capacity can't serve, capability wins and the request routes elsewhere.

Speed targets (SLO)

Keln holds supply to two measured speed targets: 50 tokens/second sustained decode and first token within 5 seconds (judged on standard-size prompts; very large prompts are excluded, prefill time grows with prompt size). Every route is measured against these bars continuously, from live probes and live traffic.

The targets drive routing, not just reporting:

  • Supply currently proving it meets the targets is preferred for every request.
  • Supply below the targets is used only when nothing better is available, and returns to normal rotation once it measures healthy again.

Routing works to meet these targets on every request, and delivered performance against them is published in the model catalog.

Predictive hedging

Each route is held to a predicted first-token window for your prompt size. When a route exceeds its window, a second route starts the same request in parallel; whichever streams first is used, the other is cancelled. A hedged request bills once, the racing attempt is part of the platform service.

Predictive hedging: the router learns each route's expected first-token time for your prompt size; when the window passes with no token, a second route races and the winner streams. HEDGED DISPATCH, ONE REQUEST, TIMELINE first-token window, learned for this route × your prompt size ROUTE A fastest predicted predicted TTFT margin cancelled window passes with no token → hedge fires ROUTE B hedge runner first token stream → you t₀ · request arrives routing preference · slow routes lose the pick, not the request

A route that falls behind its predicted window is raced by a second one; the slower attempt is cancelled.

Mid-stream failover

If a node fails while streaming, Keln re-prompts another node with the text generated so far and resumes the stream. You receive one continuous response, and the prompt bills once.

Mid-stream failover: a node fails while generating; another node resumes from the text produced so far; the client sees one uninterrupted stream. MID-STREAM FAILOVER, ONE RESPONSE, TWO NODES YOUR STREAM what you see no visible seam NODE A serving fails mid-generation reroute + resume NODE B resumes picks up with the text generated so far

The resumed stream continues from the exact text already delivered.

Stall detection

Keln watches every active stream for stalls: a generation that stops making progress is cut and rerouted, while legitimately long silences (a reasoning model thinking before its first visible token) are recognized and left alone.

No give-up timer

Keln does not time out a request that is still being served. A slow attempt is hedged and re-raced onto faster capacity (up to three times), and the wait allowed for a first token scales with your prompt size. An error is returned only when every capable route was tried and none could serve, a retryable 502/503.