Retries & Fallback

Transient failures retry automatically; provider outages fall back automatically — applied to every call with zero configuration.

PleumRouter classifies each failed provider call by error type: only failures worth retrying are retried on the same provider, everything else falls back to the next provider immediately. For models served by multiple providers, candidates are tried in price order (or your personal routing preference order), so a single provider outage does not fail your request. It is fully automatic — nothing to add to your request.

Failure classification#

A failed attempt is classified by these rules.

429 (rate limit) — retried on the same provider, honoring the response's Retry-After header (both integer seconds and HTTP-date formats). The wait is capped at 10 seconds; without the header, exponential backoff is used.

5xx, network errors, malformed 200 responses — treated as transient and retried on the same provider, with exponential backoff (from 0.5s, doubling) plus ±10% jitter to avoid thundering-herd retries.

401 / 403 — a key problem; retrying the same provider cannot help, so the next provider is tried immediately without waiting.

Other 4xx (400, 404, 422, …) — likely a request-level problem, so the same provider is not retried; the next provider gets one attempt (provider-specific parameter dialects can make it succeed).

Retry counts: a single-provider model retries twice by default (3 attempts total); a multi-provider model moves straight to the next candidate with no same-provider retries by default. Switching providers happens immediately, with no wait. These values are operator-tunable.

When a fallback happens, the response's providerfield names the provider that actually served the request, and billing uses that provider's pricing (candidates can have different prices).

response after fallback

{
  "id": "req-8f3a...",
  "model": "gpt-4o",
  "provider": "openai",
  "choices": [{"message": {"role": "assistant", "content": "..."}}],
  "usage": {"prompt_tokens": 120, "completion_tokens": 48}
}

Streaming fallback#

In streaming, only failures before the first delta (token) fall back seamlessly to the next provider. Same-provider reconnect retries also happen only while zero deltas have been emitted.

If a failure happens after any delta has been sent to the client, the stream ends with an error event (partial: true) instead of falling back, and the tokens already delivered are billed (partial billing). Switching models mid-stream would risk duplicated output and double billing, so we deliberately do not fall back.