Routing Policies

A routing policy bundles multiple models into one invocation rule — store up to 10 model entries under a single slug, then set "model": "policy/<slug>" in a normal chat call to route by that rule. There are three types: fallback, weighted, and latency (plus orchestrators cascade/parallel). Policies belong to your account and are invoked with your plm_ key.

Policy types#

fallback — entries are tried in the order you registered them. Give an entry retries (0–3) to repeat that same entry before moving to the next one. The first successful response is returned as-is.

weighted — requests are distributed proportionally to each entry's weight (1–10000). Pass a trace_id in the request body and the selection becomes deterministic on its hash — the same trace_id always routes to the same model (sticky). If the picked entry fails, the rest are tried in descending weight order.

latency — entries are reordered fastest-first using response times measured over the last hour (the same scorer as the :nitro suffix). With no measurement data, entries are tried in their declared order.

Invoking a policy#

In a normal chat call, set "model": "policy/<slug>". It works with the OpenAI SDK and your plm_ key; the response's model and provider fields show what actually served the request. Billing is based on the model and provider pricing of the entry that succeeded.

curl https://apirouter.pleum.ai/v1/chat/completions \
  -H "Authorization: Bearer $PLEUM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "policy/prod-chat",
    "messages": [
      {"role": "user", "content": "Summarize this document."}
    ]
  }'

trace_id (max 128 chars) is a routing-only field — it seeds the weighted policy's sticky selection and is never forwarded to the upstream provider. Put a user ID or session ID there to pin the same user or session to the same model.

curl https://apirouter.pleum.ai/v1/chat/completions \
  -H "Authorization: Bearer $PLEUM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "policy/ab-test",
    "trace_id": "user-1234",
    "messages": [
      {"role": "user", "content": "Hello!"}
    ]
  }'

In streaming, only failures before the first token fall back seamlessly to the next entry. Once any token has been sent to the client, a failure ends the stream instead of falling back (to avoid duplicate output). Per-entry retries apply to non-streaming calls only.

Managing policies#

Create policies on the dashboard's Routing Policies page, or via the /v1/routing-policies management API (JWT auth). See the API reference for field details.