Orchestrators

An orchestrator is a routing rule that bundles multiple models into one virtual model. Alongside the four existing policy types (fallback, weighted, latency, auto), two archetypes were added — cascade and parallel+synthesis — and every policy is now invocable as "model": "orch/<slug>" (the legacy policy/<slug> still works). Build them with the form on the Orchestrators dashboard page, or edit the JSON DSL directly in the advanced tab.

Archetypes#

fallback, weighted, latency, auto — identical to the existing routing policies. See the routing policies doc for the rules.

cascade — register entries as tiers ordered cheap → strong; each request starts at the tier matching its difficulty and escalates one level on failure. Easy requests finish on a cheap model; only hard ones climb to the strong models, keeping average cost low.

parallel — workers (entries) are called simultaneously and a synthesis model critically verifies and merges the successful answers into one final response.

Cascade — difficulty-based tier escalation#

Starting-tier detection combines two signals. ① hard_categories — a pure-regex intent classifier (no external transfer, zero cost) starts matching requests directly at the top tier. ② judge_model— when set, one cheap call picks the starting tier (billed as a normal child call). If the judge fails or doesn't answer with a number, it falls back to ①.

Escalation happens on two signals: a tier call that fails moves to the next tier, and with response_check enabled, a successful but empty answer (blank text or content-filtered) also escalates. Discarded calls were already settled, so their usage and cost are included in the final response totals (honest usage reporting).

For streaming, tier detection (heuristic + judge) completes before the first token and streaming starts from the chosen tier. Response-based re-checks (response_check) apply to non-streaming calls only.

Parallel+synthesis — many workers, one answer#

When two or more workers succeed, synthesis.model verifies and merges the candidates (customize the tone and criteria with synthesis.prompt; a default instruction is used when unset). If no synthesis model is set — or synthesis fails — the first successful worker's answer is returned. A single successful worker also returns as-is without synthesis.

Set deep_categories to run in parallel only for matching intents; every other request is served by a single call to the first worker (fast path). When unset, all requests run in parallel.

Because the final answer is only certain after synthesis, parallel does not support streaming — calling with stream=true returns a 400 error before SSE starts.

Invocation & billing#

Set "model": "orch/<slug>" in a normal chat call. It works with any OpenAI SDK and your plm_ key. Org-shared orchestrators can be invoked by every member of the organization with their own keys.

curl https://apirouter.pleum.ai/v1/chat/completions \
  -H "Authorization: Bearer plm_..." \
  -H "Content-Type: application/json" \
  -d '{
    "model": "orch/prod-cascade",
    "messages": [
      {"role": "user", "content": "Summarize this document."}
    ]
  }'

Billing is passthrough on each child call that actually ran, and the response's usage and cost are the sum across all children — including judge calls, discarded tiers, and synthesis. There is no extra orchestrator fee.

Two caps protect against runaway orchestration: limits.max_total_attempts (1–10, further narrowing the global cap of 6) and limits.max_cost_krw — once the cumulative actual cost of child calls exceeds it, no further escalation happens.

Pin provider preferences for child calls (sort, ignore, max_price, …) with routing_prefs. A provider object sent in the request body always takes precedence.

Organization sharing#

Share an orchestrator with your organization and every member can invoke it; edits and deletion belong to the author and the org's owner/admin (admins can also manage members' private orchestrators). Unshared orchestrators are visible to the author only.

JSON DSL#

The advanced dashboard tab and the /v1/routing-policies API accept the same JSON structure — the form and the DSL share one schema, so you can copy the JSON to share it with your team. Entries are 1–10 real, active, priced models; other orchestrators (orch/…, policy/…) and virtual routing parents cannot be referenced.

curl -X POST https://apirouter.pleum.ai/v1/routing-policies \
  -H "Authorization: Bearer <JWT>" \
  -H "Content-Type: application/json" \
  -d '{
    "slug": "prod-cascade",
    "display_name": "Production cascade",
    "policy_type": "cascade",
    "entries": [
      {"model": "gpt-5-mini"},
      {"model": "claude-fable-5"}
    ],
    "config": {
      "cascade": {
        "hard_categories": ["code"],
        "judge_model": "gpt-5-mini",
        "response_check": true
      },
      "routing_prefs": {"sort": "latency", "ignore": ["slow-provider"]},
      "limits": {"max_total_attempts": 4, "max_cost_krw": 3000}
    }
  }'

{
  "slug": "review-panel",
  "policy_type": "parallel",
  "entries": [{"model": "claude-fable-5"}, {"model": "gpt-5.5"}, {"model": "gemini-3.1-pro"}],
  "config": {
    "parallel": {
      "deep_categories": ["code", "reasoning"],
      "synthesis": {
        "model": "claude-fable-5",
        "prompt": "Verify the answers like a senior reviewer and merge the strongest parts."
      }
    }
  }
}