Skip to content
PleumRouter
Get started

Benchmark Smart

Auto-routes to the top LMArena model per category. No need to pick a model — just send the best result.

Send "model": "benchmark_smart" and Pleum classifies the request intent (code, reasoning, vision, translation…) then substitutes the top benchmark model for that category. The LMArena leaderboard is collected weekly, so you always reach the latest SOTA without tracking it yourself.

benchmark_smart — code request
curl https://apirouter.pleum.ai/v1/chat/completions \
  -H "Authorization: Bearer plm_..." \
  -H "Content-Type: application/json" \
  -d '{
    "model": "benchmark_smart",
    "messages": [
      {"role": "user", "content": "Write a Python function to dedupe a list."}
    ]
  }'

How it works#

The intent is classified from the request body (regex, no upstream call), the highest-scoring model for that category is selected, and a single call is made. You are billed for the substituted real model — benchmark_smart is a virtual model with no price of its own.

Categories are code, reasoning, vision, translation, creative, and general (6 total). Vision requests (with images) automatically go to the top vision model.

The score table is collected from LMArena every Sunday 09:00 (KST) into a draft, then promoted to live at 10:00. On collection failure the previous table is kept; if there's no table at all, routing falls back to a static catalog mapping — it never stalls.

Fallback chain#

If the top model is temporarily disabled, it falls back to the next-best in the same category, and then to the general category. The decision happens before credit hold, so a miss bills nothing.

Selected Smart (pool-scoped)#

To run AUTO only inside a pool of models you choose, use the auto type of routing policies (Selected Smart). It picks the top model in the pool by score, and on failure falls back through the rest of the pool.

Related modes#

Perfect runs the top N models from the same score table in parallel and synthesizes them (price-agnostic — slower and pricier, maximum accuracy). For statistics-based selection, enable analytics_smart (currently identical to benchmark; evolves to traffic learning later).

benchmark_smart — vision request
curl https://apirouter.pleum.ai/v1/chat/completions \
  -H "Authorization: Bearer plm_..." \
  -H "Content-Type: application/json" \
  -d '{
    "model": "benchmark_smart",
    "messages": [
      {"role": "user", "content": [
        {"type": "text", "text": "What is in this image?"},
        {"type": "image_url", "image_url": {"url": "https://example.com/cat.png"}}
      ]}
    ]
  }'