Benchmarks

Why benchmark?

There is no universally best model. A model that wins on coding may lose on support, tool use, long documents, or your company's edge cases. With new models arriving constantly, evaluating each integration by hand does not scale.

A benchmark makes the question precise: for this versioned set of tasks, in this versioned environment, how well does each model or policy perform? It runs every target on the same cases and aggregates reward, failures, latency, and cost.

For a simple prompt eval, the environment can be a one-turn harness whose scorer grades the response. For an agent eval, the benchmark scores the full interaction: observations, decisions, tool calls, state changes, stopping, and final outcome. Both produce the same Report.

Example: which model handles refunds best?

Continue the refund-support example. The task dataset contains duplicate charges, ineligible orders, ambiguous requests, and tool failures. The environment owns eligibility and refund tools. Its scorer rewards the correct final state. The only thing that changes is the model acting as policy.

from pathlib import Path
from plural import Benchmark, Plural, TaskDataset
from my_app.environments import RefundEnv

dataset = TaskDataset.load("data/refund-support.jsonl")
env = RefundEnv()
client = Plural(base_url="https://api.pluralintel.com/v1")

report = Benchmark(
    env,
    models=["openai/gpt-5.6-luna", "google/gemini-3.7-flash"],
    client=client,
    repeats=3,
    concurrency=8,
).run(dataset=dataset)
print(report.environment, report.win_rates)
print(report.manifest.task_set)
Path("report.json").write_text(report.to_json())
Path("report.md").write_text(report.to_markdown())

repeats exposes model variance. concurrency controls how many episodes run at once. Every target receives the same ordered task and repeat slots, so one model is never compared against an easier sample.

Read the report

Start with aggregate model stats. mean_reward summarizes task performance. Failure counts show reliability. p50 and p95 latency show the user experience, while mean and total cost show the price of that quality.

Then inspect paired win rates. Models are compared only on the same (task_id, repeat) slot; ties count as half a win, and failed or unscored pairs are reported as excluded instead of quietly changing the denominator.

for model, stats in report.models.items():
    print(
        model,
        stats.mean_reward,
        stats.failures,
        stats.p95_latency_ms,
        stats.total_cost,
    )

for pair in report.win_rate_pairs:
    print(
        pair.model_a,
        pair.model_b,
        pair.win_rate,
        pair.compared,
        pair.excluded,
    )

for case in report.cases:
    print(case.key, case.reward, case.trace_id)

Every case links back to its episode trace. When an aggregate surprises you, open the exact trajectory to see the observation, action, tool result, stop reason, score, latency, and cost. A benchmark tells you where to look; the trace explains why.

Know that two runs are comparable

The report manifest records the environment version and fingerprint, task-dataset hash, ordered task ids, targets, repeats, concurrency, runtime fingerprints, and package version. That provenance prevents a changed harness or changed prompt set from masquerading as a model improvement.

Benchmark any policy, not only catalog models

A rules engine, fine-tuned agent, or learned RL policy can occupy the same policy seat as an LLM. Plural owns the Policy contract: act() receives the exact ChatRequest built by the environment and returns a ChatResponse, one ParsedAction, or a list of actions.

Define business-specific policies in your code against that public contract. The example implements a deterministic rules baseline inline, so there is no fictional policy import or hidden adapter. Use PluralPolicy when the policy is a catalog model and ScriptedPolicy when you need fixed actions in a harness test.

Benchmark.from_policies() takes named zero-argument factories so every case gets fresh policy state. A class such as RefundRulesPolicy is itself a factory; model adapters use a lambda to bind the client and model id. Each worker also receives a fresh environment, preventing one episode from contaminating the next.

from pathlib import Path
from plural import (
    Benchmark,
    ChatRequest,
    Plural,
    PluralPolicy,
    JSONLSink,
    Policy,
    Report,
    TaskDataset,
    TraceContext,
    TraceWriter,
)
from plural.tracing import ParsedAction
from my_app.environments import RefundEnv

class RefundRulesPolicy(Policy):
    """Deterministic baseline for the refund environment."""

    def act(
        self,
        request: ChatRequest,
        *,
        trace_context: TraceContext | None = None,
    ) -> ParsedAction:
        del trace_context
        observation = str(request.messages[-1].content)
        if "Eligibility checked: False" in observation:
            return ParsedAction(
                name="check_eligibility",
                arguments={},
            )
        return ParsedAction(name="issue_refund", arguments={})

env = RefundEnv()
dataset = TaskDataset.load("data/refund-support.jsonl")
client = Plural(base_url="https://api.pluralintel.com/v1")

writer = TraceWriter(JSONLSink("policy-episodes.jsonl"))
try:
    report = Benchmark.from_policies(
        env,
        {
            "rules-v1": RefundRulesPolicy,
            "model-agent": lambda: PluralPolicy(
                client,
                "openai/gpt-5.6-luna",
            ),
        },
        repeats=2,
        concurrency=4,
        environment_factory=RefundEnv,
        trace_writer=writer,
    ).run(dataset=dataset)
finally:
    writer.close()

baseline = Report.model_validate_json(Path("baseline.json").read_text())
delta = report.compare(baseline, tolerance=0.02)
print(delta["regressions"])

Policy benchmarks do not have a client writer, so pass trace_writer when you want every scored trajectory available for inspection or further RL. Use runtime_factory when each worker needs a fresh sandbox or remote tool runtime.

Catch regressions in CI

report.compare(baseline, tolerance=0.02) checks mean reward by matching target name. By default it first verifies environment, runtime, task-set, order, and repeat compatibility. Fail CI when the returned regressions list is non-empty.

Keep reports with the workspace

After a local or CI run, paste report.model_dump() on Benchmarks and link it to the workspace environment that held the tasks. Hosted live runs are not offered; keep the scored job next to your own keys and credits.