Tracing

Why keep a Trace?

A response alone tells you what text came back. It does not tell you why that model was selected, which fallback attempts failed, how much the call cost, which customer or task it belonged to, or whether the answer eventually helped. A Trace keeps those facts together under one stable trace_id.

That record serves three stages of the product lifecycle. In production, it helps you debug individual requests and measure real behavior. In evaluation, it preserves every decision in an environment episode. In training, it becomes a trajectory with observations, actions, and rewards.

Trace kinds

• production — one live model request, including route attempts, response, usage, latency, metadata, and labels
• episode — one complete agent run in an environment, including observations, decisions, tools, stopping, and reward
• llm_call — an optional child record for one model call inside an episode

The schema is shared deliberately. A support request captured in production and a support task run in an eval can flow through the same inspection, labeling, dataset, and training tools.

Example: capture a support request

A customer reports a duplicate charge. We keep the prompt and response because this approved support dataset will be reviewed later. We attach a ticket id for joining back to the product and a workflow tag for filtering. Redaction runs before SQLite receives the record.

from plural import Plural, Message, Redactor, SQLiteSink

sink = SQLiteSink(".plural/support.sqlite")
client = Plural(
    base_url="https://api.pluralintel.com/v1",
    sink=sink,
    capture_content=True,
    redactor=Redactor(
        fields={"metadata.customer_email"},
        patterns=[r"\b[\w.-]+@[\w.-]+\.\w+\b"],
    ),
)
response = client.chat(
    model="openai/gpt-5.6-luna",
    messages=[
        Message(
            role="user",
            content="I was charged twice for order A123.",
        )
    ],
    metadata={
        "ticket_id": "T-1042",
        "customer_email": "customer@example.com",
    },
    tags={"workflow": "refund"},
)
trace_id = response.raw["plural_trace_id"]
client.flush()

Content capture is off by default. Turn it on only when the text is useful and permitted to persist. Tool arguments, tool results, and custom metadata may contain sensitive values too, so add field or pattern rules for your schema rather than relying only on the default content drop.

Inspect the call

After flushing the background writer, load the same trace by id. Its first LLMCall shows the model and provider that answered, token usage, cost, latency, and every attempted route. Request and response content are present because this client opted into capture.

trace = sink.get(trace_id)
assert trace is not None

call = trace.steps[0]

print(trace.trace_id, trace.trace_kind)
print(trace.metadata["ticket_id"])
print(call.model, call.provider)
print(call.usage, call.cost, call.latency_ms)
print(call.attempts)

Open the same hosted record on Traces when the call went through the Plural gateway. Local JSONL is convenient for shipping, SQLite for querying and late labels, and OpenTelemetry for an existing observability stack. MultiSink can write to more than one destination.

Add what happened after the response

The model call ends before the business outcome is known. When the customer confirms the issue is resolved, attach reward, structured labels, and optional reviewer feedback to the original trace. This turns an observability record into a supervised example.

client.label(
    trace_id,
    reward=1.0,
    labels={"resolution": "refund_issued"},
    feedback="Customer confirmed the duplicate charge was refunded.",
)
client.flush()

labeled = sink.get(trace_id)
print(labeled.outcome)

Collect the examples worth learning from

One labeled trace explains one ticket. A Dataset turns a filtered collection into a named, versioned artifact. Here we keep only successful refund conversations and save the complete records plus a content-hash manifest.

from plural import Dataset, TraceFilter

helpful_refunds = Dataset.from_sink(
    sink,
    "helpful-refund-tickets",
    version="2026.08",
    filter=TraceFilter(trace_kind="production"),
    where=lambda trace: (
        trace.tags.get("workflow") == "refund"
        and trace.outcome is not None
        and trace.outcome.reward == 1.0
    ),
)
helpful_refunds.save("data/helpful-refunds.jsonl")
print(helpful_refunds.content_hash, len(helpful_refunds))

client.close()

A Trace dataset contains observed behavior. A TaskDataset contains work you intend to run in an environment. Benchmarks consume task datasets and produce new episode traces that can flow back into a Trace dataset.

From production call to agent episode

The support call above has one LLM step. An environment episode extends the same Trace with a task id, environment version and fingerprint, agent-visible observations, parsed actions, tool results, safe state snapshots, stopping reason, and scorer outcome. Those ordered decisions can be flattened with trace.transitions() for RL.

Send X-Plural-Environment when hosted production traffic belongs to a stored harness. That link lets you compare real traffic with eval rollouts for the same task.