API: Evaluate

Method	Route	Does
`GET`	`{scope}/evaluations/catalog`	Models, sandboxes, and each harness’s options (`harnessOptions`)
`GET`	`{scope}/evaluations/options`	Tasks a comparison can run; any judge variables
`GET`, `POST`	`{scope}/evaluations/comparisons`	List or start comparisons
`GET`	`{scope}/evaluations/comparisons/{id}`	A comparison’s progress
`POST`	`…/comparisons/{id}/resume` · `/cancel`	Resume or stop it
`GET`	`{scope}/evaluations` · `/{id}`	Evaluations, with every trial
`GET`	`{scope}/evaluations/{id}/artifacts?name=…`	A file a trial’s agent left
`POST`	`{scope}/evaluations/{id}/cancel` · `/delete`	Stop or delete one

Request Body

Route	Body
`POST {scope}/evaluations/comparisons`	`{ id (UUID you generate; retries never start twice), models: [{ catalogId, credentialId, harnesses, thinking?, harness? }] (≤12), sandbox, sandboxCredentialId, tasks?: [{ runId, taskId }], skipCompleted?, judgeCredentialId?, harnesses?: [id] (≤4), environments?: [id] (≤4), baseline? }`

credentialId in a model choice is a saved credential’s ID, or managed-model. models[].harnesses are pi, codex, claude-code, mini-swe-agent, terminus-2, or muse-code. thinking is off, minimal, low, medium, high, xhigh, or max.

Comparisons

A comparison starts one evaluation per model, running every task on each harness as a trial. If the reply has submissionError, resume the same comparison with its id; it reuses the same evaluation IDs. For scores, read results rather than adding up trials.

Harnesses and Environments

harnesses (≤4) are saved harnesses’ ids; environments (≤4) are environments’ ids. Every task runs again with each harness, in each environment, and with each harness in each environment, at most 12 in all, beside the tasks as they are. baseline: false leaves out the tasks as they are. Results report each as its own configuration. models[].harness is one saved harness’s id, which that model runs in place of its agent as it is: on the tasks as they are and in each environment. Its agent must be the row’s. The app’s Run page picks harnesses this way.

Rule	Detail
Harness + Environment	The harness’s changes run on the tasks the environment built, named for both: `With Skills · No AGENTS.md`.
Environment Alone	Runs the tasks it built with the agent as it is, named for the environment.
Agent	A harness runs only on its `agent`, and is refused when no picked model runs on it.
Subagent Credential	Each harness’s `subagent.credentialId` serves its subagent model. Left out, the first of the workspace’s that can.
Identity	Two harnesses that change the same things are one configuration, whatever they are named.

conditions, the older form, still names each pair one by one: { saved?, environment?, subagentCredentialId? }, or the changes written out.

Subagent Models

A saved harness’s subagent runs each model as the orchestrator and another catalog model for the work it delegates: Claude Opus on Claude Code with DeepSeek subagents, say. Without one, subagents run on the model itself.

Rule	Detail
Harnesses	`claude-code` and `codex` only. The harness skips any other agent, and is refused when no picked model runs on either.
Models	Two different catalog models. The harness skips a model that is its subagent model.
Route	Both models are called through SelfBench’s gateway, so the subagent model may not be Dari’s router or on a custom endpoint. A trial that would not go through the gateway is refused.
Reasoning	`thinking` sets the orchestrator’s level. Every subagent call is served at the subagent model’s default level: `high` where it has one, whatever level the harness asks for.
Delegation	`subagent.prompt` is added to the harness’s system prompt (Codex’s developer instructions), after any the harness’s options append. Left out, the agent is told to delegate a meaningful part of each task; `""` tells it nothing. A trial’s `subagentCalls` counts the calls its subagent model served; `0` means the agent never delegated.
Shared Tools	Subagents use the harness’s tools, so Codex gets hosted web search and its image tool only when both models take them.
Results	A model with a subagent model is its own configuration, named for both, under its harness. `skipCompleted` and retries keep it apart from the model alone.

models[].subagent: { catalogId, credentialId } still names a subagent model for one model, under the baseline and every harness without its own, with the default prompt. Comparisons named it there before harnesses did. The trial’s key serves exactly its two models. Its cost, tokens, and model check all come from the gateway’s record of each call, priced at that call’s own model; a trial with an unpriced call has no cost rather than an estimate. billedCostUsd is set only when both credentials are API keys, since a sign-in’s calls are priced at list rates but not billed.

Model Costs

Trials call their model through SelfBench’s gateway, which uses the workspace credential and records each call’s cost: the provider’s own charge where it reports one, else the model’s list price. A trial with an unpriced call shows its cost as incomplete. The credential never reaches the agent, which gets a short-lived key for its one model, or for its two with a subagent model.

CLI

API

Public Results

Request Body

Comparisons

Harnesses and Environments

Subagent Models

Model Costs