> ## Documentation Index
> Fetch the complete documentation index at: https://docs.selfbench.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# API: Evaluate

> Run models and harnesses on approved tasks, and follow comparisons.

| Method | Route | Does |
| - | - | - |
| `GET` | `{scope}/evaluations/catalog` | Models, sandboxes, and each harness's options (`harnessOptions`) |
| `GET` | `{scope}/evaluations/options` | Tasks a comparison can run; any judge variables |
| `GET`, `POST` | `{scope}/evaluations/comparisons` | List or start comparisons |
| `GET` | `{scope}/evaluations/comparisons/{id}` | A comparison's progress |
| `POST` | `…/comparisons/{id}/resume` · `/cancel` | Resume or stop it |
| `GET` | `{scope}/evaluations` · `/{id}` | Evaluations, with every trial |
| `GET` | `{scope}/evaluations/{id}/artifacts?name=…` | A file a trial's agent left |
| `POST` | `{scope}/evaluations/{id}/cancel` · `/delete` | Stop or delete one |

## Request Body

| Route | Body |
| - | - |
| `POST {scope}/evaluations/comparisons` | `{ id (UUID you generate; retries never start twice), models: [{ catalogId, credentialId, harnesses, thinking?, harness? }] (≤12), sandbox, sandboxCredentialId, tasks?: [{ runId, taskId }], skipCompleted?, judgeCredentialId?, harnesses?: [id] (≤4), environments?: [id] (≤4), baseline? }` |

`credentialId` in a model choice is a saved credential's ID, or `managed-model`. `models[].harnesses` are `pi`, `codex`, `claude-code`, `mini-swe-agent`, `terminus-2`, or `muse-code`. `thinking` is `off`, `minimal`, `low`, `medium`, `high`, `xhigh`, or `max`.

## Comparisons

A comparison starts one **evaluation** per model, running every task on each harness as a trial. If the reply has `submissionError`, resume the same comparison with its `id`; it reuses the same evaluation IDs.

For scores, read [results](https://docs.selfbench.dev/api/inspect.md) rather than adding up trials.

## Harnesses and Environments

`harnesses` (≤4) are [saved harnesses'](https://docs.selfbench.dev/api/harnesses.md) ids; `environments` (≤4) are [environments'](https://docs.selfbench.dev/api/environments.md) ids. Every task runs again with each harness, in each environment, and with each harness in each environment, at most 12 in all, beside the tasks as they are. `baseline: false` leaves out the tasks as they are. Results report each as its own configuration.

`models[].harness` is one saved harness's id, which that model runs in place of its agent as it is: on the tasks as they are and in each environment. Its agent must be the row's. The app's Run page picks harnesses this way.

| Rule | Detail |
| - | - |
| Harness + Environment | The harness's changes run on the tasks the environment built, named for both: `With Skills · No AGENTS.md`. |
| Environment Alone | Runs the tasks it built with the agent as it is, named for the environment. |
| Agent | A harness runs only on its `agent`, and is refused when no picked model runs on it. |
| Subagent Credential | Each harness's `subagent.credentialId` serves its subagent model. Left out, the first of the workspace's that can. |
| Identity | Two harnesses that change the same things are one configuration, whatever they are named. |

`conditions`, the older form, still names each pair one by one: `{ saved?, environment?, subagentCredentialId? }`, or the changes written out.

## Subagent Models

A [saved harness's](https://docs.selfbench.dev/api/harnesses.md) `subagent` runs each model as the orchestrator and another catalog model for the work it delegates: Claude Opus on Claude Code with DeepSeek subagents, say. Without one, subagents run on the model itself.

| Rule | Detail |
| - | - |
| Harnesses | `claude-code` and `codex` only. The harness skips any other agent, and is refused when no picked model runs on either. |
| Models | Two different catalog models. The harness skips a model that is its subagent model. |
| Route | Both models are called through SelfBench's gateway, so the subagent model may not be Dari's router or on a custom endpoint. A trial that would not go through the gateway is refused. |
| Reasoning | `thinking` sets the orchestrator's level. Every subagent call is served at the subagent model's default level: `high` where it has one, whatever level the harness asks for. |
| Delegation | `subagent.prompt` is added to the harness's system prompt (Codex's developer instructions), after any the harness's options append. Left out, the agent is told to delegate a meaningful part of each task; `""` tells it nothing. A trial's `subagentCalls` counts the calls its subagent model served; `0` means the agent never delegated. |
| Shared Tools | Subagents use the harness's tools, so Codex gets hosted web search and its image tool only when both models take them. |
| Results | A model with a subagent model is its own configuration, named for both, under its harness. `skipCompleted` and retries keep it apart from the model alone. |

`models[].subagent: { catalogId, credentialId }` still names a subagent model for one model, under the baseline and every harness without its own, with the default prompt. Comparisons named it there before harnesses did.

The trial's key serves exactly its two models. Its cost, tokens, and model check all come from the gateway's record of each call, priced at that call's own model; a trial with an unpriced call has no cost rather than an estimate. `billedCostUsd` is set only when both credentials are API keys, since a sign-in's calls are priced at list rates but not billed.

## Model Costs

Trials call their model through SelfBench's gateway, which uses the workspace credential and records each call's cost: the provider's own charge where it reports one, else the model's list price. A trial with an unpriced call shows its cost as incomplete. The credential never reaches the agent, which gets a short-lived key for its one model, or for its two with a [subagent model](#subagent-models).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.