CLI: Evaluate

R is --repo OWNER/NAME, or another scope.

Command	Does
`catalog R`	Models available for evaluations
`catalog R --harness H`	Options a saved harness can set on agent `H`
`compare R --model ID [--model ID ...]`	Run each model on every approved task
`comparisons R` · `comparison show\|resume\|cancel ID R`	Follow, resume, or stop comparisons
`runs R` · `run show\|cancel\|delete EVALUATION_ID R`	One model’s evaluation, with every trial

Compare Flags

Flag	Default
`--harness H` (repeatable): `pi`, `codex`, `claude-code`, `mini-swe-agent`, `terminus-2`, `muse-code`	`pi`
`--thinking LEVEL`: `off`, `minimal`, `low`, `medium`, `high`, `xhigh`, `max`	the model’s default
`--credential ID`	managed model
`--sandbox managed\|e2b\|modal\|daytona` and `--sandbox-credential ID`	managed sandbox
`--task RUN_ID/TASK_ID` (repeatable)	every approved task
`--skip-completed`	rerun everything
`--judge-credential ID`	required for tasks graded by an LLM judge
`--saved-harness ID` (repeatable): a saved harness	none
`--environment ID` (repeatable): a repository environment	none
`--no-baseline`	the tasks as they are run too
`--selection JSON\|@FILE`	a full comparison body in place of the flags

Every task runs again with each saved harness, in each environment, and with each harness in each environment (at most 12), beside the tasks as they are; see Harnesses and Environments. The older --saved-condition, --condition, and --harness-option flags still work. A harness with a subagent model runs each model as the orchestrator and the subagent model for delegated work, on the credential the harness names; see Subagent Models.

CLI

API

Public Results

Compare Flags