> ## Documentation Index > Fetch the complete documentation index at: https://docs.selfbench.dev/llms.txt > Use this file to discover all available pages before exploring further. # How It Works > How merged pull requests and described vibe evals become verified tasks, and how results are counted. ![A merged PR becomes a verified task; agents then attempt it and their accuracy is plotted against cost.](https://mintcdn.com/selfbench/M4CKanW3oxa2SYd3/media/how-it-works-light.gif?s=cc8b68fd78ebeb68e45ed1236dae6c11) ## Pipeline 1. **Provenance.** A batch lists up to 500 merged PRs from non-bot authors, with their exact title and body. Common credential forms are redacted. 2. **Discovery.** A batch proposes PRs to meet its counts. **Add PRs** skips this and authors each PR you pick. 3. **Authoring.** An agent starts from the commit before the change. It writes the instruction, the reference patch (`gold.patch`), held-out tests, a Dockerfile, and `smoke.sh` and `run-tests.sh`, then calls `verify`. 4. **Gates.** Each check must pass: build, smoke (the environment works for an agent), nop (new tests fail on the base commit), oracle (the reference patch passes), and determinism (the tests pass again). 5. **Review.** An independent reviewer accepts, rejects, or sends the task back. Three failed rounds reject the candidate; nothing is backfilled. 6. **You review.** Only tasks you approve are evaluated. ## Difficulty A task takes the hardest tier its reference patch reaches, up to the tier you asked for. | Tier | Reference patch | | - | - | | easy | 20+ changed lines across 1 implementation path | | medium | 50+ changed lines across 2 implementation paths | | hard | 100+ changed lines across 3 implementation paths | Hidden tests must share no files with the reference patch, and they may not import its private helpers or pin internal details. They exercise a public API, command, or persistence boundary. ## What the Agent Sees The agent gets the base snapshot, the instruction, and its environment. It never sees the held-out tests or the reference solution. Its network reaches only its model's host. The verifier applies its patch to a clean copy, then restores the held-out test files, so tampering can only produce a tree it could have written directly. ## Vibe Evals A vibe eval starts from a description instead of a PR. * A **repository vibe eval** starts from the repository at a commit you choose (a SHA, a branch, or `HEAD`). It is pinned to that commit and can be released with its repository. * A **workspace vibe eval** lives in an **eval**, a collection of your own tasks with no repository behind it. The author writes the starting files, and the tests read only what the agent leaves in `/app`. Evals are never released. Name the mistake the agent made and what a careful person would do. Attach real files where you can. One vibe eval is one data point, so write several. ## Harnesses and Repo Ablations A run can run every task again with something changed, beside the **Baseline**, the tasks and agents as they are. Two things can change: | Changes | Is | Belongs To | | - | - | - | | The agent | A **harness**: skills, instructions, options, a subagent model | The workspace: save it once under **Harnesses**, pick it on any Run page | | The tasks | A **repo ablation**: a script run in every task's image | One repository: create it on that repository's **Repo Ablations** tab | On the Run page, each model row's **Harness** is an agent as it is or one of your saved harnesses; add the same model twice to compare the two. Pick repo ablations in the **Tasks** box: every row runs again on each one's tasks. | Harness Field | Does | | - | - | | Agent | The one agent it changes: Claude Code, Codex, Pi, … Its skills, options and subagent model are for it alone, and it runs only on that agent. | | Skills | GitHub repositories whose `skills/` folder Harbor installs for the agent: `org/repo`, `org/repo@ref`, or a URL. Claude Code, Codex, and Pi load them. | | Instructions | Added to the end of every task's instruction, or to the agent's system prompt instead (Claude Code and Codex) | | Options | Options on the agent, as Harbor takes them: Claude Code's turn limit, CLI version, and `settings.json`; Codex's web search, reasoning summary, and `config.toml`; and so on. Claude Code always runs with its permission checks bypassed. | | Subagent Model | Runs Claude Code's and Codex's subagents on another model, through the credential the harness picks. Picking one starts Instructions on a system prompt telling the agent to delegate part of each task; edit or clear it there. See [Subagent Models](https://docs.selfbench.dev/api/evaluate.md#subagent-models). | A **repo ablation** changes every approved task's files with one script, run in `/app` as the task's image is built: `rm -f AGENTS.md` removes a file, `pnpm add next@15.5.0` installs another build of a package. Each task is built and checked once, never re-authored, so its instruction, tests, and reference solution stay the same; **Build Missing Tasks** builds tasks approved since and ones whose check failed. A run in a repo ablation runs the tasks it has ready and leaves out the rest. Results count each harness and repo ablation's runs as the task's own, so each is a configuration beside the baseline, and the results filters narrow to one. Editing or deleting a saved harness changes no run that used it. Releases publish only the baseline. ## Results * Results count only **counted tasks**: a repository's accepted tasks, or a benchmark's tasks. * Each configuration's **latest scored result** on each task counts. A trial is scored when it completed, its model was verified, and it has a reward and a cost. * Costs are the provider's reported charge where it reports one, else the model's list price. ## Sharing Tasks Tasks, hidden tests, and reference solutions are private. Exports include them unencrypted; keep them private. A customer who runs tasks themselves gets a registry grant for specific tasks: `self-bench grant create` ([CLI](https://docs.selfbench.dev/cli/operator.md)). Publishing a release can make tasks public, and per-task results too. Both are off by default. Only aggregate scores are public otherwise. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.