How It Works

Pipeline

1. Provenance. A batch lists up to 500 merged PRs from non-bot authors, with their exact title and body. Common credential forms are redacted.
2. Discovery. A batch proposes PRs to meet its counts. Add PRs skips this and authors each PR you pick.
3. Authoring. An agent starts from the commit before the change. It writes the instruction, the reference patch (gold.patch), held-out tests, a Dockerfile, and smoke.sh and run-tests.sh, then calls verify.
4. Gates. Each check must pass: build, smoke (the environment works for an agent), nop (new tests fail on the base commit), oracle (the reference patch passes), and determinism (the tests pass again).
5. Review. An independent reviewer accepts, rejects, or sends the task back. Three failed rounds reject the candidate; nothing is backfilled.
6. You review. Only tasks you approve are evaluated.

Difficulty

A task takes the hardest tier its reference patch reaches, up to the tier you asked for. Hidden tests must share no files with the reference patch, and they may not import its private helpers or pin internal details. They exercise a public API, command, or persistence boundary.

What the Agent Sees

The agent gets the base snapshot, the instruction, and its environment. It never sees the held-out tests or the reference solution. Its network reaches only its model’s host. The verifier applies its patch to a clean copy, then restores the held-out test files, so tampering can only produce a tree it could have written directly.

Vibe Evals

A vibe eval starts from a description instead of a PR.

• A repository vibe eval starts from the repository at a commit you choose (a SHA, a branch, or HEAD). It is pinned to that commit and can be released with its repository.
• A workspace vibe eval lives in an eval, a collection of your own tasks with no repository behind it. The author writes the starting files, and the tests read only what the agent leaves in /app. Evals are never released.

Name the mistake the agent made and what a careful person would do. Attach real files where you can. One vibe eval is one data point, so write several.

Harnesses and Repo Ablations

A run can run every task again with something changed, beside the Baseline, the tasks and agents as they are. Two things can change: On the Run page, each model row’s Harness is an agent as it is or one of your saved harnesses; add the same model twice to compare the two. Pick repo ablations in the Tasks box: every row runs again on each one’s tasks. A repo ablation changes every approved task’s files with one script, run in /app as the task’s image is built: rm -f AGENTS.md removes a file, pnpm add next@15.5.0 installs another build of a package. Each task is built and checked once, never re-authored, so its instruction, tests, and reference solution stay the same; Build Missing Tasks builds tasks approved since and ones whose check failed. A run in a repo ablation runs the tasks it has ready and leaves out the rest. Results count each harness and repo ablation’s runs as the task’s own, so each is a configuration beside the baseline, and the results filters narrow to one. Editing or deleting a saved harness changes no run that used it. Releases publish only the baseline.

Results

• Results count only counted tasks: a repository’s accepted tasks, or a benchmark’s tasks.
• Each configuration’s latest scored result on each task counts. A trial is scored when it completed, its model was verified, and it has a reward and a cost.
• Costs are the provider’s reported charge where it reports one, else the model’s list price.

Sharing Tasks

Tasks, hidden tests, and reference solutions are private. Exports include them unencrypted; keep them private. A customer who runs tasks themselves gets a registry grant for specific tasks: self-bench grant create (CLI). Publishing a release can make tasks public, and per-task results too. Both are off by default. Only aggregate scores are public otherwise.