Evolvarium GitHub

Open-source harness optimization for Claude Code, Codex and Pi

Harnesses evolve. You watch every edit.

A Meta Agent edits the harness around your coding agent: its instructions, skills, hooks and extensions. Each edit runs on real tasks, an independent grader scores the output, and a rule frozen before the first edit keeps it or reverts it.

Quickstart View on GitHub

Data: study/lineages.json, runs.csv, heldout.json · lineage main / pi-f2-loop-l1

Open it while it runs.

evolvarium optimize loop.json --dashboard 8765 serves a page on your machine that updates as runs land: each candidate on the score chart, its hypothesis, the keep rule's numbers against its bar, the pages it built and the Meta Agent's notes.

Illustration: values from Study pi-f2-loop-l1-s2, times invented. Open its dashboard

Four steps, four owners. None can change another's rules.

  1. Edit

    The Meta Agent, Claude Code

    Writes a candidate harness and the hypothesis behind it, from the feedback the Study releases.

    Touches only its iteration folder.

  2. Run

    The coding agent: Claude Code, Codex or Pi

    Runs the search and dev tasks with the candidate, one fresh session per run.

    Touches only a fresh, sandboxed workspace.

  3. Grade

    The grader

    Scores every output. The agent never sees its checks and cannot repair the run it judges.

    Touches only the output, after the run has ended.

  4. Decide

    The keep rule

    Keeps the candidate only if its gain over its parent clears a bar set from measured noise.

    Frozen before the first edit.

Every run is recorded: its inputs, the harness revision, the agent's trace and the grade. A claimed gain can be traced back to the runs behind it.

How a page is graded. And how we checked the grader.

The study's tasks ask for open-ended web pages, so the grader opens each page in a real browser and scores it from 0 to 1 in three parts, weighted as below.

  1. Checks 30%

    Programmatic checks in Chromium: the task's own checks, plus common ones such as contrast and whether controls respond.

  2. Brief 30%

    Model judges test each condition the brief sets; one of them uses the page through a browser. A condition counts when most judges agree it holds.

  3. Design 40%

    Model judges score a design and motion rubric. The strict grader's rubric is tiered: concept, craft, motion.

A hard fault multiplies the score by 0.85: a false claim in the content or a control that does nothing, and in the strict grader also a content defect. Every judge is deepseek-flash in a sandbox with read-only tools; one grade takes about six minutes and 3.5 million tokens.

Checking the grader with blind labels

A score is only useful if it ranks pages the way their readers would. We compare the grader with blind pairwise labels: a labeler sees two pages built from the same brief, without knowing which harness built them, and says which one they would rather receive and why. So far an AI labeler following the written protocol has done this, with no human labels yet. It agreed with an earlier grader on 6 of 6 pairs that differed clearly, and on 16 of 32 that differed slightly, a coin flip. The tool and its guide are public, so anyone can add labels.

Try the labeling tool Read the labeling guide Both graders' source

The grader is the optimizer.

−33% to −70%

of each harness's measured gain, once a stricter grader re-scored the same held-out pages. We ran the loop on open-ended web-page tasks graded by model judges; the pages did not change between the two grades.

Data: study/heldout.json

Shows
The same pages of the same harnesses, two graders, two different gains. How much a harness improved is partly a fact about the grader.
Does not show
Which grader is right. On page pairs that differed only slightly, an earlier grader agreed with a protocol-guided AI labeler 16 times in 32, a coin flip; there is no human agreement data.

What the optimizer learned was often the rubric.

8 of 9 harness versions kept after the Meta Agent saw failed rubric items had copied rubric text into the harness; none of 11 did when it saw only outcomes. Here is one, kept: a design item of the strict grader, and the rule the next harness added to its own skill.

Data: both texts are read from the released files when the site is built; Markdown emphasis removed.

Shows
Feedback about failed rubric items reaches the harness word for word, and the keep rule rewards it.
Does not show
That copying makes pages worse. The copied rule may be good advice; what it measures is the grader's own checklist.

Read the study: The Grader Is the Optimizer

Twelve lineages, every revision, released.

The repository releases the study's two graders, its eight web tasks, every harness revision with its decision, and the per-run scores. Each tree is one lineage, plotted like the hero; each opens its dashboard.

Data: study/lineages.json, runs.csv · one score scale for all twelve

Run it on your own models.

Confined runs need macOS 26, uv and Node 22.19 or later. The web example also needs Claude Code for the Meta Agent and an OpenAI- and an Anthropic-compatible API; DeepSeek's serves both. Its README has the private launch files.

evolvarium — zsh 
git clone https://github.com/Yaxin9Luo/evolvarium && cd evolvarium
uv sync --locked
npm --prefix workers/pi ci --ignore-scripts && npm --prefix workers/pi run build

# A model-free batch: three harnesses, two tasks, two scripted backends, in seconds
uv run python -m examples.transfer_matrix.run --root /absolute/new/directory

# The study's experiment on your own models, with the live dashboard
uv run python -m examples.web_optimization.prepare --root /abs/runs/web-1 --grader strict \
  --launch /abs/private/pi-launch.json --provider deepseek --model deepseek-flash \
  --judges /abs/private/judges.json --claude "$(command -v claude)" \
  --meta-model deepseek-flash --meta-launch /abs/private/meta-launch.json
uv run evolvarium optimize /abs/runs/web-1/loop.json --dashboard 8765