lucid

Benchmark

The same tasks, run with and without lucid, on Claude Haiku 5.5 and Claude Opus 5.5. When the plan changed mid-build, 26 of 26 runs passed with lucid and 7 of 26 without.

Methodology#

Every run goes through headless claude -p. Everything else is identical between the two arms: the prompts, the tools, a fresh git repo per run, and project settings only (no personal plugins, hooks or CLAUDE.md). The lucid arm only adds the plugin and lucid init. A hidden acceptance test the agents never see scores the final code. The 74 runs shown cost $23.01 in total.

Scenarios#

  • Plan changes early: two workers run in parallel, and you edit PLAN.md at the first worker edit.
  • Plan changes late: the same, but the edit lands after both workers’ main edits.
  • Policy in a new session: session 1 hears an unwritten policy, and a fresh session 2 must follow it.
  • Simple task: one agent, nothing to coordinate. This measures the overhead.

Results#

ScenarioModelWith lucidWithout
Plan changes earlyHaiku 5.58/8 · $0.0314/8 · $0.030
Opus 5.55/5 · $1.011/5 · $0.91
Plan changes lateHaiku 5.58/8 · $0.0330/8 · $0.026
Opus 5.55/5 · $1.062/5 · $0.94
Policy in a new sessionHaiku 5.54/4 · $0.0244/4 · $0.022
Simple taskHaiku 5.54/4 · $0.0114/4 · $0.009
Opus 5.53/3 · $0.373/3 · $0.35

Cells show runs passed and mean cost per run.

Task success with vs without lucid. Haiku: plan changes early 8/8 vs 4/8, late 8/8 vs 0/8, policy 4/4 vs 4/4, simple 4/4 vs 4/4. Opus: early 5/5 vs 1/5, late 5/5 vs 2/5, simple 3/3 vs 3/3Task success with vs without lucid. Haiku: plan changes early 8/8 vs 4/8, late 8/8 vs 0/8, policy 4/4 vs 4/4, simple 4/4 vs 4/4. Opus: early 5/5 vs 1/5, late 5/5 vs 2/5, simple 3/3 vs 3/3

Scroll sideways to see the whole chart.

What it shows#

  • Plan changes mid-build: 26/26 with lucid, 7/26 without, across both models. Without lucid, Haiku missed every late change (0/8). Opus did better by re-checking the plan at the end (2/5), but still missed most changes. With lucid, the diff arrives before each worker’s next tool call.
  • Cheaper per working result. lucid runs cost 2–26% more, because they do the extra work the change requires. Per passing run they cost about half as much on Haiku ($0.031 vs $0.061). On Opus they cost $1.01 vs $4.57 for early changes and $1.06 vs $2.35 for late ones.
  • Plan changed by chatting with the lead: parity, 3/3 vs 3/3. In live tmux sessions, a lead without lucid relayed the change to its workers or applied it itself. lucid’s edge is for changes that don’t pass through the lead’s chat: you editing the plan, another agent, or another session. It also makes chat changes persist: when workers are active, lucid asks the lead to write the change into plan.md, and leads then did so 3/3 times (1/3 before), so the change survives /compact.
  • One agent across sessions: parity. Claude Code’s built-in memory already carries the policy forward. lucid matches it; it doesn’t beat it.
  • Overhead when there’s nothing to coordinate: +$0.002 and +5 s per run on Haiku, and +$0.026 (+7%) and +6 s on Opus. That comes from the hooks and the injected context.
  • It found a lucid bug. The first memory run failed 1 of 4 because a finished session’s file claim blocked the next session. Claims now release on SessionEnd, and the scenario was rerun.

Cost per passing run#

Total spend divided by the runs that worked. Lower is better.

Opus 5.5 cost per passing run: plan changes early $1.01 vs $4.57, late $1.06 vs $2.35, simple task $0.37 vs $0.35Opus 5.5 cost per passing run: plan changes early $1.01 vs $4.57, late $1.06 vs $2.35, simple task $0.37 vs $0.35

Scroll sideways to see the whole chart.

Haiku 5.5 cost per passing run: plan changes early $0.031 vs $0.061, late $0.033 vs no run passed, policy $0.024 vs $0.022, simple task $0.011 vs $0.009Haiku 5.5 cost per passing run: plan changes early $0.031 vs $0.061, late $0.033 vs no run passed, policy $0.024 vs $0.022, simple task $0.011 vs $0.009

Scroll sideways to see the whole chart.

Raw cost and time per run#

Mean per run, with the min–max range, for each model.

Why lucid runs cost a bit more and take a bit longer#

Measured from the run transcripts, three things explain the gap.

  1. They do the work the change requires. Most of the gap in the plan-change scenarios is work the baseline skipped: a failing run never applies the change. Compare only runs that passed:
with lucidwithout, passing runs only
Haiku, plan changes early$0.031 · 46 s$0.035 · 54 s
Opus, plan changes early$1.006 · 49 s$0.987 · 58 s
Opus, plan changes late$1.064 · 57 s$1.044 · 74 s

When the baseline gets it right, it costs about the same and takes longer, because the lead discovers the change late and the work gets redone.

  1. Bookkeeping calls. Agents run short lucid commands to claim files and log decisions. That’s about 9 per run with two workers, and about 5 even on the trivial single-agent task (10 tool calls vs 6). Each call is cheap but costs a model turn. On the simple task this is the whole overhead: +$0.002 on Haiku, +$0.026 on Opus, and +5–6 s.
  2. Injected context is small. Hooks added about 340 characters per simple run, and about 4,700 characters (around 1,200 tokens) across all agents in a two-worker run. Each run processes 0.6–1 million tokens, mostly prompt-cache reads. Hook latency is about 25 ms per tool call, roughly 1 s per run.
Opus 5.5 mean cost per run with min-max range, with vs without lucidOpus 5.5 mean cost per run with min-max range, with vs without lucid

Scroll sideways to see the whole chart.

Opus 5.5 mean wall-clock seconds per run with min-max range, with vs without lucidOpus 5.5 mean wall-clock seconds per run with min-max range, with vs without lucid

Scroll sideways to see the whole chart.

Haiku 5.5 mean cost per run with min-max range, with vs without lucidHaiku 5.5 mean cost per run with min-max range, with vs without lucid

Scroll sideways to see the whole chart.

Haiku 5.5 mean wall-clock seconds per run with min-max range, with vs without lucidHaiku 5.5 mean wall-clock seconds per run with min-max range, with vs without lucid

Scroll sideways to see the whole chart.

Reproduce it#

From a checkout of the repo, run the Haiku set (about $1.50):

In your terminal
python3 bench/run.py --runs 8 -j 4

Or the Opus set, at about $1 per run:

In your terminal
python3 bench/run.py --runs 8 -j 4 --model opus --budget 30

Then render the charts:

In your terminal
python3 bench/chart.py

Raw results are in bench/results.jsonl.

Caveats#