Benchmark
First resultsZorevaBench
We ran the same 15 coding tasks with the previous Auto mode and the new one, and counted the credits each used.
The new Auto uses 66% fewer credits
Across 15 tasks, adding up the mean of each task.
Previous Auto
207
credits in total
New Auto
71
credits in total
Valid runs
100
51 previous · 49 new
Per task
Credits per task
Greeting
0.040.01Simple question
0.440.05Small edit
1.680.27SwiftUI screen
21.397.36Kotlin/Compose screen
25.3711.44Landing section
27.3511.28Bug with a failing test
15.602.66Screenshot to code
16.915.43PDF summary
2.610.19Read-only subagent
4.121.25Catalog subagent
2.861.12General: Python script
25.0910.33General: Node CLI
23.827.64General: Arduino sketch
13.014.70Agent session, 10+ turns
26.777.24
Results
Results by task
Greeting
−75%
- Previous Auto
- 0.040.04–0.04
- New Auto
- 0.010.01–0.01
- Runs
- 3 / 3
Simple question
−89%
- Previous Auto
- 0.440.43–0.46
- New Auto
- 0.050.04–0.05
- Runs
- 3 / 3
Small edit
−84%
- Previous Auto
- 1.681.52–1.98
- New Auto
- 0.270.25–0.28
- Runs
- 3 / 3
SwiftUI screen
−66%
- Previous Auto
- 21.3918.87–23.75
- New Auto
- 7.366.59–8.42
- Runs
- 3 / 3
Kotlin/Compose screen
−55%
- Previous Auto
- 25.3717.36–32.13
- New Auto
- 11.4411.17–11.75
- Runs
- 3 / 3
Landing section
−59%
- Previous Auto
- 27.3525.93–28.16
- New Auto
- 11.2811.08–11.63
- Runs
- 3 / 3
Bug with a failing test
−83%
- Previous Auto
- 15.6015.46–15.78
- New Auto
- 2.662.07–3.34
- Runs
- 3 / 3
Screenshot to code
−68%
- Previous Auto
- 16.9114.15–23.85
- New Auto
- 5.434.53–6.19
- Runs
- 4 / 4
PDF summary
−93%
- Previous Auto
- 2.612.60–2.61
- New Auto
- 0.190.18–0.19
- Runs
- 4 / 4
Read-only subagent
−70%
- Previous Auto
- 4.123.71–4.66
- New Auto
- 1.251.05–1.61
- Runs
- 4 / 4
Catalog subagent
−61%
- Previous Auto
- 2.862.76–2.92
- New Auto
- 1.120.87–1.22
- Runs
- 4 / 4
General: Python script
−59%
- Previous Auto
- 25.0923.55–28.15
- New Auto
- 10.3310.20–10.46
- Runs
- 4 / 2
General: Node CLI
−68%
- Previous Auto
- 23.8221.46–27.55
- New Auto
- 7.647.26–8.33
- Runs
- 3 / 4
General: Arduino sketch
−64%
- Previous Auto
- 13.0112.88–13.12
- New Auto
- 4.703.95–5.82
- Runs
- 4 / 3
Agent session, 10+ turns
−73%
- Previous Auto
- 26.7723.05–30.63
- New Auto
- 7.246.55–8.16
- Runs
- 3 / 3
Total (15 tasks)
−66%
- Previous Auto
- 207.06
- New Auto
- 70.97
- Runs
- 51 / 49
Credits are the mean per task; the small numbers are the min–max across runs.
What it measures
Six kinds of work
Conversation
A greeting and a short technical question. Checks that simple messages stay cheap.
Small edits
A one-file change in an existing project, like renaming a label or adjusting a style.
Native screens
Build a full screen in SwiftUI or Jetpack Compose, or recreate one from a screenshot.
Web and debugging
Add a landing-page section, and fix a bug until its failing test passes.
General code
A Python script with tests, a Node.js command-line tool and an Arduino sketch, written and run in a real folder.
Subagents and long sessions
Read-only and catalog subagents, a PDF summary, and an agent session of more than ten turns.
Methodology
How tasks are run
- 01
Same tasks, both modes
Each of the 15 tasks was run with the previous Auto and with the new Auto, using the same prompt, files and step limit.
- 02
Real files where it matters
Code tasks run in a fresh folder: files are really written and commands, tests and compilers really run.
- 03
Repeated runs
Each task has 2 to 4 valid runs per mode, and we report the mean with the min–max range.
- 04
Failed calls excluded
Runs interrupted by an expired session or a content filter don’t count. They are excluded, not counted as zero.
Metrics
What each column means
- Previous Auto
- Mean credits per task with the previous Auto mode, with the min–max range of its runs.
- New Auto
- Mean credits per task with the new Auto mode, with the min–max range of its runs.
- Saving
- How many fewer credits the new Auto used than the previous one, on the mean.
- Runs
- Valid runs per mode: previous / new.
All numbers are in credits. With 2 to 4 runs per task, small differences may not be meaningful.
Changelog
Versions
Oct 29, 2026
First run: previous Auto vs new Auto
15 tasks, 2 to 4 valid runs per mode, measured in credits. Quality scoring is coming in a future update.
Oct 28, 2026
Methodology published
Task areas, grading and metrics are defined. No results have been published.
Want to know when quality scoring is added?
Follow the changelog, or tell us which tasks you’d like to see covered.