A coding agent can solve a request and still leave behind code that is difficult to extend. What happens when the next request starts from that code?
We introduce Multi-step SWE: 69 software engineering tasks with 574 dependent steps across nine languages. Agents must repeatedly extend, upgrade, or optimize their own implementations in a persistent workspace.
The strongest evaluated model, Claude Opus 5.5, passes 56.8% of individual steps but solves only 18.5% of complete tasks. Across all nine models, 38 of 69 tasks remain unsolved.
Paper · Leaderboard and task traces · GitHub
Step success and task completion
A task is solved only when every step receives full credit. Solve rate measures successful whole-task rollouts; steps passed measures individual steps receiving full credit; Pass@4 measures tasks solved at least once across four attempts.
| Model / harness | Solve rate [95% CI] | Steps passed | Pass@4 |
|---|---|---|---|
| Claude Opus 5.5 / Claude Code | 18.5 [11.6, 25.7] | 56.8 | 31.9 |
| Claude Fable 5.1 / Claude Code | 17.8 [9.8, 26.1] | 49.1 | 24.6 |
| Gemini 4 Argon / mini-swe-agent | 10.1 [5.8, 15.2] | 40.1 | 27.5 |
| GPT-6.1 Sol / Codex CLI | 10.1 [5.1, 15.9] | 37.7 | 21.7 |
| GPT-6 Astra / Codex CLI | 6.9 [2.9, 12.0] | 36.7 | 15.9 |
| DeepSeek V4.1 Flash / Claude Code | 6.5 [2.9, 10.9] | 35.8 | 15.9 |
| Qwen 3.8 Max / Claude Code | 4.0 [1.1, 7.6] | 24.4 | 8.7 |
| MiMo 2.6 Pro / Claude Code | 2.9 [0.4, 6.5] | 22.1 | 5.8 |
| Grok 4.7 / Grok Build | 1.4 [0.0, 4.0] | 13.9 | 2.9 |
All values are percentages; intervals bootstrap over tasks. Results evaluate model–harness combinations, using each harness's highest supported reasoning effort. Four rollouts are specified per task; the public release contains 2,483 runs, with one fewer for Qwen. Sources: paper Tables 5 and 9, released results.
Evaluating a sequence of engineering requests
Each task contains 5–15 complete engineering requests, with a 2–4 hour budget per step. A step is a developer request, not a tool call. The workspace carries forward without replacement by a reference solution. Later steps run even after a failure, allowing repairs, but an earlier failed step still prevents a fully solved rollout.
| Task family | Tasks | Example progression | Pooled solve rate |
|---|---|---|---|
| Optimization | 18 | Accelerate successive BoringSSL kernel families; preserve outputs | 15.0% |
| Feature implementation | 18 | Extend a Rust Tor client across 12 releases | 5.1% |
| Version upgrade | 20 | Implement Lua 5.1 through 5.5 in Java | 2.9% |
| Migration | 13 | Move CityVoice from Rails 4.0 to 8.1 | 13.9% |
Solve rates pool all evaluated models (paper §3.2). Version upgrades are hardest: 16 of 20 tasks are unsolved by every evaluated model.
Instructions and verifiers derive from the same specifications, release histories, or reference behavior. Grading runs in separate verifier containers, using conformance suites, differential checks, or speedups gated on correctness. Every task received two rounds of expert engineering review and is packaged in Harbor's multi-step format.
Failures often end before the budget does
Only 12% of failed steps run to within five minutes of their time limit. What agents report at the end is also poorly calibrated:
| How a failed step ends | Weighted share |
|---|---|
| Reports completion with no identified shortcoming | 52% |
| Acknowledges failing checks or missing features, then stops | 26% |
| Cites the time budget as a reason for stopping | 3% |
| Ends without a final report | 19% |
These estimates use a stratified sample of 2,157 step endings, labeled by Claude Opus 5.5 without the model identity or verifier outcome, then weighted by each model's passed/failed step counts (paper §3.3 and Appendix D). They are model-assisted annotations, not human adjudications.
Passing traces often check against independent oracles; failing traces can validate the agent's own misreading. This makes verification strategy and stopping criteria concrete research targets. Longer budgets alone remain an open question: these runs do not establish their causal effect.
What this benchmark makes possible
Multi-step SWE provides a testbed for studying persistent code state, verification policies, and completion calibration. It does not yet isolate accumulated errors from intrinsic step difficulty; a reference-initialized, isolated-step baseline is future work. Coverage also favors C/C++ (38 tasks), and memorization of public source material cannot be ruled out.
We release the tasks and evaluation code, leaderboard, and per-step trajectories to support this work.
Research by Xiangkai Zeng, Zheng Liu, and Zihao Lin, Klavis AI. See the full paper for task construction, evaluation details, and limitations.