A benchmark for agentic debugging on distributed systems

Hardest: bugs that cross nodes and depend on timing.

31.0%50.0%+19.0 ppaverage pass rate
#ModelAgent
1Claude Opus 4.6mini-SWE-agentmini-SWE-agent
51.6
64.5
+12.9$72.5$32.6−55.1%
2GPT 5.4 minimini-SWE-agentmini-SWE-agent
45.2
61.3
+16.1$7.9$6.1−23.5%
3GPT 5.4mini-SWE-agentmini-SWE-agent
41.9
67.7
+25.8$25.0$21.4−14.3%
3Claude Sonnet 4.6mini-SWE-agentmini-SWE-agent
41.9
58.1
+16.1$71.8$35.8−50.1%
4GLM 5.1 (FP8)openmini-SWE-agentmini-SWE-agent
32.3
48.4
+16.1$23.1$14.6−36.7%
4Kimi K2.5openmini-SWE-agentmini-SWE-agent
32.3
38.7
+6.5$11.2$7.5−32.6%
5GLM 5openmini-SWE-agentmini-SWE-agent
29.0
45.2
+16.1$42.3$30.3−28.5%
6Gemma 4 (31B)openmini-SWE-agentmini-SWE-agent
19.4
38.7
+19.4$4.9$4.5−8.6%
7GLM 4.7openmini-SWE-agentmini-SWE-agent
9.7
48.4
+38.7$38.5$18.7−51.5%
8GPT OSS (120B BF16)openmini-SWE-agentmini-SWE-agent
6.5
29.0
+22.6$3.7$3.3−11.5%

Overview

DDBench tests AI agents' ability to debug distributed systems.

Symptom

Raft.Shutdown() can hang permanently.

Buggy code
hashicorp/raft
Debug context
goroutine-dump.md
Agent + model
mini-SWE-agent
450 steps$530 min
Patch
+10 −4
future.go snapshot.go
Hidden oracle
reproduce.sh
BeforeFAIL
AfterPASS
Collect
We collect 60 historical bugs, confirmed and fixed upstream, from 13 open-source distributed systems.
Build
Each case pins the buggy commit in Docker, with a symptom report, debug context, and a hidden oracle that fails before the fix and passes after.
Evaluate
An agent gets the symptom and code, with or without the debug context, and passes if the oracle reports PASS on its patch.

Findings

Models that cluster on SWE-bench spread apart here.
View data
ModelSWE-bench Verified (%)DDBench Tier 1 (%)
Claude Opus 4.680.851.6
Claude Sonnet 4.679.641.9
GPT 5.477.041.9
Kimi K2.576.832.3
GLM 577.829.0
GLM 4.773.89.7

Paper Figure 1. DDBench runs are symptom-only. The GPT 5.4 SWE-bench score is from Epoch AI; the others are vendor-reported.

Context lifts weak models and cuts cost for strong ones.
View data
ModelPass rate, symptom-only (%)Pass rate, context (%)$/case, symptom-only$/case, context
Claude Opus 4.651.664.52.341.05
Claude Sonnet 4.641.958.12.321.15
GPT 5.441.967.70.810.69
GPT 5.4 mini45.261.30.250.20
GLM 5.1 (FP8)32.348.40.750.47
GLM 529.045.21.360.98
GLM 4.79.748.41.240.60
Gemma 4 (31B)19.438.70.160.15
Kimi K2.532.338.70.360.24
GPT OSS (120B BF16)6.529.00.120.11

Paper Table 1 and Figure 6a. Cost per case is total cost divided by 31.

Context helps more on harder tiers.
Gain (pp)
View data
TierSymptom-only (%)With context (%)Gain (pp)
Tier 1 (31 cases)31.050.0+19.0
Tier 2 (15 cases)48.062.7+14.7
Tier 3 (14 cases)67.977.1+9.3

Paper section 3.5.

With context, agents search less.
View data
Mean per caseSymptom-onlyWith context
Tokens1.79M0.99M
Steps57.643.1
Steps to first edit32.320.8
Read actions24.717.7
Explore actions10.96.8
Edits3.03.1

Paper Table 3: 310 paired Tier 1 runs.

Context fixes far more runs than it breaks.

Dataset

By bug family
View data
Bug familyTier 1Tier 2Tier 3Total
Protocol & recovery136019
Replication & consistency91111
Concurrency hazards45211
Input-triggered crashes001010
Resource & delivery5319

Paper Figure 5.

By language
View data
LanguageTier 1Tier 2Tier 3Total
Go169025
C++541423
Java8008
Rust0202
Erlang2002

Case registry in the DDBench repository.

By system
View data
SystemTier 1Tier 2Tier 3Total
Dragonfly (C++)111214
NATS (Go)5409
Kvrocks (C++)4318
etcd (Go)5207
hashicorp/raft (Go)4206
Consul (Go)2103
Kafka (Java)3003
ZooKeeper (Java)3003
BookKeeper (Java)2002
raft-rs (Rust)0202
Ra (Erlang)1001
RabbitMQ (Erlang)1001
RocksDB (C++)0011

Case registry in the DDBench repository.

Compared with other benchmarks

BenchmarkCross-processCode fixExecutable oracleDebug contextLanguages
SWE-bench Verifiedunit tests1
Multi-SWE-Benchunit tests8
DebugBenchunit tests3
AIOpsLabtelemetrytelemetry—
CloudOpsBenchtelemetrytelemetry—
DDBenchE2E + unit testscontrolled5

Run a case

Quick start 
git clone https://github.com/USC-NSL/DDBench && cd DDBench

# Build a case and confirm the bug
./bench.sh setup raft-358-negative-log-range
./bench.sh reproduce raft-358-negative-log-range

# Run the agent (add --context for debug context)
./bench.sh agent raft-358-negative-log-range --addon mini-swe-agent \
    --prompt "Fix the bug in SYMPTOM.md" \
    --config harness/configs/ddbench_prompt.yaml --output-dir out/

# Score a patch
./bench.sh eval raft-358-negative-log-range --patch my-fix.diff --json result.json
Full guide