IBM and Kaggle Partner on Continuous ScarfBench Migration Leaderboard

IBM and Kaggle are partnering on a continuously updated public leaderboard for ScarfBench, IBM’s open benchmark for evaluating coding agents on enterprise Java framework migrations, according to an IBM Research technical note published October 27, 2026.

ScarfBench, short for Self-Contained Application Refactoring Benchmark, tests whether an agent can take an unfamiliar enterprise Java application, migrate it across Jakarta EE, Quarkus, and Spring frameworks, and leave behind software that still works. The benchmark checks that generated applications build, start, and continue to behave as expected, using both focused workloads and whole applications. IBM introduced ScarfBench earlier in 2026 as a reproducible benchmark for AI-driven migrations; the original announcement, dated March 11, 2026, described seven components, including developer-verified enterprise applications across the three frameworks, focused workloads that isolate concerns such as persistence behavior and dependency injection, whole applications combining multiple architectural layers, automated build and startup validation, and behavioral tests verifying runtime behavior and functional equivalence. Each workload was manually implemented and validated by experienced developers for functional equivalence and idiomatic framework usage, that announcement stated.

The IBM–Kaggle Collaboration

Under the partnership, Kaggle consulted with IBM on the design of the benchmark and evaluation, independently reproduced and ran ScarfBench on its own platform while sponsoring the compute, and will maintain the benchmark long-term. IBM described the teams’ strengths as complementary: it developed ScarfBench around enterprise Java modernization, while Kaggle brings independent evaluation infrastructure and ongoing leaderboard maintenance.

Results Across 204 Migration Tasks

ScarfBench evaluates an agentic system rather than a language model in isolation: each result reflects a model paired with a coding-agent harness that determines how the model interacts with the repository and development environment. IBM cautioned that the scores should be read as model/harness results rather than measurements of the underlying model alone.

Across 204 migration tasks and ten evaluated model/harness configurations, Gemini 4 Argon with the Antigravity harness ranked first on the primary smoke-test metric at 75.2%, IBM reported, followed by Claude Opus 5.5 with Claude Code at 70.5% and Claude Opus 5 with Claude Code at 69.8%. The smoke score, which serves as the leaderboard metric, awards partial credit and weights larger test suites more heavily. On the stricter all-tests-pass measure, Claude Opus 5.5 with Claude Code posted the highest rate at 72.1%, narrowly ahead of Gemini 4 Argon with Antigravity at 71.1%.

The top ordering held under a sensitivity analysis, according to the post. Four of the 204 tasks received zero behavioral-test credit from every configuration; excluding them raised absolute scores without changing the leaders: Gemini 4 Argon with Antigravity scored 79.4%, followed by Claude Opus 5.5 with Claude Code at 74.5% and Claude Opus 5 with Claude Code at 73.7%. Even the strongest combination left substantial headroom, IBM noted, with no configuration fully passing more than about three-quarters of the migrations.

Where Configurations Succeed and Fail

The widest gap in the evaluation separated code that builds from software that behaves correctly, IBM reported. After the four all-zero tasks were excluded, Claude Opus 5 with Claude Code, Gemini 4 Argon with Antigravity, and Gemini 3.8 Flash with mini-swe each compiled 100% of tasks, and eight of the ten configurations compiled at least 97%. Smoke-test scores, by contrast, spanned more than 50 points. Among applications that successfully deployed, the leading configurations clustered at roughly 63% to 76% fully passing tests, while Kimi K3 with kimi-code fell to 53%, Gemini 3.1 Pro with OpenHands to 42%, and Gemma 4 26B-A4B with mini-swe to 26%.

IBM classified 745 failing runs: 535, approximately 72%, were runtime behavioral failures in which the application built and often booted but failed behavioral validation. That group included 304 content or assertion mismatches, 210 lost-route or 404 failures, and 21 explicit HTTP 500 or 405 errors. Compile or packaging failures accounted for roughly 20% of failing runs, at 152, and boot or dependency-injection crashes for roughly 8%, at 58.

The post catalogs task-level examples. In concurrency-taskcreator-quarkus-to-jakarta, Claude Opus 5 with Claude Code scored 0/6 after the /taskinfo endpoint returned “File not found” and a test expecting HTTP 405 failed. In address-book-jakarta-to-quarkus, GPT-6 Astra with Codex scored 2/10 after list and create pages returned HTTP 404. In cargotracker-jakarta-to-spring, Gemini 3.8 Flash with mini-swe retained JSF/JoinFaces wiring in the Spring Boot target and failed during startup while creating a facesContext bean. In cart-spring-to-jakarta, GPT-6 Astra with Codex reached a deployment failure because the packaging type left by the migration was unsupported by the target Liberty runtime.

Efficiency also varied across configurations, IBM reported. Across the 200 tasks remaining after the sensitivity exclusion, median turns ranged from 32 for Claude Opus 5 with Claude Code to 79 for Gemini 3.8 Flash with mini-swe, with Claude Opus 5.5 with Claude Code at 64. Where harnesses reported cost, covering the Claude Code, Codex, mini-swe, and kimi-code setups, cost per solved task ran from $1.25 for GPT-6.1 Sol with Codex to $10.10 for Claude Opus 5.5 with Claude Code; the Antigravity, qwen-code, and OpenHands harnesses did not emit per-run cost figures.

The ScarfBench leaderboard provides the public view of these results. Public model submission is not available at launch, but IBM said it is on the roadmap.