分享好友 Global News首页 Global News分类 切换频道

If you're going to replace 200,000 lines of code with a set of techniques, you're going to have to b

2026-08-27 17:0090RoboInd Editorial TeamWeChat Official Account
The Heart of the Machine

From the SWE-bench Verified to the SWE Pro, and then to the DeepSWE, the models have been brushing up on the topic of fixing bugs, implementing functionalities, these bounded repair tasks, and the current large models are doing well enough to make sure that they are running safely.

But real software engineering is never just about fixing bugs.

DeepSWE has already ripped a hole in the wall, and when the question went from tweaking a few lines of code to rewriting a file-length task, the top model dropped from 96% to 70% of the pass rate.

But DeepSWE is still not the end.

The final test of Coding Agents is not a local edit, but the evolution of the entire codebase.

Removing a set of 200,000 lines of systems from C to Rust, unchanging interfaces, unchanging behaviours, unchanging old code - that's the real job on an engineer's calendar.

It's a hell of a hell of a lot harder than DeepSWE.

Navers Lab of Einsia AI recently released a new benchmark, the SWE Refactor Bench, which gained 500,000 views after the release of X and sparked more than 3,000 related discussions in overseas communities.

图片图片

If DeepSWE is looking to implement a new functionality layer in an existing code library, the SWE Refactor Bench is looking to reconstruct the layer.

20 real open-source projects, including SQLite, zlib, libsodium, GraphHopper, these industrial-grade infrastructures, have a total of 86,70,000 lines of code and 10,594 files.

The mission is divided into four categories:

The requirements for each issue are the same: to move the entire warehouse to another technical shelf, delete all the old code, and the external behaviour is not bad.

图片

Layered Setting Card: Under three locks, there is no way back

The SWE Refactor Bench's rating is not a one-run test. It sets up three locks, sequentially executed, and any locks are automatically terminated.

Following the passing of the first two blocks, came the most brutal third: "Let AI be hacked, let AI be black with code written by AI itself". Six independent Coding Agent verifiers, each with an hour to attack the submission, only executing counter-examples are calculated.

The result:

图片

All the models, the entire army was destroyed.

Eight front-end models, 26 configurations, 20 missions, and a total of 520 separate attempts resulted in only 28 submissions passing all three rounds of evaluation.

What's even more troubling is that 3 models didn't produce any acceptable submissions from start to finish, so they ran 20 missions, one of which they couldn't complete.

I've tried 520 times to break it down:

Of the 20 missions, 13 were unresolved.

图片

It's not the same thing to have two abilities: to be able to migrate and to be able to behave.

He's a thief, and he's a tough guy.

This is the most anti-intuitive design of the SWE Refactor Bench.

A traditional benchmark starts with a string of bugs in the code. A string of bugs hanging up and being fixed proves that you've done it. But the migration task starts with a system that can already run.

The paper manages what's called the blind zone hypothesis: the test set isn't enough, it can't even see if the hypothesis has been modified.

And stealing is just one of them.

There's another way in which a hard-skinned person submitted the migration 252 times, but the behavior didn't hold up, and the code was changed and ran incorrectly.

Stealing, behaving well but not doing. Strong, doing well but not doing well. Both paths do not reach the end. Eight models without exception, all swinging between these two directions.

The codes for the codes are correct, and the codes for the migration of codes are true. The most powerful models today have not yet learned to do so at the same time.

The last kilometre: 99% accuracy, one step away from delivery

For a warehouse migration, any unit test that doesn't pass is a significant risk. The failure of that test is a real downstream consumer, and it won't go wrong because the other 99.99% of the behavior is wrong.

This is one of the hardest things to do in a smart body.

Of the 340 migrations that have actually been completed:

Only this last small step eliminates 35 of the 123 runs that have gone up to 99.9%; and this last small step causes every bookmark on the site, every link shared out, to fail; and the project's homepage is blank after some of the packages are released.

It's not a test set, it's not done by the migration itself.

Written at the end

The SWE Refactor Bench reveals three things:

First, behavioural testing will fail in a migration scenario. The original system would have been able to pass all the tests, and the AI would have scored. This is not a test coverage curve, it is a measurement of the wrong object curve It is measuring whether the curve has changed, not whether the curve has changed.

A few models chose to sneak up on the roots without moving, and most models chose to crash and eventually spoil the behavior. The model can restore the behavior to more than 99% but ultimately passes less than one-tenth of all checks. The difficulty is not getting close to the behavior, but the last kilometre.

Third, AI will write code, but will not do system-level software engineering. 520 attempts, only 28 successful. 20 tasks, 13 unresolved.

It's not that AI doesn't work. It's that we finally have a measurement of whether AI does work.

The code can be rewritten.

This is the essence of software engineering.

点赞 0
举报
收藏 0
分享 2