Coding Agents Fail Collectively on Test Generation, while AI Code Quality Is Overestimated

Coding Agent's ability to write code is getting stronger and the ability to write tests is also improving at the same time. From testing for function-supplementing units, writing replication use cases for GitHub Issue, to building regression tests for the entire code repository, leading edge coding agents like Codex、Claude Code have been able to browse the repository, modify implementations, and continuously modify tests with feedback.

The status of testing has also changed: it is not only a guarantee of software quality, but increasingly serves as a basis for Agent self-verification, evidence for code reviews, and even rewarding signals in reinforcement learning.

The test is poorly written and directly determines whether these downstream links can trust its judgments.

So what makes a good test?

Currently, mainstream test generating benchmarks typically use single reference evaluation ("Single Reference Evaluation") where the test generated is considered successful as long as it fails on the original code and is passed on a standard answer ("Gold Solution").

The problem is that there are often multiple correct realizations of the same need, and there are many seemingly reasonable but implicitly flawed erroneous realizations.

A test may happen to "recognize" a standard answer, but leave out the code that actually has Bugs; or it may take the details of implementing a standard answer as a requirement itself, rejecting other completely correct spellings.

Recently, a team of Nanjing University professors has proposed a new test to generate Benchmark TestPrism, which includes 300 test tasks and 3,000 candidate implementations from 17 sources, each with a validated right and wrong implementation panel, and with the Joint Success Function (JSF) as the core indicator: testing must fail in the original code, accept all correct implementations, reject all errors.

The study found that of the 14 coding agent configurations, the strongest JSF configuration was only 28.00%, while it achieved a 59.67% success rate under a single reference rating.

The study further expertly labelled 286 failed tracks, classified 3 classes and 7 test error-generating mechanisms, and proposed the TestHelix framework for improving the JSF by 11.67 to 12.00 percentage points under the same reasoning budget through heterogeneous synthesis, peer-to-peer cross-validation and recurrent self-improvement.

The paper:

Research data and code:https://github.com/TestPrism/TestPrism

Test generating ratings is not a new problem.

Benchmark covers a variety of granular tasks: TestGenEval, CLOVER and MultiFileTest focus on writing tests for functions, files, or multiple file levels, while SWT Bench and TDD Bench Verified take the test generation to a real code repository to test the problem described in the GitHub Issue.

They generally follow a standard of success: Testing fails before repair, Fail passes after reference repair, or so-called Fail-to-Pass.

This standard is simple, executable, but implies a hypothesis: as long as the test can distinguish between the two states "before repair" and "after reference repair", it is sufficient to judge whether an arbitrary implementation is correct. In reality, this hypothesis is difficult to establish.

On the one hand, the test may omit key inputs or make only weak assertions that reference answers can pass and defective implementations can pass as well;

On the other hand, testing may be overly dependent on specific writing methods for reference responses, such as fixed internal function names, output formats, or intermediate states, leading to misjudgements of other implementations that meet similar needs.

Work on implementing variables such as OmniCode, SWE Mutation, and others has begun, but major investigations into whether or not to reject the wrong variable are still lacking in systematic evaluation of the "acceptance of correct alternative implementations" side.

Table 1 Generates the scope of evaluation of existing test benchmarks against the evaluation objectives (Table 1)

Therefore, the central question posed by the paper is: How robust is the evaluation of generated tests to variation across implementations?

A reliable test should, like a filter, distinguish between different implementations under the same requirements: tolerance for correct implementation and sensitivity to incorrect implementation.

In order to answer this question, the research team constructed TestPrism. Unlike providing only one reference answer, TestPrism provided a candidate implementation panel for each test task: one candidate implementation panel: one reference implementation, four correct alternate implementations, and five error implementations.

Figure 1 shows the construction and evaluation process of TestPrism (original paper Fig. 1).

The entire construction process is divided into four stages.

Step 1: Mission screening and adaptation

The research team collected tasks from 17 public benchmarks, including the widely used SWE-bench series and the Terminal-Bench series, as well as SWE PolyBench, SWE rebench, EvoEval, USACO, AlgoTune, and other scenarios covering warehouse maintenance, function specification evolution, competition programming, and numerical optimization.

The adapter then converts tasks from different sources into the Harbor task format, while maintaining the original requirements, initial code status, verifier, reference implementation, dependency, service and execution limitations.

Step Two: Choosing a Mission

The team used three indicator screening tasks: the number of valid lines of code implemented for reference (Solution Size), the number of test functions or check items (Test Count), and the number of tests for each file that was modified (Test Density).

The filtering thresholds are calibrated according to each source's own distribution and balanced by the source quota representation. The final TestPrism contains a total of 2,000 tasks, of which 1,700 are assigned as training sets for the later TestHelix strategy optimization; 300 as test sets, training sets and test sets do not overlap.

Third step: Candidate generation and verification

Before generating a candidate implementation, the team first reviews the original verifier for each task: it must fail in its initial state; it must pass on the reference implementation.

Two experts with development experience independently evaluate task needs and verifiers, check for over-adaptation or over-covering gaps in the implementation of references, discuss clarification when disagreements arise, and directly eliminate the remaining questionable tasks (Cohen's κ = 0.68).

The candidate implementation itself is actually solved by different coding agents and models with different capabilities. In contrast to traditional mutation testing, humans inject defects in such a way that they can simultaneously capture alternatives that exist and errors that occur naturally in the process of solving the problem.

After removing blank submissions, completely repetitive and unapplied patches, the team operates the original verifier in a clean environment: all checks passed the mark as correct and any check failed the mark as incorrect. These tags remain fixed during subsequent evaluations.

Step 4: Diversity Filtering and Task Conversion

If the four correct implementations were only slightly rewritten in reference to the answer, the panel would lose its meaning.

Therefore, the team must meet the average AST distance standard for the correct implementation of the group; for the incorrect implementation of the group, different sets of unpassed check items are considered as diverse, and the set must also meet the AST Jaccard distance check at the same time. Panels are generated and replaced repeatedly until both groups meet the diversity requirements at the same time.

Finally, the test-generating adapter converts each programming task into a test-writing contract: the agent needs to submit the test, Fixture, and an executable input script in the specified directory according to the original requirements.

The 300 tests in the TestPrism test set cover six languages, including Python, JavaScript/TypeScript, Java, C/C++ and Go, and come from 67 code repositories and projects with 3,000 labelled candidate implementations.

Tasks are divided into 6 major categories, 34 sub-categories: Bug repair 167, program and algorithm 53, performance optimization 28, function extension 20, project and workflow 19, and restructuring and migration 13.

Figure 2 shows the task classification of the TestPrism test set (original paper Fig.7).

On this basis, TestPrism defines a set of complementary rating indicators:

• Reference Acceptance Rate (RAR): the rate at which the test fails on the source code, the rate at which the task is passed on the reference implementation, i.e. the traditional single reference assessment;

• Valid Acceptance Rate (VAR): the average rate at which the test accepts 4 correct alternative implementations.

• Invalid Rejection Rate (IRR): the average rate at which a test rejects 5 errors;

• H: VAR and IRR adjustment averages, measuring whether the two are balanced;

• Change Coverage and Entry Coverage: measuring how much the test covers along the SWT Bench and TestExplora definitions respectively.

The core indicator Joint Success Function (JSF) requires the test to meet three conditions simultaneously: failure in the source code, accepting all correct implementations in the panel, and rejecting all errors in the panel. If the panel is reduced to reference-only implementations, JSF is degraded to RAR.

In other words, a single reference assessment is just one of the loosest examples of JSF, where JSF will never exceed RAR, and the gap between the two is the proportion of tests that can't perform more tests through reference answers.

The first question of the study is: Can the current testing generated by the Coding Agent stabilize the right and wrong implementations?

The team evaluated 14 coding agent configurations from six model families, with the main table reporting the highest JSF configuration in each family, and prioritizing the use of native coding agents for each model to reflect the close alignment between model training and the Agency framework.

Table 2 Optimal configuration of model families performed on TestPrism

Reference answers are not the same as reliability testing.

The results show that reliable test generation is still far from solved. The best-performing Claude Fable 5.1 has a success rate of 59.67% on a single reference rating, but JSF is only 28.00%; the RAR for Kimi K3, Qwen3.8 Max and GPT 5.6 sol-max ranges from 54% to 57%, while JSF is only 23% to 25%;

The JSF for the DeepSeek V4 Pro and GLM 5.3 is only 13.33% and 11.33% respectively. For all models, a significant portion of the tests were judged successful in traditional assessments, and could go wrong when faced with more implementations.

Accepting the right realization is often harder than rejecting the wrong.

Further subdividing, five of the six configurations have IRRs higher than VAR, indicating that the more common mistakes made by current Agents in tests are too serious: the specific writing of reference implementations is considered a requirement, and other legitimate implementations are wrongly harmed.

GLM5.3 is a typical example: its IRR is 86.33%, the highest of all configurations, but its VAR is only 50.33%, and its JSF is the lowest.

Qwen 3.8 Max, on the contrary, has a VAR of 77.00%, with an IRR of only 66.00%. This indicates that it is not enough to reject errors, reliable testing must be both tolerant and sensitive. In all 14 configurations, the VAR and IRR tuning and average H are equally correlated with JSF.

Coverage is important, but not all.

Coverage rates are similarly highly correlated with JSF: in all 14 configurations, the Change Coverage and Entry Coverage and Pearson coefficients of JSF are 0.91 and 0.96, respectively. But covering up to the corresponding code, just because the test correctly judges its premise, does not mean that its assertion is able to correctly judge the behavior of this logic.

The core findings of RQ1 are that there is still a large gap in the generation of reliable tests from front-end model distances, and acceptance of correct implementation and rejection of incorrect implementation reveal complementary weaknesses, both of which have a combined impact on balance and coverage.

The second question of the study concerns the candidate panel itself: how will the success rate of the test change if more and more implementations are gradually added to the panel?

The team randomly drew 50 tasks along the same building process, expanding the tables for each task from 10 implementations to 15: 1 reference implementation, 7 additional correct implementations and 7 additional error implementations.

On the condition that the test remains unchanged, starting from a reference-only implementation panel, the implementation is progressively added in three ways: correct implementation only, incorrect implementation only, and alternate addition of the two.

Figure 3 shows changes in the diversity of JSF and candidates under the three implementation additions (Fig. 3).

The results showed that the average JSF continued to decline from 49.33% when it was only a reference implementation: it dropped to 38.10% after adding 5 correct implementations, to 30.49 after adding 5 error implementations, and to 24.33% after switching to 9 implementations; when all implementations were all added, the endpoints of the three modes were 37.33%, 29.33% and 23.33% respectively.

The test itself has not changed at all, the change is only that the panel has asked for more correct implementation of the test test whether it is too strict, more error implementation of the test test whether it is too loose.

This shows that the JSF is not a static score, but rather a dynamic reliability measure of changes with the panel: which implementations contain the same importance as which implementations contain the same number of implementations. A single reference assessment as a specific instance of only one implementation is the weakest test and also extends the boundary of panel success rates. The core finding of RQ2 is that multiple candidate implementations can expose small panel detectable test errors.

The first two studies revealed the extent to which testing was "unreliable", but did not answer why they were unreliable.

To this end, the research team extracted 50 JSF failure tracks from the best configurations of six model families, each of which was zero, excluding 14 super-time tracks, and independently labelled all 286 tracks by two experts (Cohen's κ = 0.81) to introduce a third expert when there was a disagreement.

When labeling, experts examine task requirements, submitted tests, execution results on source code and reference implementations, results on candidate implementations, and proof of corresponding patches.

Figure 4 indicator images of each model (a) distribution of error mechanisms in the 286 failed trajectories (b) (Fig.4)

Coverage gaps: Features detected but missing key inputs

Coverage gaps account for 34.6% of all failures, including missing inputs, 30.8%, and untested requirements, 3.8%. The latter is one of the most common single errors: testing performs an operation without covering an input portfolio that exposes defects.

For example, in a poker card-type judgement task, the resulting test never compares the size of the subset A as the smallest card to the size of the three, so one successfully passes the test by misjudging this subset as a "high card".

Wrong check: executed but judged wrong

Faulty Checks was the largest category, reaching 50.0%, including Weak Checks, 7.7%, Criteria Error, 26.9%, and Overfitting, 15.4%.

This category indicates that the test may not be able to correctly judge it even if the relevant behavior is performed. The over-matching response is precisely the "over-serious" problem observed in RQ1: the test takes the accidental details of the reference realization as a requirement itself.

For example, in a parenthesis parsing task, the test does construct inputs containing non-matching parentheses, but only checks the number of output results, without checking the specific content, leading to an error implementation that discarded the legitimate grouping being accepted.

It's worth noting that the tests in both poker cards and bracket analysis failed on the original code; they were all judged successful in a single reference evaluation of the reference implementation by clicking on the reference implementation. These missed input combinations and overly weak statements were only exposed after additional errors were added to the implementation.

Logical Mistakes: Testing Yourself to Run

Execution Logic Errors accounted for 15.4%, including Fixture and API usage errors (7.7%) and command execution and output parsing errors (7.7%).

For example, in an EvoEval task, a test is stored by a test function in a test class and then called it in by self, resulting in an accidental binding of an additional parameter, which causes the test to fail even in the reference implementation. This type of problem directly reduces the overall reliability of the test, but unlike the previous two types, they are also found in a single reference evaluation and do not constitute a single reference evaluation "blind zone".

The core findings of RQ3 are that test failures are caused by missing relevant behaviour, misjudging the correctness of achievement, or testing the failure of execution itself; in which missing input combinations and too weakly stated assumptions can only be exposed in the face of further error achievement through a test of reference responses.

To address the above problem, the research team proposed TestHelix. Its starting point is straightforward: since only one implementation fails to detect the test problem, multiple implementations are actively constructed during the generation phase, allowing the test to accept multiple tests.

This is the first in a series of posts on the project.

Synthesis: Allows different Agents to provide a set of tests and corrections

TestHelix assigns different exploration strategies to several isolated coding agents, each of which writes a test T and a fix P independently based on the requirements of the public task only. Each pair of tests and repairs must meet internal consistency: the tests must fail reproducibly in the original code and pass on their own repairs.

Peer-to-peer cross-validation: Testing yourself with other people's repair

Passing only on its own repair could still be a self-indulgent mistake. Therefore, TestHelix cross-executed each test on the repair of other Agents to expose overly rigorous testing or incomplete repair.

At the same time, an independent review agency that sees no repair code will review the assertion step by step based on public demand that every blocking opinion must have a public demand as a basis.

Evaluation-confirmed countercases are used for proofreading and proofreading; when several different qualifying proofreading can pass a validated test, the system also draws on the idea of differential testing to actively look for inputs that can distinguish them. Finally, the compatible tests are combined and validated again in a whole new environment.

Returning to self-improvement: Learning better test generation strategies from training tasks

TestHelix evolved from a basic strategy with the help of AutoSaddler: selecting a parent strategy, diagnosing its execution trajectory, proposing revisions, then comparing the performance of the parent strategy on the same task, and deploying replicable experience for subsequent rounds.

Optimization is carried out on 600 tasks in the TestPrism training center, completely isolated from the test set, and never in contact with hidden candidate panels. It is worth noting that the goal of optimization is to use more signal-intensive RARs rather than more stringent JSFs, as the latter are often difficult to reward for progressive strategy improvements; the requirements for multi-distinctive capabilities are handed over to heterogeneous synthesis and peer-to-peer cross-validation.

After the optimised strategy was frozen, it was directly applied to both the GPT 5.6 sol and the Kimi K3 models.

The team compared five TestHelix configurations on GPT 5.6 sol and Kimi K3 and used their respective native coding agents (Codex and Kimi Code) as a contrast. All configurations within the same model group use the same basic models and tools and share the same single-task logic budget caps.

Table 3 TestHelix dissolution experiment compared to native coding agent Table 3

The results showed that only three sets of tests and corrections were generated, followed by a random submission of one, which yielded a maximum of one percentage point of improvement over the single set; complete peer-to-peer cross-validation was enabled on the basis of the same three sets, while JSF increased by more than six percentage points, GPT5.6 sol increased from 21.00% to 27.67%, and Kimi K3 increased from 22.67% to 29.00%.

The final configuration on the two models reached 32.33% and 33.33%, respectively, for JSF, improving by 11.67 to 12.00 percentage points compared to single-cluster generation, and by 8.67 to 9.00 percentage points higher than their native Coding Agent. Under a fixed reasoning budget, the benefits of improving "how to generate and test testing" were significantly higher than simply allocating the budget to more Agents.

The core findings of RQ4 are that isosynthesis helps test cross-validation between different implementations using multiple repair checks and correction tests, and recursive self-improvement is a strategy to generate sedimentary testing between tasks, both of which contribute to clear benefits in dissolution experiments.

As coding agents increasingly rely on testing to validate their work, the test quality rating standards need to be improved as well.

The study shows that a single reference evaluation systematically overestimates the quality of the test: a test that can distinguish between two states before and after repair does not mean that it can correctly judge other implementations under the same need. Even the strongest front-end model has a combined success rate of less than three percent in the face of multiple right and wrong implementations.

At the same time, the research reveals two aspects of the problem. Testing can be either lax, omitting key inputs or only performing surface checks, leaving out defective implementation; or it can be too rigorous, taking the accidental details of the reference implementation as a requirement, thereby misjudging the legitimate implementation.

The former is easily hidden in a single reference assessment, while the latter is not found in a single reference assessment at all. The true reliability of the test can only be seen when the assessment simultaneously considers the two directions of acceptance and rejection.

From an evaluation perspective, TestPrism offers a more practical test method: using a set of verified, diverse implementations to test tests rather than relying on a single standard answer.

From a methodological perspective, TestHelix demonstrates that improving test reliability does not necessarily require a stronger underlying model, allowing multiple Agents to provide counterexamples to each other, test each other's conclusions, and use effective experience as a strategy, which can also bring significant benefits.

Of course, the paper also notes that TestPrism contains only 10 candidate implementations for each task, and that ratios such as right and wrong implementations do not represent real distributions, nor can AST-based diversity measurements guarantee full coverage at the semantic level. The extension of assessment to evolving needs, broader implementation space and more execution environments, are still directions worth exploring in the future.