分享好友 Global News首页 Global News分类 切换频道

CMU Researchers Test AI Agents on End-to-End Infrastructure Engineering Tasks

2026-11-16 11:5510nvidiaWeChat Official Account
关联企业:NVIDIA Corporation;关联品牌:英伟达、NVIDIA Omniverse

Carnegie Mellon University researchers introduced Singularity-Bench amid unrelated online speculation about NVIDIA CEO Jensen Huang’s future. The paper presents the benchmark as a way to test whether coding agents can build core AI infrastructure. It reports that Opus 5.5 running with Claude Code met or exceeded human-engineered baselines across six tasks and scored 1.25 on the Singularity Index. The authors discuss the result in relation to recursive self-improvement; it is a benchmark result, not evidence that a technological singularity has occurred.

Using Opus 5.5 with Claude Code, the researchers built components including a pretraining pipeline, an inference engine and a cluster orchestrator. The paper compares these systems with established tools, including SGLang and NVIDIA Dynamo, on the same GPU hardware.

Among 11 agents built from seven frontier models and two agent frameworks, Opus 5.5 with Claude Code matched or exceeded the human-engineered baseline on all six tasks. Its Singularity Index was 1.25.

What Singularity-Bench measures

Singularity-Bench focuses on a part of AI development that conventional capability benchmarks often leave out: the infrastructure used to train, serve and coordinate models.

Humanity’s Last Exam (HLE), for example, evaluates reasoning across a broad range of academic subjects.

Like other knowledge benchmarks, HLE evaluates reasoning across challenging academic subjects.

Singularity-Bench instead evaluates the underlying infrastructure of AI systems.

Singularity-Bench evaluates six end-to-end tasks, from model pretraining to high-concurrency inference systems.

The six tasks cover pretraining, midtraining, post-training, inference, agent-framework development and cluster orchestration. Each task requires the agent to implement the core system from a minimalist starting point and evaluate it against a mature human engineering baseline on the same hardware.

In reality, these six tasks usually require a team of more than a dozen experts, including top system engineers, CUDA optimization experts and distributed computing tycoons, and it takes several months or even half a year to complete them.

This requires not only understanding of high-level architecture, but also extremely hard-core low-level knowledge such as memory management, operator fusion, and hardware communication.

In the past, we believed that this kind of work could only be done by human geniuses with an annual salary of one million US dollars, such as those "10x engineers" who are fiercely competed by Elon Musk and OpenAI.

But CMU threw Opus 5.5 and Claude Code into this exam room.

In order to ensure fairness, all test results are strictly back-to-back compared with the strongest human production systems (such as the widely used SGLang or NVIDIA's own tools) on the same batch of GPUs.

As a result, Opus 5.5 "swept the army".

The other 10 AI systems in the comparison scored below the human baseline in the face of this hellishly difficult bottom-level task, with an average score of only 0.58. GPT-6.1 Sol was only one breath away, with a score of 0.969.

However, the final score for Opus 5.5 is: 1.25.

In the Singularity-Index scoring system, the “human baseline” is set to 1.0.

In other words, if you can achieve the level of the current top human open source tools or commercial tools, you can get 1 point; Opus scored 1.25!

Not only did it get full marks, it even directly penetrated the human ceiling. The code it wrote was even better than the top human projects.

Earn an extra $3.34 per hour, behind a tens-billion-dollar computing power revolution

The most nerve-wracking thing about this paper is the "economic account" it gives.

In a real commercial environment, the research team tested the monetization capabilities of these inference engines, which is the effective throughput they can process per GPU hour, converted into dollar revenue.

As an excellent representative of the human camp, SGLang, the current popular reasoning framework, performs stably.

Under certain test loads, it earned $8.18 per GPU hour on simulated charges. This is also the current mainstream productivity level in the industry.

So, what about the service engine "hand-made" by Opus 5.5?

The answer is: $11.52.

On the face of it, 11.52 minus 8.18 is only $3.34 more per hour per card. Doesn't seem worth mentioning?

But if you really understand the AI ​​computing power ledger, you may break out in a cold sweat when you see this number.

Because in the world of super computing power, everything is based on scale multipliers.

Let's do a simple arithmetic problem.

Suppose a technology giant has a very large intelligent computing center with 100,000 GPUs.

If you use SGLang or a similar system written by human engineers, the daily income of this cluster is: 100,000 × 24 hours × $8.18 = $19.632 million.

If the underlying system is replaced with an engine written in Opus 5.5, the revenue per day will soar to: 100,000 × 24 hours × $11.52 = $27.648 million.

That’s $8 million more per day, $2.9 billion more per year!

This means that you don’t need to ask TSMC to increase production capacity for you, and don’t rush to buy a new generation of graphics cards. You just use an extra super intelligence, and then the computing cluster has nearly 40% more throughput capacity out of thin air.

This 40% performance difference is a substantial performance gap.

In addition, in the other five test tasks, Opus 5 surpassed humans in pre-training, mid-term training, agent running framework and cluster orchestration. Only in post-training, the human baseline was almost tied, only 0.6% less than the latter.

For example, in the cluster arrangement of characters, tools such as NVIDIA's Dynamo were reduced to the background and were crushed by Opus 5.5.

Opus 5.5 proves one thing: machines understand machines better than humans do.

When human engineers write code, they may be limited by cognition, code readability, and architectural inertia; AI can directly face the physical limits of silicon-based hardware and efficiently arrange and combine them.

What the results show

The authors report that Opus 5.5 with Claude Code matched or exceeded the human-engineered baseline across the benchmark’s six tasks, with a Singularity Index of 1.25. These results describe performance on the benchmark and do not establish that the system can recursively improve itself in unrestricted settings.