Earlier, Anthropic showed a study of AI for Science: Claude generated 31,80,000 protein design schemes in 64 hours, screened, sent to wet laboratory tests, and ended up with 17 to meet the preset protein binding criteria, demonstrating the enormous potential for AI scientific design to move toward real experimental testing.
This reflects one of the most acute realities of AI for Science today: implementing programs in the virtual world is exactly the same as doing experiments in the real physical world.
So how deep is the gap between the model's proposals and actual physical experiments? No one has been clear in the past, because the industry lacks a set of physical verification benchmarks that can be repeatedly executed and compared horizontally. The code has unit tests, mathematics has standard answers, but real experiments are constrained by device differences, execution errors and costs, and the results of different labs are difficult to directly compare.
Recently, the AI Scientists' Last Exam (AISLE), jointly launched by several of the world's top scientific research institutions, attempted to establish a new "measure" of AI's real-world experimental capabilities. As the world's first public benchmark for dry-experimental closed-loop science discoveries, AISLE hopes to get different models into real-world laboratories and accept physical world testing under a unified standard.
Address of the project:
Github repo:
Link to the paper:https://github.com/MasterAI-EAM/ai-scientists-last-exam/blob/main/paper/AI_Scientists_Last_Exam.pdf
The result was that in the first round of accumulated 4079 real wet test tests, top models such as the GPT, Claude, Gemini and others submitted a noteworthy scoreboard: GPT 5.5 ranked first overall in the complete wet test list with a score of only 41.19 points, while in the safety-restricted test optimization task, the overall pass rate of all test models was 0%, and GPT 6 Astra scores were even lower.
The result completely broke the big model's "thinker's circle" on the list.
In a computer, the model calculates "1+1" 100 million times, and the answer is still 2, but in a real wet lab, having an automated fluid shifting device repeat 100 million operations will never be exactly the same. The equipment will wear out, the accuracy will drift, the measurement data will be noise-induced, and the experimental environment may change at any time...
It is also worth noting that enhanced scientific capabilities do not mean simultaneous improvements in security capabilities. When a model directly invokes experimental equipment and makes a decision that violates security constraints, mistakes in the digital world can turn into real-world equipment damage, personal injury or even environmental pollution. Even if the model is unable to identify the dangerous intentions that are gradually revealed in multiple rounds of interaction, its scientific advancement capabilities can be misused.
This is another dimension of AISLE: when global tech giants are competing for the technological vision of "AI scientists", the physical exam reveals a brutal reality: the limits of AI Scientist's capabilities are not only to see if it can make the right assumptions, accomplish experimental goals, but also to see if it can comply with the safety limits throughout the experiment.
It is worth noting that this industry-leading work brings together a pioneering research force in the fields of automation laboratories, computational chemistry and AI for Science. The project was initiated by Dr. Sheboygan, founder of regulatory engineering, the Acceleration Consortium team at the University of New South Wales (UNSW), the National University of Singapore (NUS), the Swiss Federal Institute of Technology (EPFL) and the University of California, Berkeley (UC Berkeley), co-led by Berkeley materials scientist Gerbrand Ceder, Philippe Schwaller, pioneer of chemical agent EPFL, pioneers in the field of chemicals and self-driving laboratories (SDL), Acceleration Consortium, a team of 13 leading global 4Science laboratories, and researchers from NVIDIA, Microsoft Research, Lilly and others.
Among them, Gerbrand Ceder's team led the construction of the autonomous materials lab A-Lab and collaborated with Google DeepMind's GNoME material discovery results. In 2023, a related study published in Nature magazine showed the potential for AI material prediction to be combined with robotic automation synthesis.
The last mile of AI for Science: From theory to computation to experimentation
From the point of view of the methodology of scientific discovery, a complete scientific research cycle typically consists of three phases: theoretical derivation, numerical computation and simulation, and the final execution of physical experiments.
In recent years, AI models such as OpenAI, Anthropic and others have demonstrated an increasingly astonishing generalization capability, making breakthroughs in mathematical reasoning, solving scientific problems, and entering scientific processes such as protein structure prediction, molecular design, and material discovery.
However, the ultimate conclusion of scientific discovery still needs to be reached in real experiments: whether the model's assumptions can be supported by real observations, whether the best computational solutions can be established in the physical world, and whether only experiments can give a final determination.
The problem is that the feedback from physical experiments is far more complex than digital tasks. An experiment requires not only equipment, tests and time, but the results can also be influenced by the measurement of noise, execution errors and the state of the equipment. Even if the experiment fails, it is difficult to determine immediately whether the problem is in the program, operation or the device itself.
This makes automation labs an important development direction for AI for Science, hoping to connect robots, experimental devices, and AI Agents so that models can not only analyze existing data, but also actively design experiments, execute operations, read results, and decide what to do next.
For example, in October this year, Google Research introduced the VERID framework, which uses semantic digits in the lab to track the state of the mechanical arm, the volume and consumption of the reagent, allowing the AI Agent to perform virtual previews before performing actual operations, and to reduce operational risks through security checks.
However, allowing AI to operate experimental devices solves only part of the problem, while AISLE focuses further on how to measure the performance of the model throughout the experiment and convert the experimental results into feedback signals that can be used continuously.
For a long time, real-world experimental feedback was not as efficient as code-testing results, used on a large scale for model evaluation and training. How to transform rare, delayed and noisy physical feedback into a reliable evaluation basis became a problem that AI for Science had to solve.
AISLE hopes to complete this cycle: make real experiments not only the final test site for AI, put the answers in the physical world, the model can back up the paper, print through the library, but can't back up the results of the next real experiment in advance.
AISLE: The last test of AI scientists, Nature as the Verifier?
AISLE's core idea is to create a distributed physical verifier that integrates model reasoning, computational simulation, and real-world experiments. It uses automated laboratories as an executive carrier to form a dry closed loop of "proposing a solution, executing an experiment, obtaining observations, adjusting decisions, and converting feedback from the physical world into signals that can be used for model evaluation and continuous improvement".
But unlike previous explorations based on a single laboratory, AISLE, through a unified mission, action and observation protocol, connects all of its flagship models to automated laboratories at different locations and devices, and the team took nine months to fully run.
In this case, the unified harness allows the results of different stations to be compared under the same criteria, while the device differences between stations are preserved to test the robustness of the model under different conditions.
AISLE is located at the top, allowing the model to experiment in person on real equipment. AISLE's dry closed loop framework: model assumptions, dry laboratories (similar to digital ones) screening candidates, distributed automated wet test site execution, representation of results feedback back to the model; complete interaction trajectory according to optimization, safety, adaptive robustness, multi-modal four-dimensional ratings.
As such, AISLE is both a common theme for comparative model experimentation capabilities and, like unit testing in the field of code, can provide replicable real feedback on model improvements.
AISLE's first round of evaluation focused on large models' dry-wet closure capabilities in real wet experiments, with models proposing solutions at the dry end, executing wet end and feedback on real observations.
In order to test the model's basic decision-making capabilities in dry experimental closed rings, the team designed optical target reverse design, pH-sensitive formula optimization, high-molecular compound irreversible operation planning, active pKa deduction of spectral measurements based on different pH conditions, spectral-image polymorphic reasoning, and other basic tasks, including cost, feedback speed, and repeatability, while maintaining true operating error and noise measurement.
Additionally, the team said they are also currently engaged in advanced scientific research tasks such as the Buchwald-Hartwig conjugate reaction, CDS-MPA quantum dot synthesis, and others.
What's even more remarkable is that AISLE introduces "Digital Rebellion" into this closed loop.
The simulation tells the machine that each experimental site has a corresponding digital nesting environment. The model can first perform a large number of iterations, trials and errors in the digital environment, and then send the selected program to a real experimental device to execute; the observational results obtained by a real experiment will in turn calibrate the digital nesting, making the simulation progressively closer to the actual behavior of the device.
This is equivalent to establishing a two-tiered feedback mechanism that works with each other: digital births provide low-cost, high-density trial-and-error opportunities; real experiments provide the ultimate basis for physical verification.
In this way, models do not have to consume real experimental and equipment resources each time they explore, but they can continuously use real observations to modify the simulation environment.
The GPT 5.5 also scored only 41 points, which is what the "first scale" measured.
Previously, no one in the industry had been able to confirm whether large models could replace human experiments, lacking a set of standards that could systematically measure AI's actual experimental capabilities, code with unit tests, and mathematics with standard answers, but wet experiments have long been in a rule vacuum.
Today, AISLE serves as this "measure" and the measurable results are not only enlightening but also revealing the chasm between the physical world and the digital world.
AISLE's first round of assessments covers two automated test sites, 4079 real-world experiments, 46626 minutes of running time, involving models such as GPT, Claude, Gemini, and the performance of four-dimensional assessment models from "exploration efficiency, experimental security, adaptive robustness, and multi-modal reasoning", which examine whether the model can approach targets within a limited budget, comply with security constraints, respond to changes in equipment, and infer scientific parameters from observational data such as spectra and images.
Distributed at AISLE v0.1 collaborative experimentation sites around the world, it synthesizes the core operating links of wet experiments from automatic fluids to glove boxes
The results showed that the highest score among the models that completed the wet experiment was GPT 5.5, but only 41.19 points. It is also worth noting that according to the safety standards adopted by the team, the test models failed to meet all the safety requirements, all at zero.
First, the scoreboard demonstrates that this "measure" is extremely accurate: it is able to quickly unravel the speculative bubble at the purely linguistic level of the large model, and clearly classify the maneuverability of each model in a real physical coordinate system. Second, it demonstrates that the actual gap between the large model and real physical experiments is much deeper than the academic and industrial community anticipates, which can be seen from the problems exposed in the experiment.
The first is the "security blind zone" of known perpetrators. In a set of security tests, the research team set up a restricted area and notified the model in advance. For example, in color matching tasks, certain RGB values are marked as unsafe zones, and the model must actively avoid touching the restricted area when looking for the optimal experimental path.
This indicates that the model understands the rules of safety, but it does not mean that it can always comply with the rules in actual actions, and that future AI Scientist's safety assessment will not only look at whether it can complete the task, but also need to further cover complex intent identification and risk prevention.
The second is the "self-deception" of non-stop calibration. Real experimental equipment will not always maintain the same accuracy. For example, models require the moving equipment to add liquid to a fixed volume each time, but as the equipment runs, the actual moving fluid volume may gradually deviate from expected. Human experimenters usually detect abnormalities, examine the equipment and calibrate. However, in the AISLE test, the model, although able to adjust follow-up operations based on the measurement results, attempts to compensate for the deviation, does not have the ability to adequately address the problem of actively identifying the equipment and proposing calibration requirements.
In addition, no so-called "six-sided fighters" emerged from the evaluation. There were no models leading the way in all four dimensions. Experimentally optimized models were not necessarily able to comply with security constraints or cope with changes in conditions; models performed better in robustness and were not necessarily good at multimodal reasoning.
This shows that experimental optimization, security codes and adaptive interference are distinctly different dimensions of intelligence, and that a single end result can never prove that an AI can independently take over a laboratory.
It's not just a test, it's a model training environment.
Overall, the first round of AISLE assessments gave the model a common measure of experimental capability, indicating that the current flagship model is far from ready to become a qualified independent "AI Scientist". But the ultimate goal of AISLE is not just to score, but to get experimental feedback into the real model training process and build a steering wheel for sustainable evolution of scientific intelligence.
As mentioned earlier, AISLE introduced digital births, and the combination of digital births and real-world experiments also provided new possibilities for model training.
Models can be explored extensively in the digital environment, obtain high-density feedback, and then validate the filtered strategies to real devices. While the models that are fully documented during the experiment can be used to calibrate the physical environment with full-fledged process-level data such as decision-making links at every step in the physical interaction, device action responses, and more.
In other words, digital births provide feedback density and real experiments provide physical real value. The combination of the two, for the first time, converts the feedback from previously sparse, delayed and expensive experiments into training signals that can be accumulated and used on a continuous scale.
It is understood that AISLE is currently in native integration with the Harbor framework, hoping to implement experimental tasks, equipment execution, real observation and evaluation training processes after accessing the model. The same set of harnesses, rated output scores, trained output actions, observation and feedback trajectories. Through native integration with the Harbor framework, these real experimental feedback has a pathway into the model after training. The model can not only come here for a test, but can be repeatedly tried, received feedback, and continue training in the same environment.
This means that real experiments in the future will not only be "final exams" after model training has been completed, but also the opportunity to become an environment in which models are repeatedly learned and continuously improved.
And behind this tight-knit loop is a clear understanding of the future development of AI for Science.
From a larger industry perspective, AI for Science is moving from assisting researchers in analyzing data and making assumptions to being involved in the entire scientific research process. The stronger the experimental performance of the model, the higher the requirements for reliability and safety.
The competition for the future AI Scientist is moving from those who understand scientific knowledge better to those who can reliably perform experiments in the real physical world, understand feedback and explore the unknown.
The team summarized the concept as "Nature as the Verifier", meaning that nature has always been the ultimate verifier of scientific conclusions.
This is the Infra layer that AI for Science lacks: making real experiments a verifier that can be repeatedly called by any model, a continuous learning training environment that can be used, and when the model is running the logic in the physical world, making models come up with solutions, calling devices, reading feedback, correcting judgments, all of which are done on this Infra layer.
Technology can be replicated, but automated experimental networks don't grow overnight. AISLE's core value is to take Nature as Verifier. Making real experiments a verification link that AI can repeatedly call, connecting automated experimental sites in different locations to a continuously running network. This network is constantly introducing new tasks, generating real feedback, accumulating experimental trajectories, and thus creating long-term accumulation, is the real barrier to this set of Infra.
It is understood that at the application level, the AISLE program is gradually expanding to more complex material, chemical and biological experimental scenarios.
Next, AISLE will continue to access more labs, missions and models. The team hopes to answer only one question: How far will AI go when nature becomes Verifier?
