The first exam was held at the Embassy in Bangalore, India.It is a comprehensive framework for assessing the ability of a programming intelligence body to complete robotic tasks. It is large and diverse in size, covers 1,156 tasks, 35 task suites, and 47 types of robots, and is easy to operate and evaluate: a single command can run any programming intelligence body, and each task can be scored on a maximum of three assessment settings.
The main page of the project:
embodied-first-exam.ai
The code is:
github.com/mll-lab-nu/embodied-first-exam
Trajectory data:
huggingface.co/datasets/MLL-Lab/embodied-exam-data
Summary
The early results of Agentic Robotics suggested that the use of programmed intelligent body control robots could lead to paradigm shifts. Existing robotic benchmarks are built around the learning of visual motion strategies, so whether a programmed intelligent body can reliably control different robots under the constraints of the physical world remains an open question.
This is why we propose:Embodied First Exam（EFE）A unified, comprehensive, agent-originated evaluation framework. In this framework, programming intelligence can choose to control itself: direct control, code-writing control, or training strategies for control.
We want it to be the first test before a programming intelligence enters the physical world. The benchmark covers 47 robots, 35 task suites, and 1,156 tasks, including nine new suites, with tasks beyond capture and placement, space puzzles, agility, multi-bot interaction, dynamic full-body control, and Sim-to-Sim engineering.
Each task can be run in a maximum of three evaluation settings.Standard SettingSimulating the real world: Smart entities can only receive sensor observations, and by default cannot reset the environment.RealtimeVariations will keep the environment running continuously when intelligent objects reason.Privileged SettingIt allows the intelligent body to access the full state of the simulator and to reset it indefinitely, thus finding Oracle solutions for training purposes. The success rate under this setting can serve as an approximate upper bound.
Of the 456 core missions, the best performing of the eight was the GPT-6 Astra in the Codex, which has a 34% success rate under the Standard Setting, and there is still a lot of room for improvement. To understand where the intelligence failed, we marked the failed attempts at the wrong stages and compared the failures in three dimensions from the intelligence, evaluation settings and stage categories.
1. Quotations
The way robots acquire new skills is either through artificial programming or learning from data, such as recent VLAs and world action models.
Programming intelligences powered by LLM offer a third way. They are already able to accomplish challenging software engineering tasks in the digital world. These types of intelligences are expected to be applied to robotics engineering because they are able to convert instructions into executable code and program adjustments through automated experiments. Recent early work by Agentic Robotics suggests that this could lead to a paradigm shift: Intelligences directly control robots, write robotic programs, or train strategies.
This enables our systems to assess the limits of the ability of programming intelligences:
What are the tasks of robots that GMOs can solve, what constraints of the physical world do they need to abide by, and where will they fail?
The existing benchmarks do not fully answer this question. We cannot determine the limits of a robot's capability with only a limited set of robotic tasks, or with automated programming tasks that do not impose physical world constraints. To answer this question, we need to clearly assess which robot capabilities should be evaluated, measure how physical world constraints affect success rates, and support different models and Harness through task interfaces.
In order to answer this question, we ask:Embodied First Exam（EFE）A unified evaluation framework connecting programmed intelligent agents to simulated robots, as shown in Figure 1. The framework follows three principles.
The first is to incorporate diverse tasks beyond capture and placement.Space puzzle testing the ability to reason about space constraints; Sim-to-Sim engineering testing how an intelligent body performs feedback adjustment controllers under strange dynamic conditions; dynamic task testing the ability to respond to changing environments; multi-bot interaction and full-body control test coordination capabilities;
Thus, EFE comprises 47 robots, 35 task suites and 1,156 tasks. It integrates 26 suites adapted from existing robot benchmarks, as well as 9 new suites covering the aforementioned capabilities and security.
Second, it measures how success rates are affected by the constraints of the physical world.The task should follow the constraints of real robots as much as possible. Success is achieved in the condition of being able to access the full simulation state and reset the environment, and does not indicate what the intelligent body can do when it can only receive partial observations, cannot reset the environment, or the dynamic environment continues to operate.
Therefore, we have defined a maximum of three assessment settings for each task:
Standard Setting：Following real-world robot conditions. Smart objects can only observe the environment through robot sensors, and cannot default to reset the environment. When intelligent objects reason, simulation will wait.
Realtime Setting：Standard Setting variations. The environment continues to run continuously when intelligent agents make decisions.
Privileged Setting：It provides an approximate upper bound and can also be used to generate training data. Intelligent entities can access the full simulation state and reset it indefinitely, thus searching for Oracle solutions by trial and error.
We're going to call each mission kit setup kit a kit, and we're going to call it a kit.TestingDifferent tests of the same task share initial scenarios and rules for determining success, allowing us to compare the success rates of state access permissions, reset restrictions, and changes in time conditions.
Third, providing a unified infrastructure for different models, harnesses and controls.With just one command, EFE can run different models and Harness combinations on multiple robotic tasks. Given a natural language task instruction, the intelligent body chooses its own control method, as shown in Figure 2:
Direct control:After receiving observations, the intelligent organism produces a direct output.
Code to control:An intelligent organism produces a planner, controller or other program that generates the actions of a robot by the program.
Train a policy:Intelligent entities train strategies in simulations and use these strategies to control robots.
The framework will enforce the access permissions allowed by the assessment settings and score each attempt with a validator that cannot be controlled by an intelligent organism.
Figure 2 shows examples of controls.Each way shows an extract of the command or code of the smart body. The black label marks the way in which the model represents direct control; the code in the box acts as a strategy.
We evaluated eight programming intelligences in a core task set of 456 difficult tasks. The strongest performing intelligences were the GPT-6 Astra in the Codex: on the same 391 tasks, it solved 77% of Privileged tests, but only 30% of Standard tests.
In addition to the three most powerful GPT-6 Astra, Claude Opus 5.5 and Gemini 4 Argon, Kimi 3.1 Pro performed best. It solved the Standard test for 9% of the same 391 tasks, i.e. 34 tests, as shown in Figure 11; with a success rate of 7% on the Realtime test, as shown in Figure 8.
Realtime Setting is even more difficult: 63% of the tests in this setting are not solved by any intelligent body; compared to 50% of the Standard tests for the same 54 tasks.
To determine where the failure occurs within the mission, we divide each task into several stages. Each stage corresponds to a change in the environmental state required to complete the task. For example, the task of putting tomatoes in a closed fridge contain three stages: the fridge is opened, the tomatoes are placed in the fridge, and the fridge is closed again.
For each of the eight failed attempts on 427 missions, we used another independent programming intelligence to mark the failure stage. The main findings are:
Privileged compared to Standard, section 3.2.For the GPT-6 Astra, the help of resetting environments is greater than the help of obtaining the precise state of objects. When both are present, most of the gap between success rates and Privileged Setting is eliminated. Under Standard Setting, the stages where precise operation is required become much more difficult. At these stages, resetting environments helps, but only providing precise states does not help.
Where the smart body fails, section 3.3.Failure in the same task often occurs at the same stage, usually the first stage of the task. This stage is often the same in both Privileged and Standard settings. Failure rates are highest when changing the state of the material or the stage of the mobile robot body.
The difference between the different intelligences, section 3.4.Compared to stronger intelligent agents, weaker intelligent agents are more likely to fail before the required contact is established. Under Standard Setting, the three strongest intelligent agents solve different test sets of parts. Switching Harness can change the success rate of the same model by up to 12 percentage points.
How intelligent agents control robots, section 3.5.In the Standard and Realtime settings of a soccer mission, GPUs and training libraries receive intelligent perception training strategies, while only trained strategies win the game. Under the Standard Setting, the intelligent body controls most of robots directly; in the Privileged and Realtime settings, the intelligent body mostly writes programs to control robots.
Figure 3 notes.(a) Our unified framework supports different models, Harness, controls and simulators. (b) In the Privileged setting, the smartphone shares a container with the simulator, and the verifier resets submitted responses in an independent environment. In the Standard and Realtime setting, the simulator is separated from the verifier and the smartphone and scores for the episodes that are executed in real time.
2. Embodied First Exam: A unified framework for the evaluation of programming intelligence in the field of robotics
In this section, we discuss how to evaluate programming intelligence in robotic tasks (section 2.1); resource constraints for each task (section 2.2); and tasks in an exam (section 2.3):
2.1 Framework and evaluation agreements
The existing robot benchmark focuses on learning visual motion strategies with relatively fixed input-output interfaces, whereas the programming intelligence we evaluate can choose how to control itself.
Therefore, our design principle is to give the intelligent body full freedom to act and restrict the outcome. We fix its initial state, available resources, and how to successfully determine them. For this reason, we package each robot task in the Harbor format. Harbor is the evaluation framework of Terminal-Bench, which runs tasks in containers, as shown in Figure 3a.
Smart bodies have their own workspaces in which they can install software packages, write and run programs, and train models. Each task starts at a fixed initial state; verifiers outside the smart body container report whether the answers submitted by the smart body meet the success rules of the task.
On top of this framework, we define a unified evaluation protocol, with a maximum of three settings for each task, as shown in Figure 3b. We design Privileged as a near-upper boundary of performance, allowing intelligent agents full access to simulators for trial and error; Standard and Realtime make evaluation as close as possible to real robot conditions.
This upper bound is aimed at a set of tasks, not at each individual task, because the intelligent body may solve a task in the Standard setting but fails in the Privileged setting, see Appendix D.
For a given task, all settings share the same initial scenarios, success rules, and time constraints, so their results are paired. The difference between these settings is: what the intelligent body can observe, whether it can reset the environment, whether the environment will wait, and where the simulator and verifier operate; they never limit the way the intelligent body can choose to control it.
Privileged
In the Privileged setting, the smartphone and the simulator run in the same container, so it can read the entire simulation state and reset the environment without restrictions.
Intelligent entities must submit an underlying robot trajectory; for some tasks, controller procedures are required. The verifier re-places the trajectory in a separate container, or operates the controller to determine whether the task is successful.
Standard
In the Standard setting, we use the simulator as a service, running in a standalone container. Smart entities have limited access to the simulator through a predefined API: it can query the entity's state, such as joint position, speed, and momentum; it obtains sensor data, such as labeled RGB-D images; and it sends motion blocks to control the robot.
The simulation between API requests will be suspended, so no real-time pressures will be put on the smart body. Unless the task explicitly provides a reset budget, reset environments are not allowed.
When a robot is controlled by an intelligent organism, the verifier evaluates the progress of the task online. In most task suites, the interface successfully reports the task to the intelligent organism and ends the episode.
Realtime
Realtime is similar to Standard, but with a few exceptions: simulation does not pause between intelligent body actions, but keeps running in real time. For tasks with higher computing costs, simulation runs at a fixed rate of real-time speed, for example.
Due to computational resource constraints, we currently only evaluate Realtime settings on dynamic tasks.
Figure 4 - Resource constraints in the exams.Larger numbers and scales represent the median of the tests; for GPUs, the median is only for the test calculations reserved for the GPU. The line segment represents the numerical range, and the dark box represents the middle 50% of the test.
2.2 Restrictions on the resources of programming intelligence
As shown in Figure 4, we set resource constraints for each task: set time limits and token budgets for the smartphone; set reset budgets for the robot, and step limitations for certain tasks, such as the RoboDojo itself in Figure 4, or the episode length limits described in the instructions; set CPU, memory, GPU, and disk limits.
The token budget is high enough that no one attempts to reach this limit once. The same task has the same time limit in all settings; when the time runs out, we stop the smartphone and rate its results at the time.
In the Privileged setting, there is no limit to the number of reset times. In the Standard and Realtime settings, the environment cannot be reset unless the task provides the reset budget.
We set computational resource limits based on the needs measured for each task and monitor resource use throughout the run, see Appendix C.
We observed that, under different resource constraints and evaluation settings, programming intelligences may choose different control modes, as shown in Figures 2 and 3.5. For example, if we provide a GPU, Intelligences can train a strategy to control robots, as shown in Figure 19.
A total of 411 assignments were created for the 9 sets of exams:
A total of 745 tasks were completed from the 26 existing benchmark suites:
LIBERO-MAX、RoboLab、MetaWorld+、VLABench、ManiGuard、EmbodiedSWE、AutoBio、LabUtopia、MolmoSpaces、RoboDojo、RoboTwin 2.0、ABC Sim、RoboWits、YAMLab、BEHAVIOR-1K、RoboCasa365、RoboCasa、KinDER、RoboCasa-GR1、BiGym 2.0、HumanoidBench、DexToolBench、DexJoCo、MyoSuite、MuJoCo Playground、AM-Bench。
2.3 Task Set
The exam consists of 1,156 tasks in 35 sets. One set is a set of related tasks running in the same simulator. Each task is given a goal, usually described in natural language, such as untying a rope or flipping a page with a clever hand.
For this test, we constructed 9 suites, 411 tasks, and adapted 26 more suites, 745 tasks, from the existing robot benchmark. These tasks combined use 47 robots, covering 9 human-type machines. Table 2 of Appendix A lists the simulators, robots and number of tasks used by each set.
New equipment
We built nine new suites because of fewer tasks for various robotic shapes and capabilities. The original benchmarks for these suites were mostly focused on learning visual movement strategies, see Section 4; 618 of its 745 tasks used single-arms, two-arms or mobile operating robots.
Figure 5 - Tasks in the exam.The left-hand side shows the nine task suites constructed for this examination, each of which gives a picture and the number of tasks, and is divided into three groups as described in section 2.3. The right-hand side shows a picture of each of the 26 suites adapted from existing benchmarks, in order of the main robot form, from single-arm robots to flying robots.
The new sets are divided into three groups, corresponding to the three lines in Figure 5.
RoboARC, DexHandExam and RoboPaintRequires extensive exposure to objects, multi-component components and tools.
RoboDynamicExam and two Embodied Olympics kitsIt involves objects and opponents in motion, so the intelligent body must act at the right time. These kits contain most of the Humanoid tasks in this test.
I'm not sure if I'm going to be able to do that, but I'm going to be able to do that.Tests go beyond the ability to execute a given target: they either give the target only in image form, mix hazardous requests with harmless requests, or require the controller to be adapted to hidden dynamic conditions.
Suitable accessories
A further 26 kits are from existing benchmarks, with tasks ranging from desktop operations under language conditions, long-term household activities, human-body control, use of agile tools, musculoskeletal control, aerial operations and laboratory automation, see Appendix A.2.
For each set, we retain the original benchmark scenarios, robots and success rules; in some sets, we use stricter success rules.
We save the initial state of each mission as a freeze-scenario, write instructions that explain all the agreements that the intelligent body cannot infer on its own, and access the task suite to the three interfaces of the evaluation settings.
We also removed expert strategies and sample data from the original benchmarks to prevent intelligent agents from copying known solutions from their containers.
Core Task Set
We reported the results of a core task set consisting of 456 difficult tasks, including 901 tests. This is because the tasks in the same set often require similar skills, and the current intelligent body is already able to solve many of them.
The core task set excludes tasks that we judge to be simple, as well as those already solved by the initial screening of the GPT-6 Luna; at the same time, it retains all the tasks requested by the authors of each task set.
Since the GPT-6 Luna belongs to the same model family as the GPT-6 Astra, this screening may make the core task set biased to exclude tasks that the model family can solve. Table 2 gives the number of core tasks in each set.
Each core task also has a default setting, and the controlled experiment in Section 3.2 uses the default test for each core task. 403 tasks use Standard by default; 53 tasks use Realtime by default for tasks critical to time requirements. In addition, 1 core task also includes Realtime testing.
The remaining 700 tasks remain in the exam and can be run in the same way.
We have divided the 427 core tasks into multiple phases, such as opening the refrigerator, see Section 3. Figure 6.
Relation stages are most common, such as placement or assembly of objects; most of the tasks containing Dynamic, Body or Competition stages come from new suites, accounting for 86 of 99 such tasks.
Figure 6 shows the core tasks classified according to the stages in the task chart.The graph shows an example task for each category. Each location represents a task in that category, and the category's angle is proportional to its number of tasks. If a task graph contains multiple categories, the task counts each of the corresponding categories once.
The center displays all 456 core tasks; the surrounding location covers 427 tasks with a task map. No task map contains the Safety or Information phase, as the task map of the Embodied Safety Exam lists security rules as a condition for each phase, see Appendix H.1.
Of the 456 core tasks, 255 came from the suites built for the exam, and 201 came from the suites.
3. Analysis
In the core task set, we evaluated eight programming intelligences, as shown in Figure 8. Sections 3.1 give their success rates and Sections 3.2 to 3.5 report four findings.
A smart object consists of a model and a Harness, such as the Codex or Claude Code, and uses a higher reasoning intensity. Harness is also part of our measurement objects, see section 3.4.
Each intelligent organism tries each test once; when the validator determines that the test is passed, that is, considered to solve the test, see Appendix B. The success rate is the result of an open list, see Appendix D.
Stage failure marks
The verifier determines whether an attempt is passed, but does not indicate where the error occurs. Therefore, we divide 427 of the 456 core tasks into multiple stages, as shown in Figure 7a. The edge of the task chart gives the order in which each stage must be accomplished.
Each stage has a type, such as the opening of the pump. These types fall into one of nine categories, see Table 7; each stage also marks the required contact. Figure 7b shows how many stages of the three attempts are in a satisfactory condition.
A failed attempt is most likely to correspond to a failed stage: if an irreversible event such as a fall occurs, it is the stage at which the event occurs; otherwise, it is the stage at which the earliest error that an intelligent organism has failed to correct occurs.
This error may occur before the prerequisites at this stage are met. For example, a smart body may drop a tomato before opening the refrigerator. Each failed attempt marks a maximum of two failures.
For example, Claude Opus 5.5 crashed while transporting milk, so it failed in phase 4, and the failure phenomenon was a loss of balance.
If an attempt satisfies the conditions of a stage, or fails at that stage, it is considered to have reached that stage.
We analyzed eight failed attempts on this 427 mission. GPT-6.1 Sol in the Codex wrote the majority of the labels, Claude Sonnet 5.5 in the Claude Code wrote the rest of the labels; Claude Sonnet 5.5 reviewed all labels, see Appendix H.1.
Figure 7 shows a mission map and progress map for the indoor kiting mission of purchasing items from BEHAVIOR-1K.
Figure (a) Gives the order in which each stage must be achieved, and the type and contact of each stage. Figure (b) Gives the proportion of three attempts at each moment of the rated episode to the stage in which the condition is satisfied. The red line marks the failure phase and its phenomena, see Appendix H.6.
3.1 Overall results
Figure 8 shows the success rate of each intelligent organism in its core task set.The number of tests solved in brackets. The dsh in Harness is DeepSeek Harness, which is DeepSeek's own Harness.
The number of Standard tests each solved by the three smartphones was twice as many as any of the other five. GPT-6 Astra, Claude Opus 5.5 and Gemini 4 Argon solved 24.1% to 33.6% of the 456 Standard tests, respectively.The Three Most Powerful IntelligencesThe other five are calledWeaker intelligences。
Realtime Setting is more difficult than Standard Setting. On 54 tasks with two tests simultaneously, GPT-6 Astra solved 23 Standard tests and 17 Realtime tests; Claude Opus 5.5 solved 10 and 4 items, respectively; Gemini 4 Argon solved 15 and 5 items, respectively.
Under the Standard Setting, Claude Opus 5.5 and Gemini 4 Argon solved many of the failed tests of the GPT-6 Astra; while under the Privileged Setting, the GPT-6 Astra solved almost all of the tests that at least one intelligent body could solve alone.
In both settings, the five weaker intelligences solved only a small number of additional tests. Section 3.4 further analyzes which tests were solved by the individual intelligences.
Of the mission suites with at least 5 Standard tests, most had at least half of the tests solved by none of the three strongest intelligences. Of the 22 such suites, 14 met this requirement. Of the 20 suites with at least 5 Privileged tests, only HumanoidBench met this requirement, as shown in Figure 10.
On average, 10 Privileged suites have a success rate of at least 70% for the three strongest smartphones; none of the Standard suites have achieved this level.
Each attempt in the core Privileged and Standard tests is shown in Figure 9.The sequence of intelligent agents is the same as in Figure 8. Resolved tests are marked separately based on the cost of a single attempt, i.e. the price, using different shallow representations; unresolved tests are marked separately.
Figure 10 shows the success rate of each set.The diagram shows suites that have at least five core tests in corresponding settings. The circle points represent the success rates of each of the three strongest smartphones, and the bar represents the test ratios that all three smartphones failed to solve. The IKEA assembly task in RoboARC serves as a standalone suite statistic.
3.2 What caused the performance decline from Privileged to Standard?
Privileged and Standard- For the GPT-6 Astra, repositioning the environment is much more helpful than obtaining the precise state of the object. Repositioning will increase the success rate from 28% to 53%, while providing precise state will increase it to 45%.
- Under the Standard Setting, the phases that require precise operation become much more difficult. The failure rate for such phases increased by 24 to 46 percentage points, while the roughness phase increased by 1 to 20 percentage points.
- For its failure at these stages, experiments with only hiding successful signals solved 19%, experiments with accurate states solved 21%, and experiments with repositioning solved 36%.
Figure 11 will pair 391 core tasks with Privileged and Standard tests at the same time. GPT-6 Astra solved 303 of them under the Privileged Setting, but 191 of them failed under the Standard Setting.
It solved only 4 tasks under the Standard Setting, two of which combined to solve 116 Standard tests, accounting for 30 percent. Out of 20 suites with at least 5 such tasks, its success rate decreased entirely.
The two conditions lacking in Standard Setting are: the precise state of each object, and the unlimited repositioning.
In a controlled experiment with the GPT-6 Astra in the Codex, we added one or two of the following functions to the Standard interface: querying the precise state of each object, and unlimited resets that support conservation status, as shown in Figure 12a and Appendix G.
On 391 core missions with two sets simultaneously:
After providing accurate status, the success rate increased from 28% to 45%;
The success rate has risen to 53% after repositioning;
At the same time, the success rate has risen to 68% after providing both.
The Privileged test success rate for these missions is 77%.
In cases where resetting is permitted, 66% of success occurs in episodes starting from a saved state. This means that the intelligent body can retry during difficult stages without having to repeat the entire task from scratch, see Appendix E.1.
Figure 11 contains 391 core tasks for both Privileged and Standard tests.For each smartphone, the tasks are divided according to which settings it solves: both settings are solved, only Privileged settings, only Standard settings, and neither settings are solved.
Figure 12 state, reposition and stage failure rate.
Stages that require precise operation are more likely to fail under the Standard Setting. Based on the Stage Marking of 8 smart entities, we calculate the failure rate for each stage type under each set: the number of failed stages in that type, minus the number of attempted to reach that type. We compare two settings within each set, as shown in Figure 12b.
If a phase requires close coordination, reaching the target position or path, or contact with moving objects, it is attributed toStages of precise operationIf it is only necessary to reach a certain area or container, to complete the seizure, or to change the status of the institution, attribute it toRough grain stageSee also Annex H.5.
In the comparison phase type of at least two sets, the failure rate for the precision operation phase increased by 24 to 46 percentage points, and the rough grain phase increased by 1 to 20 percentage points. For example, the failure rate for the assembly phase increased from 34% to 80%, and the placement phase increased from 23% to 35%.
The difference between the two groups is still significant if only statistically the same intelligent body reaches stages under two settings of the same task: a 25 percentage point increase in the precision operation stage and a 15 percentage point increase in the rough grain stage, see Appendix E.1.
Figure 13 shows a precise operating stage that failed under the Standard Setting. In this case, the front cover of the Claude Opus 5.5 holds a deviation of 36 to 39 degrees from the fan axis, while the permissible deviation when installed is 5 degrees; continuous pushing of the cover moves the entire fan.
Figure 13 Claude Opus 5.5 installed fan front cover in Privileged and Standard settings, from DexHandExam.The red mark indicates where the Standard attempted to go wrong. Time is represented by the number of seconds in the Standard episode relay.
In the precise operating phase, the resetting environment for the GPT-6 Astra is helpful, but only providing precise state does not help. We took the Standard test for the GPT-6 Astra that failed in the baseline experiment and grouped it according to its type of failure phase, as shown in Figure 12c.
Some of these failures are likely to be successful in any second attempt. To estimate this number, we used a set of experiments that neither added precision status nor added reset functionality, but only concealed successful signals.
Of the 91 failures in the precise operation phase:
Experiments that only concealed successful signals solved 19%;
Experiments providing accurate status were solved by 21%;
Repositioning experiments were solved by 36%.
In experiments that provided reset, 31 out of 33 successes occurred after reset or restore status.
Consequently, experiments that did not provide a successful signal solved about 2 of the 91 failures by providing an accurate state, compared to experiments that did not provide a successful signal.
In contrast, of the 97 failures in the rough grain phase, only 34% were solved by providing an accurate state and 14% were solved by experiments that did not provide a successful signal, see Appendix E.1.
3.3 At what stage of the mission did intelligent beings fail?
Key points: Where the smart body fails
Failures in the same task often occur at the same stage. For the three strongest intelligences, after failing two pairs of attempts on a test, an average of 56% of failures occur at the same stage; 31% at random.
This stage is often the same in Privileged and Standard settings. Excluding the two most common failures in both settings, 57% of tasks are still consistent in the most common failures in both settings; 9% at random.
Changing the state of a material or the stage of a moving robot body has the highest failure rate. Once reached these two stages, the failure rate is 66% and 45% respectively.
Failure in the same task often occurs at the same stage.Concentration at Failure StageTo measure this phenomenon, see Figure 14a.
For each test, concentration is defined as: the proportion of failed pairs at the same stage after two failed attempts from different intelligences. We use a localized failed phase experiment and average the ratio for each test.
The baseline is a random drawing of the stages at which each failure occurs; the probability that each relative position in the task is drawn is taken from the failure of other tasks under the same setup, see Appendix H.5.
Of the 8 SMEs, the concentration under the Standard Setting was 61%, while the baseline was 37%. Five weaker SMEs often fail at any stage before reaching the task, see section 3.4, so much of this consistency is likely to come from failure in the first stage.
Therefore, we examined the three strongest intelligences individually. Concentration was 56% in at least two of the 217 Standard tests that failed to rank; the baseline for their failure on other Standard missions was 31%.
One of the most common test failures is usually the first stage of the task: of the 264 Standard tests with the single most common failures, 65% of the most common test failures are the first stage, see Appendix E.2.
Figure 15 shows a non-first stage, but one in which multiple SMEs fail together. Under the Standard Setting, all eight SMEs failed the test, with seven of the SMEs using the cup to catch the ball thrown but failing in different ways when taking the ball again.
In Figure 32 of Appendix H.6, the three strongest brains placed the first node of the three-section battery in the container and then dropped the robot on the way to the second node battery; while four of the five weaker brains failed at the first node battery.
Figure 14 shows the failure rate and the failure rate.
Figure 15 shows two interceptions from the RoboDynamicExam.The chart shows which phases each smartphone has reached and failed in one attempt under each setup, and shows three failed phases in Phase 2 of the Standard Recoil.
The mission consists of three phases:
He's got a ball that's flying.
After the ball passes through the slot and bounces back, the ball is picked up again.
Put the ball in a moving van.
In three failed examples in Phase 2, the GPT-6 Astra's cup hit the slope; Claude Opus 5.5's cup was too high to catch the ball; and Gemini 4 Argon's cup hit the slot.
The phases of failure are often the same in both settings. We selected 188 tasks: These tasks have the single most common failure phases in both Privileged and Standard settings.
Of the 105 tasks, the most common failures in both settings were in the first phase, which we excluded. Of the remaining 83 tasks, 47 items, or 57%, had the same most common failures in both settings, as shown in Figure 14a.
If, in accordance with the above method, the failure stages of the settings are randomly drawn and the two most common failure stages of the settings are equally excluded as the first-stage sampling results, the ratio will be 9%, see Appendix H.5.
Changing the state of a material or the stage of a moving robot body has the highest failure rate, as shown in Figure 14b. The failure rate of a category at a given stage is defined as the number of attempts to fail at that category stage, excluding the number of attempts to reach that category stage; both Privileged and Standard settings are combined here.
For example, the failure rate for the Material phase, such as wiping, was 66%, the Body phase, such as walking, was 45%, and the Mechanism phase, such as opening doors, was 20%.
Each category appears primarily in a specific suite and corresponds to a specific robot, so these failure rates cannot prove that failure is caused by the stage category itself, see Appendix E.2.
3.4 What are the differences between different types of intelligent agents?
Key points: Differences between intelligent agents
Compared to stronger intelligences, weaker intelligences are more likely to fail before establishing the necessary contact. Five weaker intelligences experience this failure in the 9.1% to 18.5% contact phase, with the three strongest intelligences averaging 2.7% to 3.7%.[3]
Under the Standard Setting, the three most powerful smart-body-solved test sets are different parts. Claude Opus 5.5 and Gemini 4 Argon combined to solve 64 GPT-6 Astra unresolved tests.
Switching Harness can change the model's success rate by up to 12 percentage points. In 456 Standard core tests, GPT-6 Astra solved 34% in its own Codex and 22% in the Claude Code.
Figure 16 Characteristics of failure of an intelligent organism, overlapping test settings, and effects of different Harnesses.
Five weaker intelligences are more likely to fail without establishing the necessary contact. Among the failure characteristics in Figure 16a, this accounts for 17 to 24 percent of their Standard and Realtime failed attempts, while the three strongest intelligences account for 7 to 10 percent.
For tasks involving at least two stages, 44% to 64% of the weaker intelligences failed to reach any stage in their failed Standard attempts; the three strongest intelligences accounted for 27% to 36%, see Table 19.
If calculated according to each stage of contact they reach, weaker intelligences fail to establish contact in 9.1% to 18.5% of the stages, while the three strongest intelligences fail at 2.7% to 3.7%, see Table 19.
Figure 17 shows a BEHAVIOR-1K test: preparing a lunch box. Eight smart bodies failed all under the Standard Setting. Three of the strongest smart bodies were packed with three or four types of food, two were packed with the Kimi 3.1 Pro, and four were packed with a weaker one.
Appendix H.6 describes these attempts, as well as two other tests, see Figure 31. GPT-6.1 Sol, which is responsible for most of the labeling, belongs to the same model family as GPT-6 Astra, which may lead to bias in these comparisons, see Appendix H.5.
Figure 17 is from BEHAVIOR-1K, Standard Setting.The diagram shows a photo of the head camera of the four failed attempts and shows the location of the failure.
The three strongest intelligent agents successfully solved different test segments. Under the Standard Setting, no test set solved by one intelligent body is a subset of the set solved by another intelligent body, as shown in Figure 16b.
GPT-6 Astra solved Claude Opus 5.5, which solved 59% of the Standard tests, and Gemini 4 Argon, which solved 66% of the Standard tests. Under Privileged Setting, the two ratios are 92% and 95%, respectively, see Figure 23.
Claude Opus 5.5 and the Gemini 4 Argon combined to solve 64 GPT-6 Astra failed Standard tests. Since each smartphone only tries once for each test, some of these differences may be due to fluctuations between different runs.
Of the 69 Standard tests that were solved by at least one weaker intelligent body, the three strongest intelligent agents solved 65 of them. For each weaker intelligent body, 93% to 100% of the tests that were solved were also solved by at least one intelligent body before it in Figure 8, as shown in the last column of Figure 16b, see Appendix E.3.
Harness will also change the results, as shown in Figure 16c. We run GPT-6 Astra in Claude Code, Claude Opus 5.5 in Codex, and Gemini 4 Argon in both.
In the 456 Standard Core Test:
GPT-6 Astra solves 34% in the Codex and 22% in the Claude Code.
Gemini 4 Argon solved 29% in Codex, 25% in Claude Code and 24% in its own Antigravity CLI.
Claude Opus 5.5 solves 26 in the Codex and 25% in the Claude Code.
When the Gemini content filter blocks a response, the Antigravity CLI ends the session; this may explain some of the performance gaps in the Gemini 4 Argon, see Appendix E.3.
3.5 What controls do intelligent beings use?
Key points: How intelligent agents control robots
In the Standard and Realtime setup of a soccer mission, when the intelligences have access to GPUs and training libraries, they train strategies; only strategies that have been trained win matches.
Under Standard Setting, the intelligent body primarily controls the robot directly. In a sample of Standard experiments, this method accounted for 61%.
In the Privileged and Realtime settings, the smartphone primarily writes programs to control the robot. This makes up 82% of Privileged attempts and 97% of Realtime attempts, respectively.
Figure 18 shows how to control and the results of football missions.
In the Standard and Realtime setup of a soccer mission, when the intelligences have access to GPUs and training libraries, they train strategies; only strategies that have been trained win matches.
In Embodied Olympics - Match, there are two versions of the football game, both using the same freezing scenes and opponents, as shown in Figure 18b. In both versions, the intelligent body controls the two Humanoids by the joints, thus requiring a walking pattern.
Additional GPU versions provide GPUs, RL libraries, a copy of the game GPU for training, and a training example in the instruction. In this version, 10 of the 12 attempts under Standard Setting used RL training strategies; 7 of the 8 attempts under Realtime Setting trained strategies.
There were three victory attempts under Standard Setting and one victory under Realtime Setting; all of these victory attempts used trained tactics.
In the CPU version, the smart body handwrites the step; in 12 Standard attempts and 8 Realtime attempts, none of them won once.
Standard's statistics include Harness contrast experiments, so some models appear more than once, see Table 4. Under Privileged Setting, the smart body also wins in the CPU version, but every time it wins it duplicates the opponent's walking network.
The two versions differ in many respects, so it is not possible to determine exactly which of the differences worked with this task alone.
For example, the GPT-6 Astra trained strategy when it had a GPU and won with 10 rounds; without a GPU, it adjusted the handwriting pace and lost with 3 rounds, as shown in Figure 19.
Under the Standard Setting, the smart body primarily controls robots directly; under the Privileged and Realtime settings, they primarily write programs, as shown in Figure 18a. A programming smart body labels the controls used in 1,489 sampling attempts, see Appendix E.4.
Simulation is suspended between requests under the Standard Setting, and 61% of samples attempt to send a block of action each time and view the camera image or sensor readings before selecting the next block of action.
Under Privileged Setting, which is able to access the full state, 82% of attempts are made through program computing movements. Under Realtime Setting, which is continuous over time, 97% of attempts are made using programs, of which 82% are attempts to run control cycles by programs.
Six of the 1,489 attempts were marked as training strategies.
GPT-6 Astra in the GPT-6 Codex completed two versions of the football mission under the Standard Setting.Each line gives the methods and results used, the two-player image, and the number of goals scored by both sides during the match.
4. Related work
This section reviews the research directions most relevant to this article. Appendix K provides a broader overview.
Programmatic intelligence benchmarks.Programming intelligences and other language model intelligences are primarily evaluated on digital tasks, such as solving problems in real-world code repositories, completing tasks in terminals, conducting machine learning engineering and research, and operating websites and desktop applications.
The interface between the model and the computer affects the success rate of the intelligent organism; and studies measure the frequency at which the intelligent organism uses scoring to test vulnerabilities. Embodied First Exam extends these assessments to robot tasks. Under the Standard and Realtime settings, the intelligent organism must understand sensor observation, and usually cannot undo the action. In all settings, each attempt is scored by a program that the intelligent organism cannot modify.
It's a language model for robots.Language models have been used to organize skills into a program, to write robot programs that call perception and control native languages, to specify value charts or critical point constraints for movements that are converted by the programmer, and to write reward functions.
Embodied First Exam assesses the ability of unmodified general programming intelligences to complete the entire closed loop: read interfaces, select control modes, execute controls and make adjustments.
In the physical parameters of language models, the model selects the movements from the action spaces defined by the parameters. The test provides the robot with a control interface, so the intelligent body must also decide how to generate the movements.
Agentic Robots and its Harness.Recent studies use the three controls in Figure 2 to apply general intelligence to robots.
In terms of direct control, the GPT-6 Astra serves as a strategy on the operating benchmark. In terms of code-writing controls, the programming intelligence is used to simulate robots and real-world robot-writing programs, usually using a skill library or other additional tools. In terms of training strategy controls, the intelligence intelligence is used to train strategies in the real world or simulation.
Most of these studies have been conducted in a fixed control manner. This test supports the above three methods and gives selection to the intelligent body.
The Harness used by these smartphones varies in many respects: the interactive interface provided, whether the smart body allows learned strategies to be called as tools, and the support provided for tracking status and checking results.
The test uses unmodified Harness models to run its own suppliers and models from other suppliers via adapters, see Appendix G, so that different Harnesses can be compared on the same task.
The robot benchmark.Most robot benchmarks focus on visual movement strategy learning, evaluating already learned strategies on operations, home and Humanoid tasks, such as VLA.
A learned strategy can only accept a fixed set of observations. A programming intelligent body can run any program on its computer, so this test adds rules about what the intelligent body can access, see Section 2.1, and adapts to 26 existing benchmarks, see Appendix A.2.
Criteria for Agentic Robots have also emerged. CaP-X has changed the abstract level of the native language used by programming smart body combinations; RobotWorld covers operations, mobility, driving and flight control; EmbodiedSWE and BiGym 2.0 evaluate programming smart bodies in agile operations and Humanoid missions.
The Embodied First Exam covers 47 different types of robots from 9 different types. It also distinguishes what a smart body can do when it has full access to a simulation and what it can do only with a robotic sensor.
5. Conclusions
We proposed the Embodied First Exam: a unified assessment framework that could connect any programmed intelligent body to a simulated robot and allow the intelligent body to perform actions in three ways. The exam included 1,156 tasks on 47 robots.
With 391 difficult core tasks with two setups at the same time, the strongest smart body has a success rate of 77% under Privileged Setting, but only 30% under Standard Setting.
In controlled experiments with the intelligent object, increasing the number of repetitions can narrow the gap more than increasing the accuracy of the object; in phases requiring precise operation, resetting helps, while providing precise states alone does not.
Failures in the same task often occur at the same stage; weaker intelligent entities are more likely than stronger intelligent entities to fail before establishing the necessary contact. In the Standard and Realtime setting of a football task, when the intelligent entities receive GPUs and training libraries, they train the strategy, while only the trained strategy wins the game.
When the strongest intelligent body has full access to the simulator, it is already able to solve most of the tasks; but it still has a lot of room for improvement in the Standard and Realtime settings designed to simulate real robot deployments and testing methods.
Future work
Multiple runs on each core test can distinguish between repeated failures and failures in a single run, and can provide confidence intervals for comparisons between the devices and between different settings.
We will also be adding more Realtime tasks, because under Realtime Setting, no smart body can solve the highest percentage of tests.
Finally, since the Standard Setting only provides a robot's sensor read and control interface to the smart body, the same interface can be used for physical hardware once the results are rated.
In the preliminary demonstration, the two SMEs completed the guideline contact game using real mechanical arms, see Appendix I. We plan to systematically evaluate the performance of the SMEs on real robots through this interface.
Limitations
Each intelligent organism tries only once for each test; phase marking and control methods are generated by the intelligent organism; all results come from simulation. Appendix J discusses these and other limitations.
