Robots are entering their own era of computer use.
Yesterday, Fable 5.2 released by Anthropic brought new advances. Compared to its predecessor models, it significantly improved its ability to control physical robots, even completing tasks that had never been specially trained before.
In a RoboDojo Sim simulation review, Fable 5.2 scored 64.9%, well above the GPT-6 Astra in an Anthropic internal configuration test (29.8%).
As early as September, MIT researcher Phillip Isola suggested in a paper titled Robot-Use Agents, that in the future, robots that access the Internet and open sensors and executor interfaces could become tools for general Agent calls.
This also fits with the System 1 and System 2 business model in the field of physical intelligence, where the general model is responsible for high-level reasoning and task planning, and the underlying strategy is responsible for the execution of specific actions. Models such as GPT-6 Astra and Fable 5.2 have initially validated the feasibility of this route.
However, there are two key issues that remain to be resolved in order for it to really run.
The first is the model. Closed-source front-end models have already demonstrated good robot handling capabilities, but researchers cannot directly gain weight, making it more difficult to use robot interaction data for continuous training.
The second is systems. Different robots differ significantly in hardware structure, sensors, and control interfaces, and simulations differ from real environments. The cost of cross-environment migration and large-scale experiments would be high if every robot accessed a tool system.
Therefore, RUA requires a unified set of interactive interfaces that shield the underlying differences in different hardware and environments, allowing the model to access perceptual information, call control tools and receive executive feedback in a consistent manner.
Recently, Pieter Abbeel, co-director of the Berkeley Artificial Intelligence Research Laboratory (BAIR) at Tsinghua University, a team of researchers from Dingning, Liu Xiaobo, Xu Mondi, and Zhou Burwen, together with Pieter Abbeel, launched and open-source a uni-robot-agent to try to create a unified, modular Harness called Robot-Use Agent (RUA) that enables language models to control different robots and simulation environments through standardized tool interfaces.
Link to the paper:https://arxiv.org/pdf/2611.01560
Link to the blog:
The code link:https://github.com/Open-RUA/uni-robot-agent
On the RoboDojo benchmark, the open-weight model Qwen 3.8-Flash-Next with the framework scored 31.39 points, up 28.97 points from the GPT-6 Astra + RoboProbe officially announced earlier by RoboDojo. Under the same framework, the GPT-6 Astra scored 51.90 points further.
Qwen scored 53.50 points in a mission assessment of 19 missions with no mission-specific mitigation, while the pi0.5 and zero-sample Dream-1 scored 5.75 and 10 points respectively. The GPT-6 Astra with the framework scored 80.50 points.
In addition to the running score, the work also tested the extent to which the model can operate the robot, uncovering the reasons for the failure, capability shortcuts and running costs.
Furthermore, they wanted to build a common underpinning that would allow open-source models to access different robots and pave the way for subsequent training and localization deployments.
Why is Unified Interface the basis for trans-environmental globalization?
In the field of Coding Agent, the same set of tools can be used for different programming tasks, models can read files through similar interfaces, execute code, view run results, and be able to replicate existing tools in different environments.
Robot operation involves more hardware differences. Even when it comes to clutching objects, single-armed robots and two-armed robots may have different movement interfaces, models need to understand new parameters, coordinate systems and control rules, and the original tool calling experience is difficult to migrate directly.
Uni-robot-agent hopes to bridge these differences through a unified tool interface.
For example, Franka single-armed robots need to position, grab and stack blocks, ALOHA double-armed robots need to coordinate two mechanical arms to complete object interfacing, and real YuMi double-armed robots need to control the alignment and interfacing of the plugs and sockets. Although the hardware structure, operating space, and task objectives differ, models can use a unified perception, movement, skills, and task control interface to handle control differences between different robots by a bottom-level adapter.
In other words, the model can use consistent tool calling rules. When entering a new environment, developers primarily need to implement a corresponding adapter to reduce modifications to the upper layer Agent logic.
Reuse of tools in different environments.
Unified Interface has a deeper meaning.
If different environments use the same tool semantics, parametric units and feedback formats, the interaction trajectories produced by robots performing tasks are easier to put together, and the experience accumulated in single-arm stacking, two-arm coupling and real-machine interconnection can be brought together into the same data system.
This design creates conditions for cross-environmental Agent training, where researchers have the opportunity to use interactive data on different robots to train the same model and learn the use and operation of portable tools.
How is uni-robot-agent designed?
The design of the uni-robot-agent revolves around two goals: to provide a unified basis for cross-environmental Agent training; and to evaluate as objectively as possible the robot operating capabilities of the general model.
For this purpose, the research team divided the system into five layers: Agent, Orchestration, Tools, Services and Backends.
The Agent layer is responsible for interacting with the model, managing context and task decisions; the Orchestration layer is responsible for scheduling tool requests; the Tools layer is responsible for providing a unified operating interface to the model; the Services layer is responsible for managing running resources; the Backends layer is responsible for connecting specific robots, simulators, and model implementations.
This modular structure supports independent replacement models, tools, and robot backend. Researchers can compare the performance of different models without rewriting the entire system and accumulate consistent structural interaction data.
Objective evaluation requires as close as possible to the actual deployment conditions. Simulation environments can directly provide precise three-dimensional positions of objects, CAD models, hidden scenarios, or even the success of a task, information that is usually not directly available to real robots. If a model relies on this privileged information to complete a task, it is difficult to determine exactly how much visual understanding, spatial reasoning, and operational capability it has.
Therefore, in the main simulation experiments, the uni-robot-agent only provided the model with RGB images, camera labeling, robot body status and task instructions, without providing depth of field, object CAD assets, hidden scene labeling or task verifier feedback, and without allowing external network access. At the same time, the research team also limited the shortcuts specifically tailored to the task evaluation, avoiding the model getting high scores by providing good pre-written task responses.
So, without this additional information, how do you get a general model to perform complex robotic operations? The key is to organize the information that the model gets and ensure that the tool performs reliably.
Key design choices. Each row shows two interrelated designs, used to ensure the consistency of physical states, the accuracy of action execution and context management respectively.
For example, when a robot grabs an object, the model needs to calculate the grabbing coordinates based on the image and the position of the mechanical arm, and if the image is from the previous moment and the state of the mechanical arm has been updated, the result may be biased.
Uni-robot-agent binds camera images, labeling information, and the state of the robot to the same moment, reducing the positioning bias caused by the change in state. When performing the action, it is requested to avoid repetitive action, and monitors the state of motion of the mechanical arm, identifying the action stagnation or performing abnormalities.
After the action is completed, the system will execute the results and provide new environmental observation feedback to the model to help it determine whether the task is successful and whether the next step needs to be adjusted.
The framework also introduces asynchronous execution for problems involving longer time-consuming robot movements. Models can continue to handle other tasks while some tools are running, while motion requests on the same mechanical arm line up in sequence to avoid instructional conflict.
These designs incorporate the engineering experience of the robotics field into Harness, reducing information misplacement, repetition of actions and lack of feedback interference with model performance.
By rationally organizing information and providing reliable control tools, the uni-robot-agent can still demonstrate competitive robot-operation capabilities for general language models without relying on imitating privileged information and task-specific shortcuts.
What is the extent to which an open model can do experimental verification?
To test the effectiveness of Unified Harness, the research team first conducted a simulation experiment on RoboDojo.
The benchmark consists of 42 robot-operated tasks, covering five dimensions of generalization, precision operation, long-range execution, memory, and open vocabulary instruction. 252 assessments were performed on each group of major experiments.
The impact of Unified Harness is obvious.
The results of the RoboDojo benchmark test. Avg. indicates the non-weighted average of the indicators under the five capability dimensions.†GAP (Hybrid) indicates that the scheme is combined with π0.5.
Under the same Qwen 3.8-Flash-Next model, the RoboProbe scored only 1.90 points, while the RPent scored 7.75 points and combined with the π0.5 GAP Hybrid scored 21.01 points.
After being replaced by a uni-robot-agent, Qwen's score improved to 31.39 with a 25% success rate. Compared to RoboProbe, the overall score improved to 29.49 points.
This group of experiments showed that even if the model weight remained unchanged, the design of the tooling system could significantly affect robot operation performance.
The framework has also been adapted to other models. DeepSeek-V4.1-Flash scored 27.11 points and GPT-6 Astra scored 51.90 points with a success rate of 46.67%.
The advantages of different models are not the same. Qwen and DeepSeek perform relatively well on memory, open vocabulary tasks, compared to fine-tuning and long-range execution; some dedicated VLA, WAM models have more advantages on long-range tasks.
Real-machine testing further tested the model's capabilities in the physical environment.
The team used two-armed robots to perform 19 operational evaluations, covering space understanding, object generalization, instruction follow, combined generalization, and five dimensions of unseen tasks. Specific tasks included test tube plugs, arm delivery bottles, stacking columns in color order, scrubbing whiteboards, and completing the hanotta, among others.
A real robot mission that is representative of five dimensions.
The GPT-6 Astra scored 80.50 points, with a combined generalization dimension of 20 points; Qwen scored 53.50 points, leading the π0.5 and zero-sample Dream-1s through mission-threatened mitigation.
Real machine ratings: 20 points for each skill and 100 points overall.
Qwen and Astra use RGB images and the depth of information provided by RealSense D405 sensors, while π0.5 and Dream-1 use RGB only. Therefore, the difference in performance cannot be entirely attributed to the model itself.
More importantly, the model relies on what external information.
Qwen scored 35.89 points while simultaneously providing depth and camera labels. After removing depth and retaining labels, the score was 31.39 points. Further removing labels, only retaining RGB images and the robot's own status, the score dropped to 9.13 points.
Astra scored 55.83, 55.04 and 31.29 points respectively in the corresponding setting.
Reducing the performance of robotic operations under geometric information conditions. The diagram records the process of completing the task and the final score of the model through 771 tools in 42 tasks when enabling the skill library.
This demonstrates that the model can complete some spatial positioning tasks without direct depth measurement with the help of camera marking, multi-point observation and geometry calculations. However, after a lack of marking, the performance of the operation significantly decreases, and it is still difficult to restore reliable spatial relationships with RGB images alone.
Whether a pre-defined skill library will affect evaluation results is also a concern for the research team.
After closing the skill library and skill writing functionality, Qwen's overall score dropped from just 31.39 to 30.31, retaining 96.6 percent of its original score, and its success rate dropped from just 25 percent to 24.58 percent. This suggests that its performance is not primarily dependent on pre-packaged task skills, standardized perception and exercise tools, combined with the model's own reasoning ability, have been able to support a substantial portion of the operation.
Skills to guide the impact on the performance of tasks.
In terms of cost, Qwen's average API costs about $1.09 per mission, DeepSeek is $0.94, and Astra is $21.29.
The speed of execution also affected the efficiency of the robot Agent. The overall score reached 25 minutes for the first time, with Astra cumulatively consuming about 7.4 million tokens for 42.6 minutes; Qwen consuming 23.5 million tokens for 95.5 minutes; DeepSeek consuming 49.9 million tokens for 99.2 minutes.
In order to reduce the wait, the research team further tested the asynchronous flow lines.
Of the five tasks selected, the flowline execution time was shorter than serial execution, decreasing by 4.8% to 73.7%. Among them, the time taken for the transmission band to capture the task was reduced from about 59.7 minutes to 15.7 minutes.
When did the General Model become capable of robot operation?
Unified Harness allows different models to be tested under close interaction conditions and also allows us to ask a more basic question:
What capabilities does a general model need to have in order to accomplish robotic tasks?
The team further compared 13 different family, intergenerational and scale models, and found that the robot's operational capabilities and model size and release time did not have a simple correlation.
Intergenerational, model series and parametric scale performance of robot operations.
For example, in the Qwen series, the Qwen 3.5, 3.6, and 3.7 models tested did not achieve a complete mission success record. The Qwen 3.8, 27B models achieved a success rate of 8.06% and the Flash-Next increased to 25%.
GPT-5.6 Sol has been able to complete some robotic missions with a success rate of 19.17%, GPT-6.1 Sol improved to 37.50%, and GPT-6 Astra further to 46.67%.
These results suggest that robotic operational capabilities may have been present in general models for a long time, and that their improvement may be related to training data distribution, capability combination and model training methods.
By analyzing the failure trajectories of different models, the team categorized four key capabilities.
The diagram shows the correlation between the combined scores of each model on RoboDojo and the error rates of the following four categories of capabilities. The lower the error rate in these four dimensions, the better the robot task performance of the model is usually.
The first is visual understanding. Robots need to identify objects, judge direction, and confirm the state of operation. Some earlier Qwen models had identified the number 0 as 9, or judged the cups to be positive, resulting in subsequent actions based on incorrect observations.
In the trajectory examined, the percentage of such errors in some older models ranged from 12.2% to 23.8%.[13] No such errors were found in the Qwen 3.8-Flash-Next corresponding examination sample.[14]
The second is spatial positioning. The model needs to determine the three-dimensional position of the target based on the image, and triangular measurements can be made using multi-point observation when depth information is lacking, but cross-point matching the same physical position remains difficult.
The paper found that even with GPT-6.1 Sol, about 31.5% of the relevant triangular measurement calls failed to be checked for geometric consistency.
The third category is geometric computation. Robots need to switch positions and gestures between the camera, the world coordinate system and the mechanical arm coordinate system, with some older Qwen models having an error rate of 47.3% to 81.8%, Qwen 3.8-Flash-Next down to 8.2% and DeepSeek-V4.1-Flash down further to 1.5%.
Finally, it is physical reasoning. The model also needs to judge whether it is reasonable to grasp the gesture, whether the mechanical arm can reach the target, and whether there is a risk of collision in the movement, and some models will repeatedly fail to grasp the gesture, even if they find the target position.
The team also analysed the relationship between robot performance and general benchmarks such as ERQA, MMMU-Pro, ScreenSpot-Pro, and LiveCodeBench v6, and found that visual understanding, physical reasoning, spatial positioning, and coding capabilities are associated to varying degrees with robot performance.
The relationship between robotic performance and general modeling capabilities.
Robot operations involve continuous visual judgment, spatial positioning, geometric computation and physical decision-making, and any one link that fails can fail the task. In the future, enhanced RUA capabilities will need to focus on these interrelated core capabilities.
General framework,
What is uni-robot-agent compatible with?
In addition to direct call tools, the uni-robot-agent also supports Code as Policy、Hybrid mode (VLM collaboration with VLA/WAM), and Clean Mode (Clean Mode) with only basic tools. Different programs can be tested under the same framework.
Code as Policy allows the model to manipulate the robot by writing code, allowing the model to perform calculations, call actions in the program, and decide whether to proceed based on feedback.
The team used Qwen to compare code controls to direct tool calls in two ways, covering RoboDojo's five capability dimensions. In 252 RoboDojo reviews, Qwen scored 25.60 points.
Qwen scores in the Code as Polciy mode.
The framework also supports the Hybrid model, allowing VLM to collaborate with VLA to complete tasks. VLM is responsible for understanding instructions, analyzing scenarios, and organizing tasks. VLA generates motion segments based on observations, performs feedback, and makes subsequent judgments.
The team demonstrated this collaboration in a simulation of platforms such as Panda's mechanical arm, Allegro's Flexible Hands, Jupiter's Go2 quadruped robot, Jupiter's G1 humanoid robot, and the R1Pro wheeled robot, and accessed motion models such as π0.5, GR00T, and NaVILA, covering agility, humanoid robot operation, and mobile navigation.
VLM and VLA collaborate through a uni-robot-agent in different environments.
As the model's capability increases, some of the computing and operating logic that is currently encapsulated by the tool can be gradually handed over to the model itself, and Harness is also expected to become thinner in the future. To explore this direction, uni-robot-agent provides Clean Mode, retaining only the four basic tools of move, hold, finish, capture_scene, allowing the model to complete tasks through observation, computation, and combined action.
Under the camera-labeled setting, Qwen and Astra in Clean Mode retained scores of 90.4% and 90.5% respectively for the full tool configuration, with average token usage decreasing by 35.7% and 78.8% respectively. Compared to the historical full configuration, the model still maintains most of the task performance after the streamlined tool and context, providing preliminary evidence for Harness thinning and enhanced token efficiency.
Qwen, Astra in clean mode token consumption versus scores.
The framework's compatibility also extends to more real robots.
The team accessed real machines such as the ARX One, the Sprint Piper-X. Different hardware control differences were handled by underlying adapters, and the model continued to observe the environment through a unified interface, performing the movements, and adjusting the next step according to feedback.
The ARX One, the Piper-X, has real access.
Access code and deployment instructions are also open, allowing developers to run a uni-robot-agent on the corresponding hardware, replace models, and test their own tasks.
What is the meaning of this verse?
It's a new divide in robotic intelligence.
Traditional dedicated strategy routes are highly tied to hardware, expansion costs are high, and face problems of generalization and longevity, while the uni-robot-agent routes allow general models to manipulate robots directly through a unified interface, with significantly different expansion logic.
Uni-robot-agent is the first open-source product launched by the OpenRUA project. Initiated by a team of researchers from top universities and cutting-edge laboratories, OpenRUA covers a body-intelligence and language model with the goal of building an open infrastructure for agents capable of operating in the physical world.
The team's homepage:
From this point of view, uni-robot-agent is not only about providing a framework for robot operation, but it also demonstrates that open-weight models can be competitive in simulations and real machines, provided that information is properly organized and tools feedback reliable.
However, this is only the first step. The next key is whether the interactive data can be used to continuously update the model. Uni-robot-agent has set up the infrastructure and further training and development has just begun.
The link is as follows:
