I'm going to tell you something.What's the matter?
A robotic forecaster reported on November 9 that recently, in conjunction with the China Institute of Telecommunications and Artificial Intelligence (CITAI) in China, the company launched LocoManiLab, a unified assessment framework for human robot mobile operations. This framework is characterized by breaking down the strategic model and the full-body controller into two levels and performing comprehensive assessments within the same standard, equivalent to both brain and full-body controllers.
This framework coversThere are five categories of home spaces, 13 scenarios and 234 tasks.These include:102 atomic missionswith132 combined missionsIt is also used as a unified test core with the Yuri G1 humanoid robot and the dex 3 handyman.
Researchers can compare the performance of different strategies under the same tasks and robot configuration, and can further analyze the effects of controllers on robot execution effects.
After the team evaluated 12 strategic models, the results showed that success rates generally decreased after existing models switched to new tasks and scenarios.See you on missionThe average success rate of80.2%，It's here.Missing missions dropped to 25.2 percent.It's not.Unseen SceneIt's been a long time coming.8.2%。In field randomization tests, XR-1 led the way in both atomic and combined missions under low-intensity visual disturbance, but DM0.5 performed more steadily and achieved higher success rates under medium and high-intensity disturbance.70.5%。01.
It's a great way to create a unified testing environment.It covers multiple family scenes in the kitchen, living room, etc.Mobile operation is not simply a combination of the ability to move and grip a robot. For example, to open a refrigerator, a robot needs to first walk to the refrigerator, adjust its body position, then reach out and grab its hand and pull the refrigerator door. After the refrigerator door changes state, the position of the robot's next step to grab will change as well. Throughout the process, the movements of the legs, body balance and arm operations need to be coordinated, and the robot faces are not a series of unrelated movements. LocoManiLab starts with tasks and scenes, and builds a set of test environments covering multiple daily operations for humanoid robots.There are 13 scenarios.CoverageThe kitchen, living room, bedroom, bathroom and office.It includes two simulated scenes built from real-life scenes.102 atomic missionsIt is used to test individual skills of robots, including the basic ability to approach, open doors, grab and place.132 combined missions,They connect multiple movements into a longer chain of tasks to explore whether robots can continue to interconnect between movement and operation. In terms of data, LocoManiLab supports simulating the full-body movements of real robots through VR remote operation, and can also use motion synthesis and trajectory enhancement to expand training data.14160 simulated remote-operation data, with a cumulative duration of 51.5 hours.In addition, it also collectsAbout nine hours.The real robots are remotely operating data.
Comparison of human-robot motion control systems
Each demonstration documented the first-person view of the robot's images, language instructions, action instructions, and test status, providing the data base for subsequent strategy training.There are 12 strategic models.These models include visual-language-action models (VLAs), world-action models (WAMs) and simulated learning baseline models. These models generate movements based on images, language instructions, and robot state, and are then executed by a full-body controller.I'm not sure what I'm going to do, but I'm going to do it.The platform connects strategic models and controllers through data adapters and executes a process with 50Hz control frequency. In hardware, LocoManiLab uses the Utrecht G1 humanoid robot, equipped with two dex3-1 hands.29 body joints and 14 hand control targets.It is possible to move, stretch, and manipulate objects on the same robot.Evaluations also distinguish between scenarios and tasks within and outside the training distribution and include visual disturbances of different intensities.
02.
Layered survey strategy models and full-body controller performanceWith a unified task, data, and robotic core, LocoManiLab further divides evaluation into two levels: a strategy model and a full-body controller. This is important. Only a strategy comparison can answer who is better at understanding tasks and organizing movements; a controller is also included in the comparison to further answer whether the same action intent can be executed by different physical stability.12 strategic modelsAccess the same set of full-body SONIC controllers, as much as possible to unify the body's execution conditions, focusing on comparing the performance of different models in task understanding, action planning and task completion.
Assessment of the strategic model
Tests not only include tasks seen during training, but also include unseen tasks and unseen scenarios, exploring the robot's performance in new environments, new tasks. The platform also includes field randomization tests. Researchers can change the scene texture, lighting and camera conditions, add interference, and observe whether the robot can continue to complete the task after the visual environment changes.
Example of domain randomization in LocomaniLab
The team selected two VLA strategies, GR00T N1.7 andπ0.5, to access five full-body controllers, respectively, and test them on bathroom scenarios for five atomic missions and five combined missions. This section not only statistics the mission success rate, but also records mechanical output power, leg slippage, fall rate, body joint target change rate and relative completion time.
Two upper-level strategies with multiple indicator radar maps of five full-body controllers
These indicators can reflect the quality of a robot's execution from different angles. For example, even when a robot completes a task, there may be problems of motion instability, foot slip or higher mechanical power consumption. Therefore, the success rate of a single task is not enough. It is only by observing the results of the task and the state of physical movement that a robot can understand more fully how far it is from the actual application.03.
It's the first time I've seen this.Stranger Scene Success rates are generally decliningIn the main experiments of 12 strategic models, the research team used SONIC as a common full-body controller to compare the performance of different models in tasks seen, unseen and unseen scenarios. Test results showed that existing models performed relatively well on familiar tasks, but success rates generally decreased after switching to new tasks and scenarios.See you on missionThe average success rate of80.2%，It's here.Missing missions dropped to 25.2 percent.It's not.Unseen SceneIt's been a long time coming.8.2%。
Results of the test
The advantages of different strategies also change with the test conditions.DM0.5The highest success rate in unprecedented scenarios15.5%。Field randomization tests also showed similar changes. Under low-intensity visual disturbances, the XR-1 achieved a leading performance in both atomic and combined missions, with a success rate of90 percent and 71.1 percent.
Results of randomized field tests
However, as the perturbation increases, the performance of DM0.5 is more stable, achieving a higher success rate in both medium and high-intensity perturbations. On atomic missions, the decrease ofπ0.5 is also less than XR-1, and in higher-intensity perturbations, the decrease ofπ0.5 is less than the decrease in XR-1.78.8%The average success rate against the XR-170.5%。Therefore, the best performance of a model under one test condition does not mean that it can maintain an advantage in all environments. These combinations predict whether the strategy can still maintain a higher mission success rate when the scenery, lighting and camera conditions change at the same time.The success rate of a mission cannot fully represent the execution quality of a robot's body.In the GR00T N1.7 atomic mission test, ALMI achieved the highest success rate, but in its successful rounds, the number of falls was higher than that of SONIC. SONIC produced the lowest mechanical output power in the successful execution of the two strategies, the two types of tasks, but in some comparisons, the success rate of other controllers was higher. These results suggested that the completion of the task alone was not enough to select the execution controller. Because of the relative configuration of success rates, there may be differences in power, stability, and quality of operation.Simulation testing and real robot testingIn conjunction, further real-world missions designed in close proximity to simulations on the U-tree G1 are used to observe whether the relative performance in simulations can be echoed in real machines.
▲ Three real-machine missions and their corresponding simulation clips
The research team set up three missions on Jupiter G1, namely:Open the fridge, move the items from the shelves to the desk, and handle the boxes on the conveyor belt.It is also used to set up corresponding tasks in a simulation environment.
Average success rate of three matching missions
As shown above, the XR-1 is ranked first in both simulations and real machines, GR00T N1.7 is ranked second, and Fast-WAM is ranked third.04.
Summary: Unified Assessment Platform establishedHowever, the scope and conditions of deployment remain limited.With the rapid development of the physical field, the industry's requirements for robotic capabilities are no longer limited to the ability to make moves, but more concerned with whether they can accomplish one thing steadily under different conditions. LocoManiLab provides a unified testing platform for human robot mobile operations, which combines home tasks, remote operation and generated demonstration data, strategy training, and simulation with real-time assessment. Experiments not only explore the generalization and control performance of strategies, but also explore the impact of data on scale and micromobility, but this set of evaluation frameworks still exists in terms of scope and current constraints.
