This is the first time I've seen this.
Project homepage of HumanWAM Full presentation
From purely human data scaling to human mixed pre-training, HumanWAM brings the benefits of scale chemistry learning to zero-sample robot operation and follow-up learning.
What can a robot do directly after pre-training? Give it a target map, can it change its execution? Continue to add human and robot data, and how much more capacity can it grow? HumanWAM revolves around these three questions, showing progress from scale chemistry learning to real operations.
HumanWAM is a type of world-class motion model that combines future image prediction and motion learning in the same framework. The study extends human-machine blended pre-training to approximately 60,000 hours, and the core progress can be summarized in three points:
Zero-shot: The pre-training model is straightforward.The Human WAM-Goal can perform multiple operations without the need for target task demonstration or parameter update. Zero-sample RoboTwin has an average success rate of 13.94%; in 20 team-designed real-machine task assessments, human-machine mixed-machine pre-training will increase the success rate from 20.0% to 30.5%, compared to robot data pre-training, from 10 to 16 with successful missions on record.
Visual Goal In-Context Learning (ICL): To accomplish a task based on the objective diagram, changing the diagram can change the execution.The model combines the current scene with the target diagram to generate action; in contrast to fixed strategic parameters, changing the target diagram can change the classification of the toy to the drawer's selection and the position of the book. The target diagram can also be viewed from a human first point of view, allowing the desired object relation to be transmitted through the image to the robot.
Scaling: Ego, Robot and Robot+Ego are three paths that observe capacity growth with scale.Expanded purely human first-person perspective data to enhance unified robot post-training generalization performance; expanded purely robot data to enhance direct zero-sample execution; continued scaling after the mixing of the two types of data, and further improved success rates based on robot pre-training; expanded mission coverage.
These capabilities also provide a foundation for further learning: After-task training, HumanWAM-Goal has an average success rate of 88.65% on RoboTwin and demonstrates long-range execution capabilities such as continuous desktop sorting and cross-regional housekeeping.
On the table, the rod and pencil are placed, the robot first adjusts the pencil and then inserts the pencil. Faced with objects that need to be transferred, it picks up with its left hand, places it on the front platform for a while, and then picks it up with its right hand.
These zero-sample demonstrations have one thing in common:The pre-training model can be executed directly without the need to collect samples for the target task or update the model parameters.
Zero-sample operation of the Human WAM-Goal is key. In different tasks, the ability to grab, place, and insert with both hands becomes a direct call.
01 Direct handlers: Pre-training models are rolled over to the zero sample list
The most intuitive test of zero-sample capability is to get the pre-training checkpoint out of the training process and face the target task directly.
Under RoboTwin 2.0's zero-shot protocol, pre-trained checkpoints were assessed directly without adapting to the RoboTwin target mission. The HumanWAM-Goal human-machine hybrid model achieved an average success rate of 13.94%, leading the paper in open-source checkpoint controls that successfully completed the assessment; only the robot version of the data was 11.48%, and π0.5 was 1.6%.
Figure 2 shows the zero-sample result of RoboTwin 2.0, extract paper table 12. The mean is the arithmetic mean of Clean and Random; the graph shows some of the open-source checkpoints where the assessment can be completed.
Figure 3 Overview of zero-sample results. A list of project snowmobiles is presented as a static graph using model labels and scores in animation; models that have not been evaluated due to deployment barriers are listed separately.
The simulation benchmark gives a comparable result. Next, the more direct question is: How many operations can it perform by deploying this pre-trained model to a real robot?
02 20 real-machine missions: mixed-machine pre-training to make more operations directly possible
It's a great way to test the capability of the system.The research team specifically designed a set of real-machine zero-shot assessments that included 20 missions.10 simple tasks, 10 difficult tasks, covering contact with change of state, grabbing and moving, opening operations, space placement and alignment, stacking and inserting skills in five categories.
From ringing tablecloths, pulling drawers, putting cups on cups, stacking blocks, and raising bottles with each hand, the model faces specific objects and operational requirements.Each task is repeated 10 times, and each checkpoint is performed 200 times.The task is specified by the target diagram, the pre-training model is deployed directly, does not provide a demonstration of the target task, nor does it update the parameters. Evaluation simultaneously records the final success rate and the progress scores of the intermediate steps, observing both the steps the model can take and how many times it can take in the end.
In the same set of tasks, the study compared two training methods:Pre-training with robot data alone, and a total of about 60,000 hours of mixed robot+human pre-training.The results can be intuitively put together:
Figure 4 indicates that the same set of 20 missions were evaluated, compared to only robot pre-training and approximately 60,000 hours of manned mixed pre-training. The average success rate was 20.0%→30.5%, the mission covered 10/20→16/20, the average progress score was 2.535→4.205 ((full 10 points). The coverage indicates at least one success in each of the 10 trials; the figures are from the Standard Scaling series of papers.
The success rate increased from 20.0% to 30.5%, a relative improvement of 52.5%; the number of successful assignments increased from 10 to 16.The benefits of mixed training are reflected both in the number of times completed and the scope of the task. Medium-sized mixed models also achieved a 26.0% success rate, covering 14 tasks, forming a continuous growth of 20.0% → 26.0% → 30.5% tonnes.
In single tasks, this change is more specific: crawling and shaking the bottle is increased from 60% to 90%, putting the blocks on the tray is increased from 10% to 50%; in the three tasks of stacking the blocks, arranging the blocks, and bottling the bottles by hand, the success rate of the pure robot baseline is 0%, and about 60,000 hours of mixed models reach 30%, 20%, and 30%, respectively.More experience began to translate into more operations that could be executed directly.
Let's look at a set of zero-sample demonstrations: grab the purple blocks, move them over the green blocks, align and relax; then switch to a two-handed task, with one hand raising the drink and the other hand continuing to grab, eventually holding the two bottles on each side.
The blocks in the zero-sample demonstration of the project are stacked with double-handed bottles. Each task selects four real key tables, running from left to right, reading from top to bottom, presenting the initial state, intermediate operation and execution results. The success rate above comes from repeated evaluation, and the module diagram shows the representative execution process.
Other demonstrations also show the intermediate steps needed to accomplish the goal. The model adjusts the position and direction of the container before placing the stick in the pencil; when moving the object to the other side, it uses the front platform to allow the left hand to continue the operation.The static target diagram gives an endpoint, and the robot needs to complete the process in real-world scenarios.
Figure 6 shows another set of zero-sample operations. The upper group: initial scene, adjustment of the pencil, moving rods, completion of insertion. The lower group: left-hand pickup, platform provisioning, right-hand pickup, moving to the other side.
From repeated evaluations of 20 missions to stacking, receiving and executing hand-to-hand operations, HumanWAM demonstrates a direct benefit of scaled-up pre-training:The model was able to complete a series of real operations before the start of training after the target task; human-machine mixed data allowed this list of capabilities to continue to expand.
03 Continue reading: Moving the basics of pre-training to SOTA
The direct deployment answer is what would the AI model already be like. If it were given a demonstration of the task at hand, how high would the foundation support the performance?
In RoboTwin 2.0, the HumanWAM-Goal achieved by using 50 Clean demonstrations per mission, combined with data enhancement and post-training formula optimization:
Clean→Clean：92.70%
Clean→Random：84.60%
Average success rate: 88.65%
The result isThe paper reports on RoboTwin 2.0 SOTAThe average success rate is higher than the GO-3 Nano Preview in the comparison.3.64 pointsThe evaluation covers 50 tasks, two conditions, 100 trials per condition, a total of 10,000 executions.
The five highest average success rates of extract papers in Table 13 were obtained after RoboTwin 2.0. 88.65% came from the paper's list configuration, which included additional post-training optimization.
Figure 8 Overview of post-training outcomes. Modeling controls in the paper show that HumanWAM-Goal is at the top of the list with an average success rate of 88.65%.
From zero-sample to post-training, HumanWAM demonstrates two values of pre-training models: one capability can be deployed directly and the other provides a stronger starting point for further learning. The next complex task demonstration is being built on this basis.
04 Change the picture and tell the robot mouse how to make a mouse this time
It sounded simple, but it omitted a lot of details: which box did the toy car put in, which drawer did the tools put in, the notebook on the left or right of the shelf?
HumanWAM-Goal introduces an intuitive interface:Use a target diagram to describe the state you want to achieve.The model combines this diagram, current observation, language and robot state to generate actions. In the Visual Goal ICL experiment, the strategic parameters are fixed, changing the input's visual objectives, and the execution results can be changed.
Goal A requires the toy car to enter the blue box, the furry toy into the white box; Goal B exchanges the two types of toys. The same strategy of training after the mission, with changes in the rules of reception.
Figure 9 shows that each target has four execution keys. The left-handed blue box is the input of the target chart, the right-hand side is the left-to-right reading from top to bottom, and the green box is the state after execution. The two controls use the same training strategy after the task, and the reasoning parameters remain unchanged.
The target diagram can also change the operation steps that need to be passed. Putting the ribbon and curve into the upper drawer, and inserting the middle drawer, corresponding to different opening and placement movements.
Figure 10 shows the target contrast between the upper and middle layers of the drawings. The left side of each group is the target diagram, while the right side shows the initial scene, the opening, the transfer of objects and the result of placement, showing how the target changes the collection process.
More interestingly, the goal doesn't have to come from a robot's perspective.
When the input is replaced with a first-person perspective image, the robot can still adjust the layout of the notebook accordingly: one image requires the brown notebook to be on the right side of the file shelf, and another image requires it to be on the left side, resulting in a change in execution.
Figure 11 Each human target image corresponds to four robots performing key tasks: initial scene, picking up a laptop, inserting a designated position, loose-handed status. The blue box retains the actual input of the human hands and scenes, and the green box marks the placement results.
The value of these controls is to make the task requirements more specific: categories to, spatial relationships to, and object locations can all be transmitted through images. Human operational experience connects with the ability of robots to control and achieve the same visual goal.
05 Next step: From table sorting to cross-regional housekeeping
Real-life tasks often require multiple objects to be handled continuously. After the cups are removed, documents are collected, and books are drawn; each step changes the environment, and the next step is to proceed from a new state.
HumanWAM is showing an article14 Stages of Continuous Table Planning TrajectoryCover and move the cup, lock the pen and stick in the pencil, open the separate drawer to store the ribbon and scroll, and insert the notebook into the file shelf.
Figure 12 shows six key steps from the same 14-stage trajectory, read from left to right, in ascending and descending order.
In limited desktop space, subsequent moves also need to take into account previously completed layouts. The cups are left next to the display, the documents are in place, robots continue to process the drawers and file shelves, bringing together the skills of moving, inserting, opening and so on.
Another homework presentation expanded the scope to several areas: first classifying toys at the main table, stacking books, then moving to the side table to collect the empty bottles and throw them in the trash can, then collecting clothes from the two chairs and putting them in the laundry basket.
Figure 13 outlines eight key trends in cross-regional housekeeping, including table sorting, bottle picking, discarding and clothing collection.
Visual targets let robots know what kind of clutch they want to make, while continuous execution extends the request to multiple nodes. The combination of the two shows the possibility of moving from a single operation to a complete workflow.
06 Where did the ability come from? Humans, robots and mixed data, three scaling paths linked together
The results and demonstrations above have a common research problem:Can the experience of robot learning continue to expand with the addition of human data?
Ego Scaling: Learning from the first-person experience of humans.The study constructed 50,000 hours of first-person human-operated content with hand-muscle movement and language labels, and conducted screening, weight-loss, and motion alignment to make daily operations a model-training experience.
When the purely human pre-training scale was expanded from 100 hours to 50,000 hours, the Scene-OOD success rate increased from 49.1% to 73.2% after unified robot post-training.Random is fromIncreased gains of 21.5% to 42.9% are particularly evident in the context of environmental change.
Figure 14 Pure Human Data Scaling Experiment, paper Fig. 6. The lateral axis is the number of hours of human pre-training and the longitudinal axis is the success rate after unified robot training; these values are the result of post-training generalization.
The value of human data is also reflected in its utilization efficiency.1,000 hours of human data pre-training and more than 5,000 hours of robot data pre-training in Scene-OOD and Random TestingIn another small group of experiments, after 5,000 hours of human pre-training, only five robot samples per mission were used, and the performance on Scene-OOD exceeded the baseline of the 50 samples that were not pre-trained.
Robot Scaling: Robot data itself can also drive capacity growth.Pure robot pre-training was expanded from 2,005 hours to 14,618 hours, and the average success rate for RoboTwin zero samples increased from 2.80% to 11.48%.
Robot+Ego Mixed Scaling: Combining the two types of data can further enhance capabilities.Continuing to expand to about 30,000 to 60,000 hours of human-machine mixed data, RoboTwin's zero-sample average success rate rose again to 13.16% and 13.94% respectively. Previous real-machine ratings showed the same direction: from pure robots to two-tier mixed scales, with average success rates of 20.0%, 26.0% and 30.5% respectively.
Zero-sample success rate increases with the size of the data. The two curves show the results of Clean and Random respectively, and the mixed training of light green area markers. The average success rate increased from 11.48% of the largest-scale pure robot model to 13.94% of the mixed model, an increase of 2.46%..
Mixed training also extends the range of tasks that models can touch: the coverage of single checkpoints ranges from the largest-scale version of pure robots.18 increased to 22。
Figure 16 Zero sample tasks cover as the data scale expands. The green curve covers individual checkpoints, while the grey zero line covers the cumulative merger of checkpoints of different sizes. The coverage standard is at least 2% of the average success rate for each task: Clean and Random; 22 are for single checkpoints and 29 are for cumulative mergers.
In controlled post-training experiments, adding human data pushed Random's success rate from 70.80% to 74.36%.
Purely human data can lead to scaling, and purely robotic data can lead to scaling; success rates and mission coverage continue to grow after the combination of the two types of experiences.Human data provides a rich experience of scenarios and interactions, robot data provides executable control oversight, and mixed training gives the value community of both now on robotic behavior.
This also requires model design to take the experience into action. HumanWAM-Goal connects different sources of data through a unified action representation, and uses visual targets explicitly as inputs to participate in future video and action production in conjunction with current observations, language and robot states.
Figure 17 of the Human WAM-Goal architecture, paper Fig. 16. Objective Chart refers to a periodic view state in which a model combines current scenarios to generate future video and action.
Scaling accumulates actionable operational experience, and Goal condition provides an interface to express task requirements. The two jointly support the zero-sample capabilities of the pre-training phase; on this basis, post-tasking training extends to more complex target response and long-range execution.
From picking up a stick, to changing classifications based on a chart, to collecting household chores across regions, HumanWAM has turned the large-scale experience of picking up a stick into a set of concrete answers. As human and robotic data expand together, there is room for robots to continue to grow.
Documentation: This article is based on the paper, project pages and supporting materials. The graph presents the key threshold selection for the original project video. RoboTwin zero samples and post-training results use their respective evaluation protocols; complex visual target control and long-range demonstration use post-training strategies. Real machine fragment samples 30.5% correspond to approximately 60,000 hours of mixed models in the Standard Scaling series of papers, with selected demonstration of the execution of different tasks in the project.