Hands lift the fruit basket, turn around, and send it to another table. For humans, this is just a normal daily activity; but for robots, it requires constant coordination of moving the tray, the dryer, the arms, the head, and the dexterous hands. However, with the introduction of mobility and dexterity, the collection of remote demonstrations by robots becomes more complex and even requires the collaboration of multiple operators.
Human day-to-day operations involve a wealth of full-body co-operation experience. So can we directly use these human examples to help robots learn the sophistication of moving arms?
A joint research team from institutions such as the Hong Kong University of Science and Technology, Beijing Institute of Intellectual Property, Beijing University, and others has proposed DexRoam, a complete system for the collection of human samples from the first perspective, the alignment of body movements, to robotic strategy learning and machine execution. The work has been accepted by CoRL 2026.
DexRoam's core value is that while addressing human-machine motion differences, it maintains continuous, fine-tuning full-body synchronous relationships in the human demonstration, enabling it to be used in conjunction with robotic data for strategic training, to expand data sources for complex mobile arm agility operations, and to reduce reliance on high-cost robotic remote demonstration.
The paper is titled: DexRoam: Learning Mobile Bimanual Dexterous Manipulation from Egocentric Whole-Body Human Demonstrations
I'm not going to be able to do this.
The main page of the project:https://dexroam.github.io/
This is the first time I've been able to get a link to the code.https://github.com/zhourui9813/DexRoam
The following is a link to the Strategy Training Code:https://github.com/zhourui9813/DexRoam-Policy-Training
Framework for Methods: Learning from human whole-body demonstration to robotic strategy
Lightweight collection of whole body samples of humans and robots
Figure 1 DexRoam data collection system
DexRoam builds a portable first-person full-body collection system with no need for external motion capture devices. The demonstrators only need to wear a Meta Quest 3 headset and a head-mounted ZED Mini binocular camera, which can simultaneously record first-person video, body movements and hand keys, without the need for external cameras, environmental markers or additional physical tracking devices. The system also displays camera images in real-time on the headset, keeping the visual experience during the collection in line with the visual observations of subsequent strategy training.
In addition, the team has built a single, full-body remote operating system that uses VR headsets, controllers and data gloves to simultaneously control the robot's bottom, trunk, arms, head and thumb. The two systems provide human and robot demonstrations, respectively, to build a data base for further joint learning.
Transcending human differences and maintaining continuous full-body collaboration
Figure 2 DexRoam 3 Stage manned operations coordinated flow
To overcome human-machine differences while maintaining continuous full-body synchronous motion, DexRoam designed three stages of body alignment, motion semantic alignment and time alignment.
The first step is body alignment: creating a mapping of human-robot interactions.
Depending on the physical differences between humans and robots, DexRoam uses the human skeleton as a reference to create a coordinate system that maps human movements, stiffness, arm and head movements to the corresponding parts of the robot and re-directs the hand to generate subtle hand directions that preserve the relative motion of the entire body.
The second step is the syntactical alignment of the movements: the expression of the movements of humans and robots.
Only the actual body mapping was completed, and the movements still contained differences such as human body height, arm length and initial gesture. DexRoam further represented the movement of the chassis, dryer and arm as relative movements, reducing the dependence on absolute human gesture, so that the two types of demonstrations had a consistent control meaning.
Step three, time alignment: the rhythm of matching and execution by robots.
Depending on the differences in the speed of execution between humans and robots, DexRoam resampled human trajectories as the mission progressed, while maintaining the original trajectory, matching the time scale of the robot demonstration to provide more consistent chronological monitoring.
Learn how to move from human samples
After three stages of alignment, the human-robot demonstration can be used for VLA strategy training by means of a unified observation-focused interface, without the need to modify the model structure for human data, i.e. learn the synergies between motion and agility from the human demonstration.
Using the two VLA models GR00T N1.7 andπ0.5, the team explored three training strategies: human-powered combined training, human-powered data mitigation after pre-training, and human-powered combined mitigation after pre-training, to verify the effectiveness of this framework under different models and training methods.
Experimental results: DexRoam moves from action to machine learning
Let's really adapt human actions to robots
Figure 3 Visualization of the distributed PCA of manned movements
The team first verified the human-machine alignment effect of DexRoam. The PCA visualization results in Figure 3 show that the distribution of the two types of movements is still significantly different when this weight orientation is completed; after complete alignment, the degree of distribution overlap increases significantly.
Figure 4 Alignment of MMD and SWD indicator differences in the distribution of front and rear manned motion
In quantitative assessments, the MMD and SWD indicators decreased by an average of 41.7% and 31.1%, respectively, and human trajectories after conversion could be successfully recycled on the machine.
These results validate the effectiveness of DexRoam's three-phase alignment approach, which both ensures the executability of human movements on robots and reduces the differences in human-machine motion distribution to provide more consistent motion monitoring for joint strategy learning.
Transforming human experience into machine ability
Figure 5 DexRoam main experimental results, GR00T N1.7 vs.π0.5 success rate and completion rate on five missions
The team further evaluated the strategic learning effect of DexRoam through real-world experiments. As shown in Figure 5, the results showed that in two VLA models and five missions, three training strategies combined with human samples resulted in performance improvements. Among them, "Human Pre-Training + Human Combination Minimization" performed best, with the average success rate of the GR00T N1.7 rising from 29% to 56%, and π0.5 rising from 32% to 57%.
In addition to improving the success rate, the human demonstration also introduced a new way of operating for robots. For example, using only robotic data training strategies to grab mainly from the front, easily overturning them when objects are facing change; after joining the human demonstration, robots can grab to the side depending on the choice of objects. This action only exists in the human demonstration and does not appear in the robot training data, showing directly how the human operating experience expands the patterns of action learned by robots.
Let the limited demonstration of robots have greater value
Figure 6 Contrasts between the effects of learning strategies in combination with human demonstrations in a variety of robot samples
In addition to improving the success rate, DexRoam also demonstrates the potential to reduce the need for robot data. The team gradually reduces robot demonstration training while keeping the number of human samples unchanged.
As shown in Figure 6, the results show that in two VLA models, combined with human demonstrations and halving the amount of robot data, tasks can still be performed in close proximity to the entire robot data base. This means that in the setting of the above experiments, a coordinated human demonstration can replace about 50% of high-cost robot remote data while maintaining close task performance.
Further small-sample experiments have also shown that with the introduction of a small number of robot demonstrations, strategic performance is significantly improved, reflecting the value of human operational experience combined with limited robot data.
Summary: Moving from human demonstration to full-body agility
DexRoam demonstrates the potential of human data in complex whole-body operations: even if it requires the movement of the chassis, the movement of the arms in conjunction with fine-tuned fingers to control continuous synchronization, the human demonstration can transcend physical differences to become a direct robot-learning monitoring of movements.
This result also provides a new direction for robotic data scaling: the value of human data lies not only in the abundance of visual and mission information, but also in the vast amount of fineness and interrelated whole-body experience involved. As these experiences are effectively migrated to robots, the learning of complex mobile maneuvers is expected to gradually break away from the high dependence on high-cost, whole-body new data to open up the possibility of larger-scale robotic skills learning.
Currently, the team has open-source DexRoam collection with aligned code, strategy training code, and support for further related research.