Failed video transcription, interested classmates see the original themselves
01. What is Human Data?
A worker packs clothes into cardboard boxes. What steps are needed to convert this demonstration of operation into robot training data?
In the previous article, we made a basic point: there is no better teacher than a man for a clever robot hand. We introduced a pipeline by collecting human bare-handed operational samples and redirecting them to robot hands in a simulation environment.
From then on, we continued to develop the pipeline, pushing for its deployment in real-world industrial scenarios. This article will share our latest developments, explain why we invest in Human Data, and explore how to transform human demonstrations into the skills robots can use in a factory environment.
Video 1: Human Data collection. Demonstration of human operations to train robots to record data.
02. Why do we need Human Data?
Human Data provides an scalable way for robots to learn from the skills that humans already have in real industrial work. Our goal is to transform the operational experience of skilled workers into reusable data assets to help enterprises train robots to perform industrial tasks.
The potential of Human Data comes from two complementary advantages: first, the scale of cross-task, cross-environmental expansion of data collection; and second, the replication of data already collected between different robotic entities. Currently, we are building a reliable data base to prepare for future data collection and replication on a larger scale, starting with the targeted demonstration of industrial tasks.
Expand the scale of data collection. Demonstrations of human operations can enhance the scale and diversity of training data in the following ways:
Reduce collection costs. Remote operations require a robotic body, control interfaces, and operators who are skilled at using the interfaces. Directly recording the process of human execution can reduce reliance on these resources, allowing the same budget to collect more sample data.
Improved collection efficiency. Direct recording of human operations can boost data collection throughput. Research by people such as Chi shows that in cup-setting missions, data collection throughput in a bare-handed sample is about 6.6 times that of a SpaceMouse remotely operated.
It is easier to collect in real-world environments. The deployment of robots and remote operating systems in industrial sites leads to higher logistical and operational complexity. Camera-based human action collection software is easier to install and operate, making it more suitable for collecting sample data in different locations, workflows and operating conditions.
Expanded cross-robot replication capabilities. Separating human data collection from the redirectional actions of specific robots allows the same sample data to be applied to different robotic hands and hardware configurations. The replicable data processing pipeline reduces the need for data re-collection for each robot's own body. Of course, each robot still needs adaptation and validation, but the original recorded data can be used as a shared data resource.
What kind of human data is good?
We first targeted specific industrial tasks with operational demonstrations aimed at identifying which data are really useful for robot learning, and then gradually expanded the scope of data collection and reuse.
For the industrial task we are concerned with, high-quality Human Data must retain enough information to rebuild the interaction between hands and objects in detail throughout the demonstration process. This is primarily dependent on data collection: camera coverage, timing and time synchronization must support accurate 3D rebuilds, including when a hand or part of an object is obscured.
So we're working around specific tasks that robots need to learn, designing data collection and processing as a whole, and viewing data quality as an end-to-end requirement.
How does RLWRLD use Human Data?
We use an end-to-end Human Data pipeline to translate human skill demonstrations into data sets that can be minimized for robot models. The entire process consists of four stages, each based on the output of a previous stage:
Human Video Capture: is a multi-visual video recording of human operations through synchronized, labeled demonstrations.
3D Dynamics Reconstruction: Using RLHND (read as "Real Hand Reconstruction") to estimate the posture, touch and action of the opponent's body, combined with the results of 3D scanning of the object, to reconstruct the hand-to-object interaction.
High-Precision Robot Action Generation: Re-directs dynamic information that is reconstructed to the target robot to generate robot action trajectories.
4D+ Context Synthesis on 4D+: Visual observations based on Isaac Sim rendering generated to match these actions. We are also studying World Model as a new context synthesis solution. Combined with mission instructions, these observations and actions together form a set of robot model training data sets.
Our goal is to generate a complete data set containing actions, observations, and language that can be used directly to fine-tune robot models while ensuring that robot movements don't just look visual.
Figure 1: Human Data Pipeline. The human demonstration is followed by four phases: video acquisition, 3D dynamic reconstruction, robotic action generation, and context synthesis on 4D+.
Next, we will introduce each of the components of the Human Data pipeline step by step.
Step 1: Collecting human videos
Reliable reconstruction starts with data collection. For each real industrial task, we first build a collection system to ensure that the hand-to-object interaction information needed for reconstruction can be retained.
Our program combines stereo egocentric cameras with exocentric cameras. The number and installation location of external cameras depends on the specific task and site conditions.
Double-eye first view images can provide close-up observation for hand tracking and depth estimation. External first view images can provide additional information when the hand is obscured or moved beyond the first view camera's view, thus improving tracking continuity and rebuilding accuracy.
We use self-developed collection software to synchronize time between first-person and external viewpoints to align images from different viewpoints over time to support multi-point reconstruction.
Step 2: Rebuilding 3D Dynamics
Step 1 consists of demonstration videos that are timed and synchronized. Step 2 requires the conversion of these videos into a 3D recording of the interaction process: how the hands move, where they come into contact with objects, how much force is applied, and how objects respond to it.
To support this rebuilding process, we first performed 3D scans of objects and then used the self-development tool RLHND to rebuild muscle movements.
After completing the object scan, we used RLHND to reconstruct the hands and their interactions with the objects. The RLHND extracted single-eyed perspective images from the two-eyed first-person perspective video, and gradually estimated the gestures and shapes of the hands in 3D under the official camera coordinate system, while predicting the position and impact of each hand on the object.
RLHND architecture
Figure 2: RLHND architecture. Pre-trained Cosmos 3 backbone receives each video through its conditioned shutter interface and adapts it through trainable Patch Embedding and LoRA Adapter. Gesture Branch (Phase 1) Predicts MANO arm parameters and parallelism under the official camera coordinate system.
NVIDIA Cosmos 3 is used as a video encoder.
From a single point of view, tracking the hand can be very different: fingers overlap, objects overlap parts of the hand, and the depth of information needs to be inferred from images. Video diffusion models that have been trained in large-scale data training have learned a wealth of hand movement and object interaction skills.
We used the pre-trained NVIDIA Cosmos 3 model in RLHND as a segmental feature extractor for determination. We did not enter clean video into the noise path below the manually-set zero noise level, but used the conditioned input interface of Cosmos 3 to process clean up the context. This way the input form is consistent with the data form that the encoder is exposed to during pre-training.
We adapted the model with a lightweight LoRA fine-tuning and extracted a time-space feature grid for each 81-ton video clip.
There are two expert branches.
We used these features to build a branch of gesture experts to predict MANO arm parameters, including the angles of joints that are constrained by the anatomical structure of the human body, the shape of the arm that is consistent throughout the video, and the official 3D parallelism. These constraints prevent unreasonable movements such as twisting the joints of the fingers that can only bend.
After freezing the gesture branch, we trained an independent tactile expert branch to predict the likelihood and impact of contact at each vertex on the surface of the hand. Since data with pressure tags are much less than gesture data, separating the two branches can be used to avoid damaging the accuracy of hand gesture estimation while introducing tactile monitoring.
Video 2: RLHND runs at 1x the speed of a single head-mounted camera.
Left: Input video and estimated wrist grid (blue for left hand, red for right hand)
Medium: Predicted area of contact with the wrist, displayed on the surface of the wrist after stretching.
Right: Predicted power distribution (the deeper the color, the greater the power)
Bottom line: Each hand has a total force that changes over time.
When the worker grabbed the bag and pulled it out, contact and force were concentrated mainly on the fingertips; both were reduced to zero when the hand was loosely opened.
The results of the experiment.
On the public benchmark ARCTIC, HOT3D and zero-sample EgoDex, RLHND achieved SOTA accuracy in both hand gesture estimation and exposure to action estimation.
In order to evaluate the model's performance in our own collection environment, we compared RLHND estimates based on a single head-mounted camera with reference measurement results provided by multiple viewpoint collection systems.
The experiment evaluated a total of 201 episodes, containing about 150,000 tons of data on the hands. The RLHND used only one-way video in a head-mounted ZED Mini camera, which was estimated to match the four fixed ZED X double-eyed cameras (a total of eight angles) with the hand triangle.
Specifically, the average difference in position of the wrist is 1.7 cm, and the arm joint movement error (PA-MPJPE) is 0.7 to 0.8 cm. In 96% of hammers, the position difference is controlled by less than 3 cm, and in all hammers it is successful to keep track of the hands.
Since the multi-point reference results themselves have several millimeters of error, the differences reported also contain uncertainty in reference measurements.
On a single H100 GPU, the RLHND processing speed is about 39 FPS; on the A100, it is about 20 FPS. On the H100, its processing speed exceeds the real-time requirements for 30 FPS video.
Step 3: High-precision robot action is generated
Step 2 provides precise 3D records of each stage of the demonstration, including how the hand moves, what it touches, and how objects respond. But these records still describe human-handed movements that robots can't directly execute.
In order to give a specific machine an action, we will reconstruct the interaction of objects (HOI) with the robot's URDF model. URDF defines the robot's connectors, joints and their mobility limitations. We map human movements to this model to generate trajectories that the robot can execute. The quality of these trajectories is close to remote control data.
The accuracy of this phase depends on the quality of the previous few nodes. We do not start generating movements from rough 2D estimates, but directly use precise reconstructed 3D dynamic information to generate robot movements. Therefore, the final robot data obtained is more accurate, even allowing the model to achieve convergence in the process of minimization using only a few hundred episodes.
Through original video, reconstructed scenes, hand movements, predicted contact areas, movement trajectories, and generated robot movements, explore the entire pipeline. Select the video and switch to different data layers to view the components. Click Play to play synchronously and adjust the speed of play for close observation.
Step 4. 4D+ synthesizes the context
In order to build a training data set, we also need visual observations seen by robots, as well as instructions to describe tasks.
In Step 4, we described scenarios and mission text as conditional, synthesizing visual observations that match the actions produced by Step 3. Currently, we use a rendering scheme based on NVIDIA Isaac Sim, while also working on a new way to synthesize the World Model in context.
Eventually, this phase will align motion, observation, and language to form a data set to be used to fine-tune robotic models.
Robot observation in a simulated environment
Our simulation pipeline is divided into three stages: trajectory preparation, observation rendering, and alignment of action labels.
First, we convert the tracked motion of the hand into the robot's coordinate system and enhance the data on the position, direction, initial posture and speed of the trajectory. To maintain the relative geometry between the hand and the object, we make a total transformation of the two while filtering out the episodes of tracking or redirecting wrong movements.
Next, we rebuilt the workspace using 3D Gaussian Splatting and relayed the over-directed trajectory into the Isaac Sim. The labelled virtual twin-eyed camera collects observations from the perspective of the target robot.
Finally, we adjusted the delayed action tag based on real robot controls and exported Episodes in LeRobot format for modeling tweaks.
Bottleneck: the rebuilding of object movement
Reliable 3D object positions are estimated to be a bottleneck throughout the pipeline. For transformative objects such as clothing, the problem is more complex: reproducing the interaction process of such objects requires not only capturing position and orientation, but also recording changes in their shape.
This prompted us to explore a new direction of research: using the World Model to generate scene responses based on robotic motion, rather than explicitly reconstructing each object.
New scheme: World Model guided by robotic rendering
Our previous workshop, Robot-Factored World Models via Robot Rendering, laid the foundation for this project.
This method generates controlled video using three input conditions:
Robot Rendering: Specifies the shape and movement of a robot.
Static Scene Reference: Provides visual information about objects and their background.
Text Prompt: Describe the tasks the robot is performing.
For human demonstrations, we first rendered redirected robot movements in Isaac Sim, then removed humans from the first sample image through Inpainting to generate scene references. World Model subsequently generated interactions between robots and scenes based on these conditions.
Figure 3: Contextual synthesis based on the World Model.
First, human movements are redirected to robots and rendered into video. Then, the first sample image is Inpainted, removing humans from it, and then the image is reproduced into static scene video.
The two videos were coded as conditional inputs to Cosmos 3. The model de-noises the video token through Joint Attention, which ultimately decodes the robot-operated video.
From WAN to NVIDIA Cosmos 3
In the initial work, we used robot video data from DROID and RoboCasa to adapt to WAN via Low-Rank Adaptation (LoRA).
Scenario references, robot rendering and video generation constitute separate token sequences, respectively. The corresponding location-sharing space and time RoPE (Rotary Position Embedding) and the connection between the different sequences through Joint Attention allow the model to understand the location and motion of the rendered robot and the content of the scene being generated.
On this basis, we further compared Wan 2.1-14B, Cosmos3-Nano and Cosmos3-Super.
All three models use the same DROID and RoboCasa data to fine-tune and evaluate on the same video clips, while keeping the scene reference and robot rendering input consistent.
Video 4: WAN and Cosmos 3 perform on the DROID mission.
In contrast to real reference video and model-generated video sequences, which cover tasks such as drawer opening, object grabbing and pencil release, demonstrating the differences in the effects of object motion generated by robots during interaction with scenes.
We used PSNR, SSIM, and LPIPS to measure visual reconstruction quality. For DROID data sets, we also used the Motion IoU indicator adapted from Physics-IQ to evaluate the degree to which the video generation matches the reference video in motion position and time.
In most ratings, NVIDIA Cosmos 3 outperforms WAN.
In the DROID data set, Cosmos3-Super has the highest PSNR and Motion IoU, as well as the lowest LPIPS, and the highest SSIM for WAN.
In the RoboCasa data set, Cosmos3-Nano scored best on all three reconstruction indicators.
Table 1: Results of visual reconstruction and movement alignment on DROID and RoboCasa.
The same DROID and RoboCasa data are minimized to Wan2.1-14B, Cosmos3-Nano and Cosmos3-Super, and evaluated using the same video clips as well as matched scene references and robot rendering inputs.
Subsequently, we applied Cosmos3-Super to retargeted human samples to explore the potential for this program in the Human Data pipeline.
The following examples demonstrate two notable capabilities: the ability to synthesize mutable object interactions in the presence of non-obvious heavy building blocks, and the ability to generate observations that are suitable for different robot shapes without additional training for different robot bodies.
1. Synthetic convertible object interaction
In the case of clothing packaging, robot rendering specifies hand movement, while World Model is responsible for the appearance and movement of the fabric.
These qualitative results demonstrate the potential of the scheme: synthetic visual observations involving variable objects without the explicit need for object-level 3D scanning or detailed material modeling. This solves a key problem in the simulation-based data processing pipeline.
Video 5: Synthetic clothing packing interacts.
Robot rendering guides ALLEX's hand movement, while Cosmos3-Super synthesizes the appearance and movement of the fabric without the need for explicit object-level 3D scanning or detailed material modeling.
2. Expanded to different robot bodies
We'll use the same model in a demonstration of ALLEX and RB-Y1 redirects without additional training for the two robots.
For the RB-Y1, we tested three different hands-on configurations: XHand1, Wuji2, and Tesollo, and used the corresponding robot rendering results as the generating conditions respectively.
Clothing packaging and carton transfer cases show the potential for replication between different robotic platforms and handset configurations. In the next step, we will evaluate whether these generated visual observations can further improve the effectiveness of strategic learning and real-world mission performance.
Video 6: ALLEX synthetic cardboard transfer interactions.
Cosmos3-Super generates carton transfer sequences based on ALLEX movements that are retargeted, with no additional training for ALLEX.
Video 7: Synthesis of the above context across different robot arm configurations.
The same World Model is based on different robot configuration rendering results, interacting with RB-Y1 production boxes with XHand1, Wuji2 and Tesollo, respectively.
05. Summary and plan for next steps
Our Human Data pipeline is capable of converting skilled human operational models into robot training data for multiple robotic body and hand configurations.
We retain 3D information during the data collection phase and continue to maintain this accuracy through subsequent rebuilds, movements, and contextual synthesis. As an advanced World Model, Cosmos 3 has helped us achieve more precise hand tracking and synthesize more realistic robotic observations. As these models continue to evolve, we plan to integrate their capabilities into the pipeline to produce higher-precision training data on a larger scale.
Next, we will focus on improving the efficiency of the entire pipeline and the practicality of industrial deployment. While maintaining the accuracy required for reliable operation, we will further optimize the way sample data is collected and reduce the computational costs of converting human samples into robot training data.
As this work continues, we will continue to share technological progress, engineering experience and experimental results.
References
Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots
Chi, C., Xu, Z., Pan, C., Cousineau, E., Burchfiel, B., Feng, S., Tedrake, R., & Song, S.（2024）。arXiv:2402.10329。
RLHND: Video Foundation Models as Physically Grounded Hand Trackers for Robot Learning
Moon, S., Jeon, S., Kim, S., et al.（2026）。arXiv:2610.09455。
Robot-Factored World Models via Robot Rendering
Kim, B., Kim, T., Cha, H., & Joo, H.（2026）。arXiv:2607.22535。
Wan: Open and Advanced Large-Scale Video Generative Models
Team Wan, Wang, A., Ai, B., et al.（2025）。arXiv:2503.20314。
Cosmos 3: Omnimodal World Models for Physical AI
NVIDIA（2026）。arXiv:2606.02800。
