For the first time ever, human video offers something other than action.
Back from the action to the world tour
At CoRL 2026, NVIDIA officially released the GR00T 2. Compared to the GR00T N1, the world's first open-source and fully customizable human-like robot base model, which debuted at the GTC in March 2025, the GR00T 2 has completed a target-level upgrade:
From a VLA with action output at its core, to a world-action model for predicting the future of video and action from multiple perspectives.
Video screens are no longer just inputs, they are monitoring signals; human videos are thus able to provide information outside of action labels for the first time.
Show the model a video of a person doing a chemical experiment and it can generate an image of "the robot does it itself" and then take action.
▲Nvidia's official tweet:
Bet on Predicting the Future: Video and action in the same sequence
Dividing into two models can also be done, at the expense of the action model not seeing the unproduced future, only reading the already determined images, and predicting the constraints on the action is broken.
In the same sequence, the action token can simultaneously read the evolving latent video while making noise, and the two are bound to each other.
A 2B transformer, video and action sequences
The GR00T from 1.0 to 1.7 has been coding on the same path: stronger skeletons, larger movement heads, synthetic data-filled scenarios. The position of human video on this path is therefore fixed: it provides VLM with world-filled knowledge to help models understand scenarios and missions.
This is what GR00T 2 has changed: the next few images are written into the output, the video itself becomes a surveillance signal, and human video for the first time can provide something outside of action.
In the concrete implementation:
The baseline is the Cosmos-Predict 2.5-2B video scattering transformer, which can be directly inherited from the video predecessor.
The vision is compressed using frozen Wan 2.1 VAE (time 4×, space 8×); at 224×320 input, each path camera is cut into 280 tokens per sub, with a learning perspective embedding to distinguish which path camera;
Read instructions through cross-attention with the frozen Cosmos-Reason1 on the language side;
The state and noise action blocks are projected into the width of the transformer by a small MLP belonging to each of the native bodies, and then activated by a dedicated decoder.
Training with flow matching, video and movements is a loss; when deployed, video noise is 1 ton 2 steps, 2 ton 4 steps is enough, reasoning the budget is mainly spent on the movements, not on the screen.
▲Graph 1 GR00T 2 Overview: Pre-training with human and robot data, inputting language targets, multi-visual video and robot state, jointly predicting multi-visual future video and action; the following are four X-Adapt paths, RL post-training, Agent drive, and GR00T 2 Turbo commanding SONIC to move its entire body.
Why "Imagination" can be a translator
It is difficult to transcend the body, not because one of the parameters is not aligned, but because: the camera's internal and external input is different, the kinetics are different, the semantics of the motion are different.The GR00T 2's thought process is to calculate the geometry of the input and output: re-projecting the wrist view to align the field of view and direction, re-expressing the movement into the target robot's motion space, and adding the remaining appearance to the "how to do this" by joint prediction.
The model predicts the future of the human hand and the blue UMI fingers when the same toy self-assembled car is trained with only the UMI claw, and the image becomes the YAM's own mechanical arm after joining the task-related YAM data synchronization training.
▲Graphics of the future GR00T2 generated by the YAM robot versus the deep blue body of the YAM robot.
So what needs to be completed across the body is not a correspondence table between the movements, but the model's forerunner to the way the target body looks and moves.
This is also the most fundamental difference between a world-action model and a pure VLA: the difference is not just in the output of a few frames, but in how the robot can grow into an object that can be trained to do this.
How to teach new tasks without remote operation by target robots
If the target robot doesn't give a remote operation, where else can the demonstration come from? Several paths shown in GR00T 2 revolve around this problem.
▲It doesn't rely on remote learning to learn new tasks. Each row is a demonstration of the "no-target robot demonstration" and gives a demonstration of the GR00T 2's performance on the robot and the real machine's success rate.
Four collecting paths + one simulation path
Moving handheld programs allow people's hands to be mapped directly to their hands
Operators wear head-mounted cameras, wrist-mounted cameras and Manus gloves to pick up movements, and the original video is unchanged and redirected offline, and the movements are mapped to the seven-degree mechanical arm with the Sharpa 22 high-density finger point touch array.
As the target hardware for the whole set of experiments, Sharpa's ingenious five fingers were embedded in a microscopic camera with more than 1,000 touch pixels per finger; the size was close to the human hand, re-orientation was not required; it could also measure sliding, force changes and contact positions in addition to pressure, and the hardware had contact locks.
In practical operation, Sharpa's agile hand operates in a fixed rigidity mode, not enabling the ACP adaptive smooth control on XHAND, the model is based on a 62-dimensional action target consisting of a two-arm position and the angle of the wrist joint; at the same time, the entire Tianji + Sharpa platform is also the carrier for evaluating all agile tasks on the Dexter-50 benchmark.
DexUMI's wearable outer skeleton is strongly targeted for agility.
The hardware originally replicated the XHAND1 Pro-U's 16 freestanding structure and contact surface, and could simultaneously record the three-dimensional torque of the finger tips and the joint torque; in the same-body scenario, the joint angle could be sent directly as a robot instruction, and the force signal could also be transmitted to the model.
The UMI claw is a lightweight collection route for the claw system.
The device itself is equipped with a robotic-pattern wrist fish-eye camera, claw width encoder and IMU, which directly generates the operational data of the aligned robot coordinate system when demonstrating the device in human hands, and can only be trained for subsequent field of vision correction.
A single human video simulated
All it takes is an ordinary human-operated video to reconstruct human hand, object, and interaction trajectories through Video-to-Data, and then generate massive simulation samples in Isaac Lab for modeling.
It's not enough to use just one.
This is why we are here.Demonstration data provides task supervision, mission-free exploration data of the target object provides body supervision; 1:1 mixing, so that the model can simultaneously acquire mission semantics and the appearance of the target object.
In addition to the appearance and mobility advantages of the model side, the engineering cost of replacing the main body is also dependent on the difficulty of integration on the hardware side. Sharpa Wave uses a direct-drive transmission architecture, with 22 degrees of freedom all integrated in the palm of the hand, without tying specific small arms, and can be flexibly assembled on different mechanical arm platforms.
Human video as a surveillance signal
Human video has neither an action tag nor the appearance of a target robot, and it cannot be used under a purely VLA framework.
The GR00T 2 was able to use it because the pre-training objective contained video predictions: a first-person video, even if it didn't know the corresponding movements, could monitor what the image would look like.
The same architecture, the same post-training data, only changing the pre-training ratio of the Human-1M (0%,1%,10%,100%), 72 Dexter missions have a real-time success rate of 18% → 41%, video projection success rate of 30% → 51%, and the largest one has not yet been matched.
▲The impact of human video on pre-training: 18 percent to 41 percent success rate, 30 percent to 51 percent success rate, both of which have not been compiled on the largest scale.
Human video can be used as a surveillance signal, and the core mechanism is GR00T 2.Joint video-action prediction goalsA first-person shooter video, even without an action tag, can monitor how the image should look.
But video predictions are not enough, and pre-training is mixed at the same time.Human-1M Human video (40.27 million hours) 、UMI-10K handheld claw demonstration (1020,000 hours, covering 2,897 missions)and2.35 million hours of remote dataThe latter two provide robot action supervision.
After the pre-training, two patches were added.
Contact intensive missions, where the ceiling of the SFT is
Fixed tubing, finger placement, cable insertion, charger insertion are all four tasks, and the SFT baseline is only 40 to 76 percent. This type of error cannot be corrected by simulating learning, because the distribution of successful samples in the demonstration is narrow, and the failed state has never been corrected.
Therefore, researching GRPO running GR00T 2 directly on a real machine, replacing the determination action with noise in a CPS sample to generate exploratory noise, and then using successful trajectories to simulate itself.
▲The results of four real-machine intensive-contact missions, the error line of 50 trials, are 95% of Wilson's range.
And also, this set.The reinforcement learning program that dramatically improves the success rate of real-world machine operation, whose key configuration was not derived by manual modeling by human engineers, but by an automated research agent who spent about 35 days running around in a simulated environment.。
Agent when System 2
The point of failure of long-range assembly is usually not to move, the piece is dropped in the middle, stuck, displaced, no one is noticed or recovered, the strategy only produces movements.
GR00T 2 is divided into: Agents responsible for what they want to do: dismantle tasks, write code, monitor, restore; and GR00T 2 responsible for how to do it.
The real-machine stress test is the best way to see the division value: three long-range assemblies, in which the parts are randomly dropped on the table before each start, a disorder that has never been seen in post-training, with a single test of 5x14 minutes. The SFT on Hot Wheels is only 23/100; plus DAgger mentions 84/71/78; then Agent returns control after repairing the parts and cartons, becoming 98/97/97.
In the re-plug-in task, the same agentic framework replaces the SFT with the RL-trained GR00T 2, cable from 12/20 to 19/20, charger from 2/20 to 20/20.
▲Graphically, visual guidance has led to higher average success rates on both levels of assessment: round points for grabbing targets, forks for releasing targets, and color labels for executing arms.
Assessment: Not to be confused with converting aluminum to aluminum
Why not report absolute success rates?
The success rate of a real machine varies greatly depending on the location, the operator, the hardware, and the state of the day, so NVIDIA built the GearArena: A/B blind test.
The same scene, the same operator, ran 2 to 4 checkpoints in a random sequence in a matter of minutes, and the drift was shared and offset within the pair.
From July to November, 10 YAM jobs, more than 20 operators, a total of 195 rounds of campaigns, 16,044 scoring laps, 11,767 double-digit comparisons, and 71 checkpoints were synthesized using Bradley-Terry's Elo, which relied heavily on sampling "tasks" rather than "experiments".
The results of the experiment showed that the GR00T 2 was higher than π0.5 at 236 Elo, 95% CI 171x325, which is about 80% victory rate, and the GR00T 1.7 at 95 Elo.
▲Instead of saving the date, the column is the best time to refresh at 90% of the bootstrap range.
How much can imitation represent?
Simulation is used as a proxy for evaluation, but it is not omnipotent.
Twenty-five YAM missions, fourteen checkpoints, and Pearson's simulation with the real machine, respectively, were 0.93 mission progress, 0.92 success rate, 0.93; Elo 0.93; and five more general checkpoints for the DROID environment rebuilt (0.98/0.95/0.97).
This is close to the maximum threshold a perfect simulation can achieve under the same number of trials.
During the pre-training, the final candidate will be pressed to three rebooted machines using a simulated pump checkpoint.
▲Here's how to simulate the performance of a real machine: a) mission progress, b) success rate, c) Bradley-Terry Elo
Written at the end
GR00T 2 transformed the pre-training objective of the basic robot model from the back-to-back movements of the mouse into the world-driving gameplay. After this step, human video, which has neither the action tag nor the data of the appearance of the robot, became available for the first time as a surveillance signal; the transcendental body also evolved from the re-orientation of the mouse + alignment + fine tuning mouse into the imagination within the model.
But it also exposes the boundaries of this path: the accuracy of prediction and the content of prediction are two books. Faster lateral reasoning involves giving up visual details, and the cost of giving up details is half the success rate on trivial tasks.
In the evaluation method, the approach of GearArena's A/B+ cumulative Elo blind test may be more valuable than the model itself.
After all, the most lacking in these two years of physical intelligence is not a stronger strategy, but a reliable comparison.
Edited by Dr. Little Monkey
I'm not sure what I'm doing.
For reference to the paper:
GR00T 2: Scaling Robot Foundation Models for Generalization, Dexterity, and Adaptation
