This is the first in a series of posts on the topic.
The paper:
Hello, I'm Qiayuan from UC Berkeley. Today I'm introducing our work to SuperMimic.
In comparison to the most advanced methods of the time, we reduced the number of GPU hours needed to train to three levels. Every time I say this, everyone will hear three times as much noise, even my instructors are no exception.It's a thousand times, not three times.
Many of the amazing robot videos you've seen are done by tracking movements. However, most of them use specialist policy, which is a special strategy for training individual movements.
As the training scale expands, action tracking is expected to be a pathway to training general Humanoid underlying controllers.
There are three issues that need to be addressed.
The first is data. The computer vision community has solved this problem.
The second is the need for a training method that can be applied to all kinds of movements.
The third is how to solve the problem of arithmetic.
One approach is to train experts for different actions and then distill them into one strategy; the other is to use large GPU clusters to directly train a general strategy.
Why is this so hard? Humanoid is a complex system that operates at a large and uneven data distribution, and we want the strategy to be as efficient as possible and eventually deployed to real robots.
A large number of different human movements are shown here. They vary greatly, but the bodies of those movements are unchanged, and they share the same dynamics.
So our idea is: can we learn the dynamics of sharing with a model and hope that some of the good qualities of this model will help us scale up our training?
This is not a new idea. Many model-based approaches have already been explored in this direction. These approaches need to decide where to calculate, where to reverse propagate, and how to update strategies. The last line is real robots, which also involve their own uncertainty and partial observability.
Both forward and reverse calculations of SuperMimic are performed in a learned world model, using the SHAC update strategy, and can run on the real Unitree G1.
What you see now is not a simulation, but a world model running in a browser. We can drag a part of a robot's body, and a general motion tracking strategy will try to get the robot back in balance.
But how do we know that this world model isn't just about remembering strategic behavior?
If I shut down the strategy, the world model would predict that the robot would fall to the ground. Even if it had never trained with zero-action data, it could still predict that.
So how do we train world models and action tracking strategies?
First, let the strategy run and collect data in the simulator; then, use the data to fit a world model; then, let the strategy perform a visualization in the world model to improve the strategy by reverse propagation across the world model.
The whole idea is simple, but very efficient. Why is it so effective?
First, a world model can be replicated between all the movements. As the movement data increases, the model's predictive errors remain essentially stable.
Second, the imaginary rollout in world models is ten times faster than the rollout in simulators.
Third, a ladder is very effective. Even with the same world model, PPO requires more steps to achieve the same return.
As a result, SuperMimic can train on a GPU in three hours, while SONIC can train on more than 100 GPUs in a week.
In fairness, the data sets we use are smaller and the GPUs are faster. But that's not enough to explain a thousand times the gap.
Another question I'm often asked is: why should I care about this outcome if I come from a GPU company, or a big tech company that doesn't lack computing skills?
There are three reasons.
First, the returns were higher. SuperMimic surpassed SONIC in less than three hours.
Second, fewer failures. We often talk about a 99.9% success rate, but on a large data set, this still means more than 100 failures. After three hours of SuperMimic training, less than 30 failed moves. If we continue training, we can achieve zero failures on more than 100,000 moves.
Thirdly, faster training means faster conversion.
Moreover, even if this general action tracking strategy is trained entirely in the fantasy world model, it can still migrate to real robots. All the videos shown here come from the same general action tracking strategy.
When we didn't publish the code, someone told me that we gave too much detail: one student completed the replication in just two and a half hours.
Let's see what everyone has done.
Some consider it to be a fast-paced chatting challenge, similar to Karpathy's nanochat, which takes less than three minutes to train on the PH 1 dataset. The dataset contains about two hours of motion data.
You can also use SuperMimic to access other methods, search for parameters, or even save teacher-student training.
Because of the high computational efficiency of this method, you can apply it to a new body, or even a human model with 700 muscles.
What surprised me was that someone removed the reference action and found that it could still be used for operations under scarce rewards.
We have shown that this is possible, and other researchers have pushed it further.
I'm going to share some of my work in the lab.
We can make a full-sized humanoid with a hand-held moving object. We can also make a strategy and a model handle a hundred different objects.
We can replace strategies relatively easily, because there is no need to calculate the likelihood of a strategy.
We can also write security problems into constrained RL problems and handle them with high-level optimization.
For robots without visual input, the model can also be used to estimate the robot's own state and terrain. The world model can serve as a rapid, microscopic forerunner to dynamics.
Finally, someone is building a new project based on SuperMimic.
Back to the first three questions: we have solved the problem of arithmetic.
Another thing: the code is now open-source, and the complete submission history is included.
You can also see how I wrote code when I didn't use a coding agent last year.
Thank you very much.
Questions and Answers
The question is:What are the advantages of using a micro-world model over using a micro-simulator directly?
Qiayuan：Micro-simulators require high computational costs, especially when reverse propagating. In contrast, the world model is just a simple MLP that can be reverse propagated more easily.
The question is:If you were rich and had a thousand real Humanoids, do you think you could learn a model of the micro-world directly in the real world, or, first, learn a model in a simulation and then fine-tune it with real data?
Qiayuan：If you really have a thousand Humanoids and a thousand motion capture systems to access privileged information, then you can try to apply this to the real world.
The question is:Now you have a strategy that can track many movements. But in a new deployment scenario, the robot may encounter an out-of-distribution state. For example, it enters an unprecedented physical configuration in the process of operation, and the original strategy may lose its balance.
Qiayuan：Many generalizations come from the actor's design and network architecture. SuperMimic does not conflict with these methods.
For example, some researchers have combined Rubber's general motion tracking method with SuperMimic. You can add methods to the SuperMimic framework to improve Rubber's versatility or generalization.
