With 5080 trained for 23 hours, I put a dog and a bear's fur into the Yuri G1.
This is probably one of the most exciting headline moves of the past month.If a humanoid robot jumps, will it jump?
From a technical point of view, flickering footsteps, cat-and-mouse, bear-and-mouse are a very representative class of test moves:
They are not slow swing movements, but hip-hop movements, fast weight-shifting, and high-frequency, coordinated dynamic movements of the limbs in a short window.
A strategy, if it can keep up with these moves, indicates that it has crossed a not-too-low threshold in full-body coordination, contact handling and reference action tracking.
This shared record is a complete practice of using the Utrecht G1 to move these dances from a short video to a robot.
The project implementation process for this article is based on the Deep Blue College curriculum for human-robot movement controllers, and a new phase is about to begin.
Reference to action data generated
First, download the dance video of the dog and bear whining and the flashing steps, and then divide the video according to the movements, which should be noted when selecting the video clips to keep the person in the video can continue to see the whole body, the movements complete.
This standard will directly filter out a lot of material. The reason for this is that the accuracy of the entire flow line is determined at the first step. The next step is the gesture recovery algorithm is sensitive to input: only one person can be seen in the image, the whole body is visible, the background is relatively concise, and the light is uniform.
If the video is heavily overshadowed or multiple people crossed, the recovered trajectory will deteriorate significantly, and this error will lead to the final tactical training, which cannot be reversed.
GVHMR: From video to human movement trajectory
Traditional human gesture estimation methods can usually only get a local gesture relative to a camera's coordinate system: it can tell you how high your arm has lifted, but it can't tell you how far your arm has walked or turned in space.
The essence of movements such as flashwalks and wave balls lies in the vast range of movement and rotation of the body.
GVHMR (Global-View Human Motion Recovery, an algorithm from Zhejiang University to restore global human movement from single-dimensional RGB video) has made a central contribution to the restoration of global human movement trajectories in the global coordinate system, including global parallelism and rotation.
It's an output of a PyTorch..ptThe document contains a complete parameter sequence of the SMPLX human body model:
Global translation, global orientation, body posture and hand posture.
These parameters are sufficient to reconstruct a complete human 3D grid at any time.
Environmental configuration
It can be installed in accordance with the official installation documentation, and download weighted files such as yolo to the specified folder.
If 50 series graphics cards are used, there may be problems with the compatibility of the python version, see Issue: Feature Request: Support NVIDIA Blackwell (RTX 50 Series GPUs, etc.) recommendations for installation.
Converting data
Once the environment configuration is complete, the following command can be executed to convert the video to SMPLX motion trajectory:
cd /path/to/GVHMR
python tools/demo/demo.py --video /opt/data/yangge/baishi.mp4 -sIt's the same thing.--videoThe parameters specify the path of the input video,-sThe parameter represents the saved result..ptThe document must document the complete path of the document, as the next step for GMR is to use it.
├── outputs
│ └── demo
│ ├── baishi
│ │ ├── 0_input_video.mp4
│ │ ├── 1_incam.mp4
│ │ ├── 2_global.mp4
│ │ ├── baishi_3_incam_global_horiz.mp4
│ │ ├── hmr4d_results.pt
│ │ └── preprocess
│ │ ├── bbx.pt
│ │ ├── vit_features.pt
│ │ └── vitpose.pt
│ ├── biandanshi
│ │ ├── 0_input_video.mp4
│ │ ├── 1_incam.mp4
│ │ ├── 2_global.mp4
│ │ ├── biandanshi_3_incam_global_horiz.mp4
│ │ ├── hmr4d_results.pt
│ │ └── preprocess
│ │ ├── bbx.pt
│ │ ├── vit_features.pt
│ │ └── vitpose.ptThe output of GVHMR after processing the video is shown above, including the video generation in the middle and the SMPLX data generated in the end.
In practical operations, it is recommended to place the video file in a uniform location under the project directory and to create a separate output directory for each video, facilitating the subsequent management of multiple action data.hmr4d_results.ptFollow-up will serve as input to the GMR.
GMR: Translating human movements to robots
If you take the trajectory of the human body, the next question is: the human body, the robot doesn't.
The human body has more than 200 skeletons and hundreds of degrees of freedom, while the Unitree G1 robot has only 23 or 29 controllable joints.
There are substantial differences in the structure of the bone marrow, the range of joint movement, and the proportion of limb length.
For example, the human shoulder joint is a ballistic joint that can rotate freely on three axes, while the shoulder joint of a robot is typically approximated by three chain rotating joints.
Directly assigning SMPLX's joint angle to robotic joints is not only physically impossible, but can also result in self-collision or joint excess.
This is the problem to be solved by Motion Retargeting.
The core idea behind GMR (General Motion Retargeting) is to select critical body parts that correspond between the human body and the robot (such as the hands, feet, head, and torso) by minimizing the differences in the position of these parts in the Cartesian space and solving the sequence of joint angles of the robot. This process automatically processes bone length differences and joint constraints to ensure that the output robot movements are physically feasible.
Because the next step is using the GVHMR framework, here's how to redirect hydrogen Retargeting from GVHMR to Robot in GMR:
python scripts/gvhmr_to_robot.py --gvhmr_pred_file /opt/zouyu-workspaces/amp/GVHMR/outputs/demo/guziyangge_80s-92s/hmr4d_results.pt --robot unitree_g1 --record_video --save_path ./datasets/guziyangge/tibeigu.npzAmong them:
- `--gvhmr_pred_file` ：上一步 GVHMR 生成的 `hmr4d_results.pt` 文件路径
- `--robot unitree_g1`：目标机器人型号
- `--record_video`：录制重定向过程的可视化视频
- `--save_path`：输出 .pkl 文件的保存路径Once the conversion is successful, the program automatically plays a visualized video of the result of the redirect and can intuitively check whether the robot's movements are visually consistent with the movements of the original human body:
The BeyondMimic framework requires input into the reference action format as a CSV file, so it is necessary to further convert the GMR output.pkl file.
Run the following commands:
python scripts/batch_gmr_pkl_to_csv.py --folder ./datasets/yanggeIt's the same thing.--folderThe parameters specify the directory path of the.pkl file. Each row of the converted CSV file represents a set of data in the format:
root_pos_x, root_pos_y, root_pos_z, root_quat_x, root_quat_y, root_quat_z, root_quat_w, joint_0, joint_1, ..., joint_N The first 7 rows are the general position of the root node (basin) and the quadrilateral direction, and the subsequent rows are the angular values of each node.
This CSV file is a critical data format that is taken up throughout the entire process and will be used in subsequent BeyondMimic training.
Beyond Mimic: Getting out of the gym is not about standing physically
Here, we get just a series of sharp-angled digital shots. The reorientation ensures that it is possible to move, that is, that robots can put their postures out. But dancing is a question of kinematics: balance when supporting a single foot, angular momentum when turning quickly, instant impact when landing, which is not a word in the CSV file.
If you send the string directly to the machine, the first one turns and falls.
For this reason, we chose BeyondMimic (a collaborative open-source human-robot full-body tracking framework between UC Berkeley and Stanford) based on the NVIDIA Isaac Lab simulation platform, which uses the classic PPO (Proximal Policy Optimization) algorithm to train robots to track reference movements with high accuracy in physical simulation.
BeyondMimic's core advantage over earlier motion simulation methods is that it not only handles static or low-dynamic movements, but also delivers high-dynamic skills such as jumping spins, impulses, flip-flops on real hardware, while maintaining industry-leading motion quality.
Environmental configuration
The training side uses the official library of Utrecht, first clone the official library unitree_rl_mjlab;
git@github.com:unitreerobotics/unitree_rl_mjlab.git
You can also install the installation environment according to the documentation settings.
Data processing
BeyondMimic is trained not to read CSV files directly, but instead to convert them into NPZ formats that contain complete physical information.
This conversion process involves not only the conversion of formats, but also the insertion of crankshaft values and the computation of linear dynamics.
python scripts/csv_to_npz.py --input_file /opt/zouyu-workspaces/GMR/datasets/yangge/csv/baishi.csv --output_name yangge/baishi.npz --input_fps 30 --output_fps 50 --robot g1Among them:
- `--input_file`：上一步生成的 CSV 文件路径
- `--input_fps`：输入 CSV 的帧率（通常为 30fps）
- `--output_fps`: 输出数据的帧率
- `--output_name`：输出数据名称
- `--robot`: 指定机器人型号The script executes the following key steps:
First, move from the input torque rate (e.g. 30fps) to the input torque rate (e.g. 50fps, corresponding to the control frequency dt=0.005s, decimation=4);
Then, in Isaac Sim, calculate the position of the world coordinate system for each part of the body, gestural quadratic numbers, line velocity and angular velocity, using positional kinetics;
Finally, all data is packaged into NPZ files and uploaded to the WandB Registry.
The data fields contained in the NPZ file are as follows:
{
'fps': 50, # 帧率
'joint_pos': [T, N_joints], # 关节位置序列
'joint_vel': [T, N_joints], # 关节速度序列
'body_pos_w': [T, N_bodies, 3], # 各身体部位世界坐标位置
'body_quat_w': [T, N_bodies, 4], # 各身体部位世界坐标姿态
'body_lin_vel_w': [T, N_bodies, 3], # 各身体部位线速度
'body_ang_vel_w': [T, N_bodies, 3] # 各身体部位角速度
}
Multi-reference action expansion
The process is already underway, but there is an engineering problem:
Unitree_rl_mjlab and the official library of BeyondMimic whole_body_tracking support only single Motion training.
There are more than twenty dance moves here, and it's time-consuming to train a strategy for each one.
git@github.com:unitreerobotics/unitree_rl_mjlab.git；
So there's a lot of expansion here.MultiMotionCommandsIt's all about support.
For each Motion, a weight is assigned according to the corresponding number of environments. For example, if the weight of a dog, a bear, a flipper, and a wave ball is 1, 1, 2, for a total of 4096 environments, the number of environments assigned is 1024, 1024, 2048.
In this way, one strategy can master all the moves, and difficult moves can get more training resources by weighting.
Training
A total of 31 reference moves were selected, with the most difficult moves, such as dog, bear and wolf balls, having a weight distribution of 2, and the other moves having a weight distribution of 1.
The instructions for training are as follows:
python scripts/train_multi.py Unitree-G1-Tracking-No-State-Estimation-Multi-Priors --agent.logger tensorboard --motion_weights_file /opt/zouyu-workspaces/unitree_rl_mjlab/src/assets/yangge.yaml --video True --video-length 200 --num-envs 13312It's the same thing.--motion_weights_fileSpecifies a yaml file, and the contents of the yaml file are listed, with each element including a reference action file path and allocation weight.
- file: /opt/zouyu-workspaces/unitree_rl_mjlab/src/assets/motions/g1/yangge/cheshuibu.npz
weight: 1
- file: /opt/zouyu-workspaces/unitree_rl_mjlab/src/assets/motions/g1/yangge/xiaolunbi.npz
weight: 1A total of about 20000 rounds were trained, and it took about 23 hours to get down to 5080, with a mean_reward of about 32.
Play
After completing the training, it is possible to view it through the play script, and it is recommended to select viser as the display back end of mjlab.
Dogs and Bears
It's a flash.
The wave is playing
This link unravels the core questions of human-robot movement control: where the action comes from (data and redirection), how to learn (training tracking strategies), how to stand (physical simulation and migration).
These, only really moving hands running over and over again, will allow people in the video to only half the body to cause a leg trajectory of residual torsion or the torsion to re-orient the legs in the video while sliding, in turn understanding the reason for the existence of each step of the design.
Whether or not a dance can be danced ultimately depends on your understanding of the depth of each translations layer of the entire data pipeline, rather than the parameters of a single script.
If you want to run something similar on your own G1 instead of staring at other people's jumps, the robotic maneuver control yoga class provides a more complete starting point.
The course, which uses the Yūshū G1 as a practical carrier, will be taught by researchers at the Southern University of Science and Technology and Ph.D. student instructor Confucius in 11 practical projects ranging from perception/action space, reward function design to AMP, BeyondMimic, full-body operation control and sensory operation control locks.
The third edition opens on 13 November.
Controlling robots in monkey movements: part of a practical project
Deep Blue Academy pupil robot movement control pupil course
Edited by:
I'm not sure what I'm doing.
