I'm going to tell you something.What's the matter?
According to a report by the robotics forecaster on November 6, recent days, there has been a significant increase in the number of robots in the world.ApplesOfficially releasedHALO(Humanoid Autonomy, Learned in simulation, running Onboard) is a system for controlling the movement of human-like robots powered by a set of data pipelines.
Apple programmatically generated 1,000 virtual residential scenes, in which it synthesized nearly 300 hours of paired video, language, and motion data, totaling 162,698 clips.Apple said:HALOIt's a good idea.Unitree G1Master a range of physical skills, including:Sitting in a chair, navigating close to a designated object, passing sideways from a half-open door, bending over the table.And pushing the boxes.HALO can also coordinate and use these skills to solve problems.It's a complex, long-term task.In the video, when performing individual movements such as sitting, waving, passing, following, the robot does not stop much, the overall movement is fluid. In addition, Halo is also faster in understanding long-distance instructions, and there are no obvious delays and stoppages in the process of executing tasks.The company revealed that it had never seen a real home scene during training.In this case,HALO is possibleZero-sample migration, completed 360 trials in six unconstructed real-world scenarios, with a 91% success rate.01.
Three-tiered architecture.Let the robot understand the instructions and make its own movements.HALO autonomous system pumps contain three layers of structure.
The first layer is the coder (orchestrator), a ready-made visual fluency language model, called every two seconds), which breaks down natural language requests into a series of subtasks, each subtask specifying a skill and its role object. After that, a motion generator running at 3Hz combines the VLM encoder with the DiT (Diffusion Transformer, SpreadTransformer) decoder, mapping RGB simultaneously and the subtasks to the entire motion trajectory (kinematic motion plan), containing a five-second angle of articulation with the pelvic position. The paper notes that the first two layers of this architecture address the semantic understanding of the half (where to sit on the chair?), while the tracker addresses the physical contact and execution of the half (where the chair actually sits on it).
From a video standpoint, HALO demonstrates better generalization capabilities in the face of different scenarios, being able to execute the same commands. Apple says HALO can also respond to free-form natural language requests, organizing these skills into longer multi-step sequences. However, the long-range mission capabilities shown in HALO in the video appear to be a combination of several single actions, and the company needs to release more demos to determine the limits of HALO's capabilities.
Additionally, in the video, Apple demonstrates the ability to respond to HALO requests, but with only one simple task request per request. That is, it is unclear whether HALO can understand multiple tasks at the same time and perform continuous execution.02.
It's a great way to get control of the whole body.The first person to integrate deep informationIn the paper, Apple compared HALO to other remote-controlled humanoid robot systems. The company noted that the uniqueness of HALO is that it can be used by humans.Online connectivity skills that integrate the body's external sensory systems directly into the motion tracker.
First, HALO uses a hierarchical whole-body control architecture at the strategic level.It's time to move on to the next level.HALO will plan the next five seconds of motion while re-planning the strategy at a frequency of 3Hz. HALO will only execute the front end of the motion trajectory until new perceptual feedback is obtained.A high-level orchestratorThis layered structure is extended to translate natural language requests into a series of subtasks that continue forward and forward when the subtasks are successfully executed, and when they fail, HALO actively retry.
In the perceptual low-level control, Halo introduced a perceptual tracker to bridge the power gap between traditional visual sensors and tracker programs.Integrating deep information from the first personIt is used to track all body reference trajectories generated.03.
It's a great way to break down the movement and synthesize the data.Mission success rate above 90%At the Motion Generation stage, the company divides each task into a series of movement segments, with each segment guiding text hints, optional path points, and parameterized key positioning on the terminal that defines the object interaction. In the task shift to the armchair, Apple breaks it down into three segments: forward-facing, forward-facing, forward-facing, forward-facing, and forward-facing. For this segment, the company's path point guides the robot to the front of the seat, and in the final segment, HALO ensures that the key positioning on the terminal that defines the object interaction is fixed to the target asset.
In order to construct a complete motion trajectory for motion segments, Apple sampled the effective position of robots and target assets by calculating navigation paths using a non-holonomic unicycle planner. The company then passed text with geometrical constraints (pathpoints and key axes) to KIMODO (a model of motion propagation that is consistent with sport). KIMODO then generated a compact full-body motion trajectory for each separate segment, eventually splicing it into a single, continuous global motion trajectory.（Scenario staging）At the top, Apple used the Infinigen simulation to generate 1000 rooms of three different types, sizes and layouts: living rooms, offices, and bedrooms. Then, the company changed the candidate's scheme by sampling a two-dimensional special Euclidean Group in 2D, SE (2) to synthesize the trajectory and its target assets into a combined embedded scenario, and chose to accept the first configuration that avoids collisions between the trajectory and the target assets and surrounding furniture.
As a result, Apple notes that the corpus database can be rendered in a new resolution using different cameras, or in a whole different physical simulation, without having to re-generate the underlying motion trajectory.
To narrow the gap between visual simulation and reality, Apple adopted a wide range of domain randomization (domain randomization), enabling simulation scenes to cover different colour temperatures, exposure, contrast, motion blur and sensor noise. Finally, Apple performed first-person egocentric rendering of the results obtained, repositioning the predicted trajectory in its INFINIGEN-generated scenes in a mathematical manner, using simulator fish eye lenses matched to the object, to achieve first-person vision rendering.
As a result, Apple obtained nearly 300 hours of paired video, language, and motion data covering 1,000 different scenarios. To synthesize this set of data, the company invested 250 pieces of H100GPU, and it took 48 hours; the motion generator was trained on 64 pieces of B200GPU, and the motion tracker used 4 pieces of H100GPU.
▲ Apple revealed that single-task evaluation is performed in 100 simulation scenarios and 6 real-world scenarios, with multi-task evaluation using 50 simulation scenarios and 1 real-world scenarios. Each task is tested 10 times in a scene.
At the same time, Apple is also doingI'm not sure what I'm saying.Ablation study）。
▲SIT success rate on 100 reserved simulation scenarios, 10 trials per scene, roughly the HALO system, Δ is a change in success rate relative to the setting.
First, the company changed the number of simulation scenarios used for training from 1 to 100, keeping all other conditions unchanged, and co-developed strategic strategies on all six tasks, finding a dramatic increase in success rates.43% up to81%However, if the number of simulated scenes continues to expand to 1,000, the success rate will increase by an additional 3 percentage points. Second, in the absence of real-time blocks (RTC, real-time chunking), the game plan will bounce off with each switch, causing the success rate to plummet to zero.24%If a smooth transition of the RTC steering gradient is used in the reasoning phase, which promotes strategy continuity but also introduces additional reasoning delays, the success rate is still lower than the training-time conditioning method used in the HALO training phase.18 points.If the flow-matching DiT is replaced with a return decoder using the same Transformer backbone, the success rate drops from 84 percent to 44 percent.19%Finally, if the other conditions remain unchanged, HALO's ability to perceive deep information will be eliminated and its success rate will decrease after only sensory information is left.10 points.04.
The encoder uses a thousand-question model.Generators run asynchronouslyApple also explains how its motion generator training and motion tracker training are trained.
The motion generator uses first-person input of RGB images and language instructions to output a complete motion trajectory over a time interval of H=150 tons. HALO produces 29 joint angles in each tube, and a 3D pelvis translation and a 4D pelvis quaternion are stacked together, both of which are represented by the moment t of the pelvis coordinate reference, so that its output to the entire region does not change.Asymmetric encoder-decoder (sometimes called asymmetric encoder-decoder)It's the codec.Qwen3.5-2B VLM。The decoder is aModels of the Distributed Transformer(diffusion transformer, DiT), which was trained by Apple from scratch: 8 blocks, 512 in width, 8 attention heads, 8 blocks, 512 in width, 8 heads. The decoder is embedded as input with VLM characteristics to generate dense body motion trajectories.Flow matching for trainingApple has also used a 15-step Euler sample on its generators.Asynchronous generation (asynchronous generation)The system calculates the next trajectory while the tracker is executing the current trajectory. In order to prevent the trajectory from switching, Apple did not adopt a real-time block method, instead adopting the following trajectory:Conditioning during trainingIn this way, the robot will confirm the execution of the previous P=15 tonnes (0.5 seconds) as a clean, noisy token directly entering the generator. These tokens occupy the previous P tonnes of the sequence, and the model will only make noise to the remaining H−P=135 tonnes (4.5 seconds).Apple pointed out that because the training goal is to strictly complement the all-given prerequisites, the model thus internalizes the continuity of the strategy at the native level.As a result, the company believes that smooth movement transitions have become an inherent property of the network rather than a costly inference-time patch, allowing the new movement trajectory to be continuously streamlined to lower-level trackers.- training of privileged single-skill experts;
- Distilling them into a unified depth-based student;
- Fine-tuning via reinforcement learning with actuator modeling and domain randomization for robust sim-to-real transfer
05.
Summary: Beyond data, systemic competence is a new challengeHALO has demonstrated to some extent the feasibility of a simulation data engine, demonstrating a scalable training pathway. But at present, 91% of success rates are still based on limited skills and shorter task chains, faced with open environments, complex interactions with longer-range tasks, and the limits of HALO's ability remain to be verified. In other words, as the simulation data engine matches the data collection industry chain, it is expected that data will be gradually supplemented by short boards on the volume chain.