With the continuous learning process of understanding space and embedding basic models, can robots better cope with changes in reality?
When the robot reached for the fruit and vegetables, someone took the target away. The object that had been left in place was thrown back onto the table. In another group of experiments, the table continued to rise, and the claws needed to keep up with the altitude of the target.
Figure 1 SLA real machine operation video
The Hong Kong University of Science and Technology (HKU) and Ola Dimensions (OLA Dimensions) have proposed the SLA (Spatial-Language-Action Model) to address this issue.
In the RoboDojo-Real public virtual machine assessment, SLA-Preview topped the list with an average score of 42.3 points and a 33.9% success rate for full missions; compared to π0.5, the success rate improved by 21.1 percentage points, about 2.6 times that.
Figure 2 RoboDojo-Real Real Machine Chart, SLA-Preview ranked first by average scores
In eight real-time machine tests set up by the research team, the average object-level success rate of SLA was 83.8%, up 24.0 percentage points from π0.5. Tests ranging from open lists to dynamic crawling, desktop upgrades, SLA demonstrated its ability to cope with real-world scenario changes.
The core idea of SLA is to simultaneously develop spatial representations of supporting movements in robotic learning. Spatial supervision extends from the post-training phase of specific tasks to larger-scale continuous pre-training and continues to be maintained in downstream adaptation.
01 After the top of the list
See how robots cope with change.
RoboDojo-Real covers the three ARX X5, Piper and Piper X robot cores, for a total of 18 missions. SLA combines the demonstration of the three cores for post-training and testing with the same model.
In the same public assessment, an average score ofπ0.5 and a success rate of 22.9 and 12.8 points respectively, and an SLA of 42.3 and 33.9% respectively. The score reflects the progress of some tasks, while the success rate measures whether the entire task is completed.
The results of the list give a unified reference. Intuitively see the role of spatial capabilities, but also see the movement of targets and the upgrading of the desktop.
In fruit and vegetable grabbing experiments, the researchers first trained with 100 basic scenario samples and then applied the same strategy to the three basic scenarios, location disruption and dynamic interference. During the test, objects may be thrown back onto the table and the grabbing targets may be moved by humans.
There is no additional collection of corresponding interference demonstrations for post-training with position and dynamic interference conditions.
Figure 3 Basic fruit and vegetable operations, location disruption, dynamic interference and small sample cross-body adaptation.
The object-level success rate of SLA is 82%,π0.5 is 52%; 76% and 42% respectively when the target continues to move. Both are 78% and 44% respectively.
The cross-body experiment also had a clear data condition: in addition to the original 100 Piper samples, with the addition of 10 Franka samples combined after training, Franka had an object-level success rate of 78%.
Eight classes of deployment conditions evaluate 50 instances of objects that successfully require the completion of the entire operation process of the object. They all point to a real question: can the strategy be adjusted accordingly when the observed spatial relationships change?
02 The understanding of space
How to get to the foundation model of robots
VLA connects images, instructions, and movements to make use of pre-trained knowledge gained by visual language models. But robots perform operations that also deal with the relationship between distance, direction, object shading and claws movement.
The ability to identify a cup is not exactly the same as the ability to move the claw directly to the side of the cup. The position of the object changes and the approach path may change; the angle of grasping changes and the wrist direction also needs to be adjusted accordingly.
A common route is to introduce geometric information such as depth charts, point clouds, and so on during post-training. However, real data does not always have a consistent three-dimensional marking: depth may be missing, camera position and marking state vary, and some public demonstrations are only RGB images.
If spatial learning relies solely on integrated, complete geometric inputs, it will be difficult to make full use of the already accumulated experience of heterogeneous robots.
SLA chooses to align spatial information with the VLA's own intermediate visual representation during training. It inherits the π0.5 pre-training architecture and weight, continuing to use spatial supervision through continuous pre-training learning spatial representation and adapting to specific tasks after training.
Figure 4 SLA overall framework. Unified heterogeneous robot data for spatial alignment during continuous pre- and post-training stages.
Specifically, the frozen three-dimensional perception model OmniVGGT serves as a spatial instructor. It reads multicolored RGB images and uses existing depth or camera-marked information in the sample to extract spatial features.
While learning to predict movements, VLA also aligns the intermediate visual characteristics with the spatial characteristics of the teacher. Spatial supervision thus enters the characteristics on which the movement generates dependence, allowing the model to obtain geometric information during learning operations.
In the face of different scenarios, textures and camera perspectives, SLA will also bring together multiple levels of visual characteristics based on input customization. In RoboTwin fusion, the multi-layer customization Clean2Random success rate was 72.3%, 5.0 percentage points higher than the best single-layer fixed solution.
Another design was to discard the status of the training period: with a certain probability of removing some of the sample's robot state inputs, encouraging the strategy to make more use of the current image, and reducing reliance on fixed state patterns in the demonstration trajectory.
At the deployment stage, the space instructor, alignment projector, and layer door controls were removed, with depth and camera marking not needed as a new strategy input. Space alignment branches did not add to strategy reasoning calculations. The RoboDojo longtime task also used historical image memory, whose listing results reflected the complete system performance of the mechanism.
03 6th class of robot data
Physical significance of the unified movement
Space surveillance is going to work on a larger scale, but it also needs to answer a basic question: How do we put data from different robots together to learn?
SLA uses six sources: InternData-A1, AgiBotWorld-Beta, RoboCasa 365-Pretrain, RoboCOIN, DROID and MolmoAct, covering real and simulation, single-arms and arms. Prior to screening and processing, these sources combined approximately 171,000 tracks and 14,330 hours of demonstration.
Figure 5 Continuous pre-training data sources. Figure 5 shows the scale before screening and processing and the number of tasks combined by statistics for each data set.
In addition to the size of the data, control representations must also be consistent. The number and arrangement of joints in different mechanical arms differs, and the number of joints in the same group does not have the same physical meaning.
The team used the end-user position to describe the state and movement, and unified the robot's coordinates: one arm at the base of the arm, two arms at the center of the base of the two arms, and the coordinate axis pointed forward, left, and upwards. Thus, the direction of movement, rotation, and claw movements have comparable physical meanings.
The rotation indicates the same processing. Directly decreased eura angles may result in a cluster value at an angle around the boundary, and SLA combines SO (3) to obtain a relative rotation, represented by a rotational vector, to be combined with a path-by-path homogeneity.
For high repetition simulation tracks, the team samples by source and mission and filters out inappropriate or abnormal data. Continuous pre-training stages also do not use post-training data sets to reduce the risk of flow evaluation leakage.
This phase uses 48 NVIDIA H100s and takes about 110 hours. Data processing and spatial supervision together form the basis of continuous learning.
04 The data continues to increase
Will space capacity grow together?
If spatial understanding is to become a basic modeling capability, it also needs to be subjected to data-scale experiments.
The researchers used five percent, 25 percent, 50 percent, and 100 percent continuous pre-training data from a complete mixed data set by task sampling, and controlled whether continuous pre-training and post-training used spatial alignment, respectively, and compared performance in the RoboTwin Clean2Random setting.
When spatial alignment was used in both phases, the success rate increased from 49.2% to 60.5% and 67.4% respectively, eventually reaching 72.3% respectively.
Figure 6 Continuous pre-trained data-scale experiments. Both phases use spatially aligned schemes that perform best under different data sizes.
In total data, the success rate of using space alignment was 63.2% for continuous pre-training only; 57.1% for post-training only; 48.2% for both stages.
This control group supports SLA's technical claim that spatial representations can first be learned in a broader robotic experience and then reinforced by downstream adaptation. Within this experiment, training goals and data coverage co-influenced the final performance.
Other public assessments further tested different capabilities. In the 50 tasks of RoboTwin 2.0, SLA used 50 clean samples combined after training, Clean2Clean, Clean2Random success rates were 86.0% and 72.3% respectively, with an average of 79.2% respectively.
In the 24 kitchen tasks of RoboCasa-Panda, each task used 50 downstream samples, with an average SLA success rate of 68.4%. In LIBERO-Plus, the model received disruption tests, such as camera, language, lighting, background, after training with standard LIBERO samples, with a weighted success rate of 87.4%.
05 From Learning to Action
Adapting to Change
The SLA study, carried out by the Hong Kong University of Science and Technology (HKU) and the Eurovision team, extends space supervision to the ongoing pre-training phase of the foundation model and tests its effects through lists, melting experiments and real-machine operations.
Existing results are still applicable: there are significant differences between different missions and entities, and long-range operations, open missions, and larger real deployments need to continue to be verified. SLA demonstrates a pathway to continue to cultivate space characterization based on inherited VLA capabilities.
Papers and projects
SLA: Scaling Spatial-Language-Action Model through Continual Pre-training
This is the first time I've seen this.
Project homepage and dynamic experiments:https://embodiedsla.github.io/
The code repository:https://github.com/OpenHelix-Team/Spatial-Forcing
The following is a list of RoboDojo machines:
