Summary
Robot learning is often considered constrained by data scarcity, but the research community has actually accumulated tens of thousands of hours of publicly available demonstration data. Therefore, the key challenge is how to effectively replicate heterogeneous datasets collected by different robotic platforms.
We proposed camera-centered motion representation as a unified interface for robot data. This representation illustrates the relationship between vision and motion while abstracting the kinetic differences inherent in different robotic entities. Based on this idea, we constructed CamX: a unified dataset containing 20,000 hours of open-source robot demonstration data from 37 sources and converted into a unified camera-centered representation.
To take advantage of CamX, we trained the Unified Video-Action Model CamUVA. The model uses camera-centered motion representation and identifies each camera's perspective and motion flow through camera posture. This design enables the model to derive value from inter-body data centers and generalize directly to unseen robot bodies.
CamX and CamUVA together provide the foundation for two systems studies:What factors are importantIt is important to know which design options can achieve effective scaling.What can be achieved?Finally, we want to show through this proof-of-concept study a path to expanding the scale of robot learning: effectively replicating the growing heterogeneity of open-source robot data.
See data samples and strategy rollout
camx-anonymous.github.io
1 Quotes
There is a common saying in the field of robotic learning: there is a lack of data in this field. However, the robotic community has accumulated an astonishing amount of open-source robot data. Today, open-source datasets contain tens of thousands of hours of demonstrations covering different robots, tasks and environments.https://umi-data.github.io/
However, most existing strategy kits, even those with multi-body training, do not make full use of the vast amounts of data from different robotic entities. Therefore, the challenge is more about how to effectively use existing data.
The main obstacle to making better use of existing data is heterogeneity: robot datasets are collected using different bodies, operating spaces, camera configurations, coordinate systems, and sensor models. Works such as Open X-Embodiment have shown the benefits of aggregating data from multiple robotic platforms, but strategies trained on these datasets often require additional fine-tuning to fit the target body and its operating space. This suggests that merely accumulating data sets is not enough: the way data is represented in a unified way is as important as the size of the data.
Figure 1: Unified heterogeneous robot data is represented by camera-centered movements.A rigidly mounted camera on a robot's mobile connector, such as an end-user or head camera, can respond directly to observations related to related movements and tasks without attachment to a specific entity.
Our core intuition is that camera-centered motion represents a natural, intrinsically unrelated interface between perception and movement. As shown in Figure 1, rigidity is installed on a camera on a robot's connecting rod, such as a wrist, elbow or head camera, to maintain a fixed rigid shift between the connecting rod and move with it. Thus, camera motion is able to capture critical information that is closely related to the task.
To validate this idea in large-scale data, we constructed CamX: a unified dataset that collected about 20,000 hours of open robot data from 37 sources, totaling 1.7 million episodes, as shown in Figure 2.
Based on CamX, we developed CamUVA, the Camera-Centric Unified Video-Action Model. It is a video-action model that combines predictive visual observation and camera-centered action. CamUVA is designed to address huge differences in camera configurations and data formats between different robots, including camera numbers and installation locations, image resolution, torque and sequence length.
For this purpose, we use camera positions to identify different perspectives rather than using fixed camera IDs, allowing the model to be generalized to any camera configuration. In order to adapt asymmetrical observation spaces, observations and movements are packaged as sequences of variable lengths rather than being filled into fixed shapes. In addition, CamUVA combines training on video and action targets, supporting both video dynamics modeling, and strategic performance is superior to response models that only use action target training.
CamX and CamUVA allow us to systematically study large-scale training of robot models.What factors are importantDesign decision-making andWhat can be achieved?This helped us identify a set of key design choices that could enhance interpersonal performance, including: representing the relative camera trajectory as motion, using camera posture to encode multiple viewpoints, and combining training videos and movements.
More importantly, CamUVA demonstrates some of the difficulty in studying a single dataset: without the need to fine-tune a specific entity, it achieves performance comparable to a single entity strategy on different entities that have emerged in training; it can also run directly on unprecedented robot configurations, including configurations with different camera angles and claw shapes, demonstrating the ability to generalize samples across entities.
The contributions to this article include:
CamX：A verified dataset containing a total of 20,000 hours of open-source robot demonstration data from 37 sources and using unified camera-centered representation.
CamUVA：A flexible unified video-action model that supports arbitrary camera configuration while providing video dynamics modeling and strategic learning capabilities.
The results of the study:Systems studies design choices when learning from heterogeneous robot data and analyzes factors affecting cross-body migration capabilities.
2 Related work
Transnational data sets and strategies.Open-source robot data now covers single-arms, two-arms and Humanoid platforms, handheld wrist camera platforms, and polymer data sets. The general strategy is to compensate for the underlying differences by: using separate heads or encoders for different underlying objects; using unified, decentralized end-user movements; filling in the action vector; or using underlying cues. These designs may require micro-adjustment or adjustment of input and output mapping when dealing with new robots.
Another type of work seeks shared action interfaces, such as learning potential action from video, learning general action coding from robot action tags, using language, or using the action of an end-user displayed in a reference camera coordinate system.
The latter type of method still uses the end-executor as a controlled object and uses labeled or estimated transformations to map the predictions to the robot coordinate system; CamVLA predicts this transformation without providing camera input. In contrast, CamX rigidly installs the camera on a portable connector rod as a controlled object, allowing the handheld, single-arm, arm and Humanoid demonstrations to share the same trajectory format and unified camera coordinate system settings, see 3.1.
The video-action model.Video prediction can be used in strategic pre-training, combined with visual programming to decode inversion dynamics movements, and synthetic demonstration generation; V-JEPA 2 can be programmed by predicting the underlying spatial dynamics as conditions. Combined video-action models simultaneously generate future video clips and movements, and some recent methods have also been adapted to pre-training video generation models.
Cosmos Policy allocates time slots for different viewpoints and encodes the action as a potential cube; DreamZero flatens the different viewpoints on the space dimension and uses the action encoder and decoder; UVA uses independent scattering heads to decode video and action.
In addition, work has been done to inject the camera's geometry information into the visual Transformer: one way is to use pixel-by-pixel radiation conditions, the other way is to use relative geometry coding in the attentional mechanism; Gamma-World adds an agent axis based on 3D RoPE.
3 Ways
3.1 CamX builds data sets
A unified format for different entities.For the trajectory of the end-executor, we convert the end-executor coordinate system of each data set to its wrist camera coordinate system, using a fixed shift between the two to eliminate the differences in the coordinate system, see Appendix C.1.
For the joint trajectory, we use the robot's URDF to calculate the direction of motion, and the resulting phalanx position will be combined with the phalanx's fixed change to the camera. For the static third-person camera, we use the open camera exterior. For mobile operating robots, we incorporate the underbody motion into the camera trajectory: the camera fixed at the base records the movement of the underbody, while the wrist camera records the movement of the underbody and the mechanical arm simultaneously.
We organized the data into multiple groups of performers, such as the left wrist, right wrist and head. Each group contains a main camera trajectory, a claw-width data stream, and one or more RGB viewpoints, including a binary camera viewpoints.
The rigidly installed viewpoints on a group of performers are stored by their fixed switching relative to the group's main camera. We also record the external parameters of fixed third-party cameras, which are used to identify the viewpoints. These one-time switches resolve the differences in each of the different datasets, allowing the strategy to receive the same camera-centric input and predict the same camera-centric output on single-armed robots, two-armed systems and mobile operating robots.
When deployed, we transformed the predicted relative camera trajectory into the target position of the target camera by switching the labeled camera to the FRAN.
The data is verified.We used the inside of the camera and the camera's transition to the claw to render the claw mesh and verified whether the result of the rendering matched the actual observed claw to verify the correlation between the camera and the claw, as shown in Figure 2a.
By examining each data source and collecting an overlay chart of configurations, we found no labeling errors in some raw metadata, see Appendix C.3.
Figure 2: CamX data collection.We colored the claw grid in each camera view, and the green outline in the diagram represents the result of the coloring to verify the coordinate system of the camera and the change from camera to claw. The tiny gaps from the manufacturing differences between the different robots and the tracking errors in each episode will not affect the relative camera trajectory.
The CamX includes approximately 20,000 hours, 1.7 million episodes, and 1.95 billion tons, including handheld devices, single-armed robots, arm robots, and mobile basement platforms.
3.2 CamUVA model design
The model architecture.CamUVA is a Mixture-of-Transformers model. Each layer contains three branches and a shared attention mechanism on a sequence of tokens packed together.
Of these, the 7 billion parameter understanding branch processes language instructions and the 7 billion parameter video branch processes video sub variables; both branches use Cosmos3-Nano initialization. The 1 billion parameter action branch is initialized from scratch to process camera trajectories and claw width tokens.
Following the Fast-WAM design, clean tokens focus only on language tokens and clean tokens. Therefore, when deployed, the keys and values of these tokens are only calculated once; only the noise-bearing tokens need to make noise every step of the way.
It is also used as a point-of-view identifier.Previous multi-point models identified points of view by spatial splicing or time splicing, as shown in Figures 3a, 3b. This allowed the identity of the point of view to be tied to the chosen layout. Simplex Rotary Agent Encoding allowed different identities to be kept equidistant in rotating-angle space without encoding the geometric information of the camera.
In contrast, we use Camera Pose Encoding (CaPE) for each input token, as shown in Figure 3d. For a token that carries a camera posture, we use four consecutive channels to convert the query into a key.
Figure 3: Identify the point of view.(a、b) through slot position identification; (c) through equal distance position identification, but not containing geometric information; (d) through absolute camera position coding for all tokens, relative changes are made in the process of attentional computation.
The logit of attention between the token from the camera and the token from the camera thus changes, depending only on the relative position between the two.
A video token carries the position of its camera in a conditional string, an action token carries the position of the wrist of the group to which it belongs, and a language token carries the position of the average position of the wrist of the camera. All positions are represented relative to the average initial position of the wrist of the camera, thus eliminating the differences in the origin of the world coordinate system from various data sources; these positions are given by the results of the determination of the camera.
The code itself is unrelated to a flexible sequence packaging.As shown in Figure 4a, we share the same Wan2.2 video VAE, camera trajectory MLP and claw-width MLP between all nests, so we don't have to set exclusive weights for specific nests.
We then applied alternate 3D MRoPE to each data stream and timed the timing of the video sequences and action sequences; the action token and the index were set to zero. Thus, the two viewpoints get the same MRoPE index, distinguished only by CaPE. This design does not restrict the number of camera views, trajectories, and claw-width data streams in architecture.
Finally, we flatten all the input fields into a one-dimensional token sequence and use FlexAttention to create separate attention masks for each sample. In training, each token can only focus on the current or earlier chunk of the same token, as shown in Figure 4b.
Figure 4: Coding and packaging of heterogeneous data.(a) Each model uses an encoder and is shared by all viewpoints; each token receives 3D RoPE, as well as its corresponding camera's CaPE. Therefore, the viewpoint or trajectory is identified by its position, not its slot. (b) A model serves all intrinsic and UVA modes: only the sequence of variations contains which data streams and which of these tokens are provided as known conditions.
Video-action training goals.We trained with five UVA goals: Policy ((P)、Policy Planner ((PP)、Video Generation ((VG)、Forward Dynamics ((FD) and Inverse Dynamics ((ID)), as shown in Figure 7.
For all UVA modes, each chunk consists of 33 noise observations and movements, in which we randomly sample noise levels to add noise to the noise-bearing model in the generating tube, while maintaining the noise-free condition, as shown in Figure 4b.
We used Flow Matching to train two models, and we only calculated losses on the tokens that needed to be generated.
4 Experiments
4.1 What factors are important?
We use a unified training configuration to perform dissolution experiments on various design options, see Appendix B.4. Unless otherwise stated, we use the Cosmos3-Nano-Policy-DROID Initial Understanding Branch and Video Branch to train the movement branches of 1 billion parameters from scratch, and on DROID we use about 4,000 GB300 GPU-hour training models to save a checkpoint every 500 GPU-hours.
We evaluated each checkpoint on all 120 of the RoboLab benchmarks, running 10 episodes per mission, and reported the average and standard deviations of the three best performing checkpoints in each training.
RoboLab's rollout configuration can be found in Appendix D.1.
Figure 5: The position of the camera is a better representation of the movements of interbody consistency.Using the same model, either training on DROID alone (1emb) or co-training on DROID、AgiBot and ABC (3emb) respectively, the success rate for both types of action spaces decreased, but the decrease in the joint space strategy was 2.3 times that of the camera pose strategy.
It's about the moves.We compared the camera posture and joint location movements described in Section 3.1 in three data sets collected remotely in the CamX: DROID, AgiBot-World and ABC-130k.
Joint action uses a 7-dimensional joint space, with all the mechanical arms sharing a single MLP encoder; for the 6-freedom YAM mechanical arm in ABC, its action extends to 7D by adding zero.
Even training on DROIDs alone, replacing camera posture with joint movement can reduce RoboLab's success rate, as shown in Figure 5, even though basic checkpoints use joint movement training.
In the cross-body training, we sampled uniformly from the three data sets and used three times the computation. After adding the other two nodes, the success rate of both motion spaces on DROID decreased, but the joint space strategy decreased by 8.1 percentage points, more than double the decrease of the camera pose strategy by 3.5 percentage points.
For all three datasets, PSNR results from Forward Dynamics also support the use of camera poses. Thus, camera poses provide a shared motion space for different natives, and joint motion needs to be studied for each natives separately.
Find the first one:For strategic learning, camera posture as a transcendental movement is superior to joint space, especially when training on heterogeneous robot data.
The camera position is encoded.We compared the method of perspective recognition described in Section 3.2. Inspired by the sensory feedback between the UMI's palms and claws, we hypothesized that injecting the position code into the camera position would improve the coordination of the arms.
To validate this assumption, we constructed a Shared Grasp task in RoboLab using the two-arm YAM configuration: the two mechanical arms must hold a box from both sides, lift it up, and place it in the red target area.
All strategies use two wrist camera angles as inputs and predict the relative camera trajectory. If position recognition angles are not used, the accuracy of the box positioning in the arms will decrease significantly: in most episodes, the box is exposed to the target contour line after landing, or beyond the contour line, with success rates decreasing by 22 and 19 percentage points, respectively.
Figure 6: Arms coordinated by Shared Grasp.The two mechanical arms carry a single wide box that the claws cannot hold together and place it in a randomly generated red target area; successfully requiring the entire bottom of the box to be within the contour line. In the initial state, the positions of the other episodes are shown in light color.
Discovered 2:Depending on the camera's position position recognition perspective, the same checkpoint can also be used to operate a single to four point of view to improve the coordination of the two arms.
Video-action training goals.We studied how the sampling strategy of the UVA mode affects model performance while keeping the 4,000 GPU hourly budget and batch size unchanged. Surprisingly, direct sampling of five UVA modes with the same probability, compared to training with P alone or PP alone, will damage the performance of P and PP simultaneously, as shown in Figure 7.
Figure 7: UVA model sampling strategy.A Policy sample contains only one set of observations, so its token number is about the same as the other four model samples. Improving the sample ratio of the Policy to match its token number to other models (Token-Matching) allows the success rate of the Policy to be restored; adjusting the sample ratio according to difficulty (Difficulty-Aware) allows the success rate of both the Policy and the Policy Planner to exceed the single-mode training, while keeping the PSNR of Video Generation and Forward Dynamics unchanged.
We found two main reasons.
First, the Policy sample contains only one observation, so the number of tokens is much lower. In the same probability sample, the token ratio of the Policy sample is much lower than in other models, resulting in poor performance. After improving the sample ratio of P to match the number of tokens to other models, the performance of P returns to a level comparable to that of P training alone, while the performance of PP exceeds the results of PP training alone.
Second, we use task difficulty as a sampling inspiration: P and PP require task-related planning, and the action may be multi-modal, so we prioritize sampling both models. We allocate less samples to VG because PP also needs to predict future video; and less samples to FD and ID because they are conditioned to move as conditions or to future observations respectively.
These additional conditions can reduce uncertainty, but under partial observation conditions do not make either of these tasks a deterministic task; here the difficulty sequence is a design hypothesis.
Following a manually-adjusted Difficulty-Aware sampling ratio, the success rates of P and PP exceeded their single-mode training results, while the PSNR of VG and FD remained consistent with the Equal Probability and Token-Matching strategies.
Find out three:Mixing multiple UVA training objectives depending on the difficulty of the task can improve strategy success rates while maintaining dynamic modeling capabilities.
Model Scaling studies.To study how performance changes with model scales, we trained a Cosmos 3-Edge variable: 1.3 billion parameters that are frozen to understand branches, 1.3 billion parameters that have been minimized to video branches, and 500 million parameters that have been trained from scratch to action branches, for a total of about 4 billion parameters.
For two models, we adjusted the global batch size of each iteration from 1 million tokens to 16 million tokens and trained a maximum of 4,000 GPU hours. Nano evaluated once every 1,000 iterations, Edge evaluated once every 2,500 iterations.
Figure 8: Model Scaling studies(a) Five Nano trainings and (b) Five Edge trainings in the RoboLab success rate changes with the amount of training calculations in the Policy mode; each curve corresponds to a global batch size. (c) The best success rate achieved by each model within a given computing budget; all checkpoints are shown in light color.
Figure 8a/b shows how RoboLab's success rate varies with each training. Nano models with 16 billion parameters achieve convergence much slower than Edge models with 4 billion parameters, and ultimately achieve much higher success rates.
Larger 16-billion-parameter models also achieve higher peak performance when larger batches are used; smaller 4-billion-parameter models peak when batch size is 4 million tokens and continue to increase batch size without profit under the same computational budget.
In Figure 8c, we superimpose the first two charts and highlight the best performing checkpoints under each computing budget. On a lower budget, the two models are similar in performance; Nano models with 16 billion parameters are significantly stronger when the budget exceeds 1,000 GPU hours.
We attribute this advantage to the stronger predecessors and greater capacity of the larger pre-training backbone: they improve language grounding and task generalization, both of which are critical to RoboLab evaluation.
The fourth discovery:Under higher computing budgets, larger models are better than smaller models and are better suited to use larger batches; under lower budgets, smaller models are closer in performance, increasing the number of iterations.
4.2 What can be achieved?
Following the design selection in Section 4.1, we initiated the final CamUVA model from Cosmos3-Nano-Policy-DROID and trained 30,000 GB300 GPU hours on mixed data on CamX, see Appendix B.4.
Because all robots share the same camera-centric interface and use a flexible sequential packing method, this checkpoint controls single-arms, two-arms, and mobile robots with different camera angles without requiring any specific subtlety.
In addition, each single-task benchmark dataset accounts for up to 1% of the training token, and we do not perform any task-specific mitigation. All assessments in this section use the same CamUVA checkpoint and use the fastest Policy mode.
Section 4.2.1 Assessment of distributed intrinsic hosts: The configuration of robots and claws is fully consistent with the training data for the task. Section 4.2.2 Assesses cross-intrinsic performance, including intrinsic migration that has occurred only in other missions; and intrinsic migration that has not been seen during training, i.e. zero sample intrinsic migration.
4.2.1 Distributed intrinsic evaluation
We evaluate the final CamUVA checkpoint at the standard benchmark and compare it with advanced methods of optimization or training for each benchmark individually.
In the RoboLab simulation benchmark, we compared CamUVA to four advanced baselines: two post-training checkpoints on DROID, namely Cosmos3-Nano-Policy-DROID and -DROID; and two cross-core models, namely LAP-3B and Cloak.
We evaluated each model on a comprehensive benchmark containing all 120 missions, with each mission running 10 episodes. Configuration, roll-out, frequency and uncertainty estimates are detailed in Appendix D.1.
The result:In P mode, CamUVA has a delay of 0.15 seconds, see Appendix B.1. Its success rate is 32 percentage points higher than the best performing trans-core baseline, and is much faster than the only supporting PP mode, the Cosmos3-Nano-Policy-DROID delay of 2.6 seconds.
This suggests that after cross-body training, the model retains a stronger ability to generalize tasks and follow instructions.
Figure 9: CamUVA manifests itself within the distribution.All assessments use the same CamUVA checkpoint, with no specific post-training for the benchmark. The brackets indicate the claw body used for each task and the proportion of the corresponding data to the CamUVA training token. The definition of error lines in the RoboLab diagram is given in Appendix D.1.
In the real world, we use the standard single-arm UMI configuration test CamUVA to adjust the assessment scenarios according to UVA settings and compare the results reported by UVA in the original assessment set.
We also run CamUVA on mobile robot RB-Y1 and compare its success rate with the results reported by ModPack.
All baselines are native-specific models for training for corresponding tasks; CamUVA is a unified cross-native model, with each of these benchmark datasets accounting for up to 1% of the CamX training token, as shown in Figure 9.
The result:In Cup Arrangement and Towel Folding tasks, CamUVA is superior to the multi-tasking UVA and Diffusion Policy exclusive models. In particular, CamUVA is able to recover from its own errors, which is not present in the sample data of these tasks, see Appendix E.5.
In the Dish in Rack mission, CamUVA consistently placed two plates in unseen environments, with a success rate of;ABC-VLA was able to pick up plates, but difficult to complete placement in non-distributed environments, see Appendix E.4.1.
In the Box Placement mission, CamUVA predicted the camera's trajectory and transmitted it to HoMMI's reverse-motion-based full-body controller, see Appendix E.3. Its success rate was 30% compared to the ModPack's full-camera, non-power matrix input of the joint space Diffusion Policy; the latter success rate was 60% compared to the version using the joint force matrix.
Finding 5:Strategies trained on CamX hybrid data can be directly deployed on native bodies that have appeared in multiple trainings, competing with advanced models that are specifically optimized or post-trained for each benchmark.
4.2.2 Cross-body evaluation
We assessed the generalization capability of the CamUVA open box itself. Differences in mechanical arm mechanics can be directly abstracted through camera-centered motion representations; in contrast, differences between the claw and the camera require generalization capabilities across the body model itself.
Specifically, we tested CamUVA on four different missions, each with data collected from a specific host:
Water Pouring in UMI data;
Cup Flipping in iPhUMI data;
Cup in Box in the mission center of GenRobot V4;
DROID and RoboMIND collected data using Robotiq's Banana in Pan claw collection.
For each task, we used the same checkpoint to evaluate five different nationalities: three of which appeared in the CamX, namely UMI, iPhUMI and GenRobot; the other two that were not seen in training, namely the new claw on the iPhone camera, and the UMI claw on the RealSense camera.
In the training data, the UMI claw is only used in pairs with the GoPro and iPhone cameras. All configurations are installed on the same ARX5 mechanical arm.
Therefore, each of the columns in Figure 10 corresponds to one of the following three situations: the entity used to collect the task data, i.e. the distributed intrinsic entity as a reference; the entity that appeared in CamX but never performed the task, i.e. the intrinsic entity migration; or the entity not seen in training.
Each column evaluates 10 episodes, using different backgrounds, objects, and locations to test the robustness of the strategy.
Figure 10: Results of cross-body missions.Above: the four tasks, the claw body used to collect the task data, and the ratio of the task data to the training token. In the middle: the success rate of each task on different entities; below each column is the average completion time of successful episodes.
The result:CamUVA performs well on the distributed intrinsic properties of each task and typically has a shorter completion time.
In the cross-body evaluation, we found that two factors were most important: the shape of the claws and the type of the camera.
Similar claw shapes mean similar interactions between claw and objects. UMI and iPhUMI use exactly the same claw fingers and thus perform well on each other's missions; GenRobot claw shapes are closest to Robotiq and therefore perform best on Robotiq missions.
Similar types of cameras, such as penetration cameras or fish-eye cameras, can improve spatial estimates of models. Both GenRobot and UMI GoPro use fish-eye cameras, so although the two claws look completely different, they still perform well on each other's missions.
The other three test subjects and Robotiq are also used to view cameras, so they have a higher cross-body success rate.
The biggest difference between iPhUMI and GenRobot is that the two have different claw shapes and camera types. The GenRobot claw completely failed in the Cup Flipping mission due to errors in spatial estimates and dynamic estimates, see Appendix E.4.2.
Finding 6:Using a unified camera operating space for cross-body training, the task knowledge learned on one host can be transferred to other hosts that have appeared in the training as well as to unseen hosts, achieving a higher success rate. Similar claw shapes and camera types help improve the ability to migrate.
5 Conclusions
When 20,000 hours of open-source robot demonstration data is shared with a camera-centred interface, it can be used to train the same model. CamX converts 37 data sources into camera trajectories and position-recognition viewpoints; CamUVA does not require any robot-specific modules to process the data.
Camera posture movements, based on difficulty mixed UVA training objectives, and point-of-view recognition based on posture, all have an impact on performance; using computational volumes for a larger backbone is still the main means of performance improvement.
Single checkpoints trained on CamX mixed data can run on multiple cores, compete with specialized strategies, and transfer mission knowledge to an unprecedented claw and camera portfolio without the need for new data.
