Can Robots Learn Everything from Millions of Hours of Data? Inside the Embodied-AI Data Race

When training data takes millions of hours, what exactly can robots learn? From grabbing, sorting objects, to continuously performing multi-step operations in a strange environment, the requirements for robot capability vary from task to task, and the training data required varies with it. Human first-person video, real-world interaction recording and simulation data provide different learning signals, and their value varies with model architecture, training phase and application goals. How to select and combine data, validate the benefits, and continue to obtain and use effective data at reasonable cost, form the core of the next battle for data intelligence.
On November 5th, the "Triple T" Zhangjiang Towers Summit Technology Salon, jointly organized by Zhangjiang AI Innovation Township Ecology and Machine Heart, was held in Zhangjiang AI Innovation Township under the theme "Data In Motion: The Next Data Battle for Physical Intelligence", focusing on the front-end exploration and practice progress of physical data through a two-person technology crossover, three Open Mics and an industrial crash site panel, exploring the pathways and bottlenecks of transforming hardware data into robotic capabilities, and building a platform for collaboration between research and industry.

Technically, where do you put your next data?

The same data budget is used to cover more human-operated scenarios, or to accumulate feedback on the actual actions and interactions of robots? This choice affects not only what capabilities the model can acquire in the pre-training phase, but also the deployment of subsequent missions to adapt to real-life scenarios.

The talks were chaired by researchers from Shanghai University of Technology, assistant professors, doctoral students, head of the YesAI laboratory, and the founder of Instantaneous Technology, Shingon. Zhao Huang, chief data scientist and assistant professor at Qingdao University's Institute of Artificial Intelligence, discussed the division of human experience and machine data capabilities, data inputs at different training stages, and verification of pre-training results, and further explored key issues such as touch feedback, cross-body migration and model generalization.

Yellowstone believes that if prioritization is required for pre-trained data inputs, real machine data should be prioritized at this stage, because robots need to build reliable operational capabilities through direct interaction with the physical world. In contrast, large-scale human video data can be used to support mission understanding and high-level planning, but there are still problems of inaccuracy in motion trajectory, force control, and exposure to feedback.

He emphasized in particular that touch should not be merely a complement to the post-training phase, but should be integrated into basic capability learning as early as possible; it is also worthwhile to explore low-cost ways of obtaining touch data through technologies such as non-body acquisition, electronic skin.

In R&D practice, the team should first build a basic understanding of the causes of missions and failures through machine data and then gradually scale the data. For verification of pre-training results, it should not only look at the success rate of existing missions, but also examine the model's learning speed in the face of new missions, adapt the required amount of data and the ability to generalize across scenarios, and identify real benefits through multitasking assessments.

Liu emphasized that the choice of physical data does not have a unified answer, which depends on the application positioning and the training stage of the robot. In the industrial scenario, tasks such as precision seizure and installation rely heavily on real-machine data, while general-purpose physical intelligence in an open environment needs to make full use of the scalable advantages of human experience data.

He argues that pre-training usually places more emphasis on data coverage, while post-training enhances specific task performance through higher-quality data. However, the structural and control differences between human action and the robot's body, even if able to accurately restore human operations, do not mean that it can be directly migrated to the robot. The potential value of human data also includes active perception, task planning, hand-eye coordination and long-range behavior linkage.

For tactile data, he believes that the cost of data collection, data quality and coverage should be taken into account as a whole, and it remains to be verified whether a small amount of specific structural data can effectively improve pre-training capabilities. In addition, the lack of reliable independent assessment methods for pre-training currently available, whose real contributions are easily disguised by post-training effects, and how to effectively distinguish the benefits of the two remains an important challenge.

From left to right: researchers at the Shanghai University of Technology, assistant professors, PhD students, the head of the YesAI laboratory, the founder of instantaneous technology, Shi Shino, the chief data scientist at Penguin Wave Technology, Huang Zhen, and assistant professor at the Chinese University's School of Artificial Intelligence, Liu.

Tech Open Mic: What have robots learned from the data?

The selection of data routes needs to be further implemented as a specific method of collection, training and verification. The three technologies Open Mic discuss how to convert data into verifiable and sustainable operational capabilities, separately from the acquisition of information from physical interactions, data formulation and evaluation, and closed-loop learning in real deployments.

Vision can record the state and movement of objects, while touch can supplement information such as the forces and movements that are difficult to judge directly from images during contact. The founder and chief scientist of the Thousand Robot, Professor of Transportation at Shanghai University, Head of the Robot Operations Sensation and Intelligent Laboratory (MPI Lab), Madoulin, in charge of the multi-modal collection and physical interaction of data, emphasizes that touch includes forces and geometry, motion and movement, which are an important basis for the finer operation of robots. To transform these information into operations, both through high-quality, high-consistency sensors collect high-quality data, to establish standardization of the collection of tasks, and to promote the analysis of models and tests of models, to launch a set of high-quality tests to evaluate the quality of computer language, and to demonstrate the ability of computers to integrate data in a high level of machine learning.

The founder and chief scientist of Thousand Thousand Robots, professor at Shanghai University of Transportation, head of the Robot Operations Perception and Intelligence Laboratory (MPI Lab)

After the data entered the training link, how to combine samples from different sources, and how to determine whether the combination is effective? Vice-Professor of the School of Artificial Intelligence at Shanghai Transport University, full-time instructor at Shanghai Institute of Innovation, Head of the RhOS.ai lab, Li Yong Liu, brought the data formulation and evaluation of robotic learning, which will be discussed to advance data selection and capability verification. He pointed out that the data formulation should proceed from the tasks that robots need to complete, while combining the model structure, training objectives and evaluation methods.

Deputy Professor at the School of Artificial Intelligence at the Shanghai Transport University, full-time instructor at the Shanghai Institute of Innovation,RhOS.aiThe head of the lab, Lee Yong-lu.

The value of data will ultimately be reflected in the continued growth of robotic capabilities. Shenzhen Intelligent Co-Founder and Vice President of Technology, Lee Min-min, went from experience to intelligence: physical data scaling to continuous learning, with the theme of advanced training supported by high-fidelity experience, and the boundaries of continuous learning to expand capabilities. The team revolves around experience, characterization, evaluation and feedback, explores how to reduce reliance on real-world models through HiFi-UMI, tri-dimensional monitoring, physical error correction feedback, and explores the collaboration of big models such as GPT-Astra with physical strategies, making real-world deployments a testing ground for both capabilities and a source of growth, driving the ability of machines to interact with the physical world in continuous learning and integration.

Li Ming, Co-Founder and Vice President of Technology at Shen Park Intelligent

Industrial collision sites: How to reduce the amount of data from collection to training?

From the benefits of validation training to the continuous supply of effective data, there is also a need to address issues such as data alignment for different hardware, quality of collection and production efficiency. How can new information be really used by models and how can the costs of collection, processing and adaptation be reduced?

Around these issues, the Industrial Clash Field, titled "New Variants of Physical Data: Diversity, Multimodelling and Scalization", was hosted by Machine Heart partner Zhou Fong, PhD student at MMLab, University of Hong Kong, founder of the Lumina Physical Intelligence Community, Chen Zheng, CEO of New Intelligent Body, and co-founder of Modeling Technology and CTO, combined their practices to make effective use of modeling around the value of data diversity, touch, and the quality and cost of mass production.

Chen Chen said that judging the value of data should focus on both distribution and quality, as well as on model architecture and pre-training capabilities. Whether new data can bring widespread benefits requires evaluation of long-range tasks, complex interactions with scenario changes, combined with horizontal comparisons and dilution experimental verification.

Zhou Yi stressed that full-body touch plays an important supporting role in the brain's physical world cognitive training, and it also has far-reaching benefits for the closed-loop control of small brain models. Data needs to be stable, accurate, and timely synchronized with motion disruption filters.

Liu Xiaobo pointed out that the requirements for data quantity, quality and accuracy vary at different training stages. In order for the model to make effective use of touch, it is necessary to first solve the data representation of different hardware and then explore the integration method. In team experiments, premature integration of visual touch representation can reduce performance.

From left to right are: Machine Heart partner Zhou Fan, PhD student at MMLab, University of Hong Kong, Lumina, founder of the Intelligent Community, Chen Tianqing, CEO of New Intelligent Body, Liu Xiaobo, co-founder and CTO of Motion Technologies, Zhou Na.

From the selection of data sources, to the verification of training gains, to the cost of production and use, the "Triple T" Zhangjiang Tower Summit series of technology salons break down "data size" into specific issues related to tasks, models and deployment conditions.

Further exploration of these issues will also require ongoing exchange and collaboration between research and industry. In the final "Link up · Ecological Connection" segment, guests and on-site audiences will continue to exchange views on relevant topics. This will be the starting point for a gathering in Zhangjiang AI Innovation Town, a collaboration around data production, model evaluation and real-world applications, with the hope that the on-site discussion will extend to new research and practice, promoting the conversion of data accumulation into robot capabilities that can be verified in real-world missions.