The BEHAVIOR Challenge is a physically intelligent challenge based on the BEHAVIOR-1K simulation environment designed to evaluate the ability of robots to complete complex household tasks. The 2026 competition selected 100 complete household tasks, covering seven scenarios, and provided approximately 20,000 human remote operation demonstrations, with a total duration of approximately 1,950 hours.
The competition used a partial Q-score to average 100 tasks and assess the system's ability to adapt to different initial states through hidden task instances. In this competition, the HiPhys-RSI program won, with a hidden Q-score of 0.683 and a complete task success rate of 48.2%. They did not limit the problem to training an end-to-end operating model, but instead used a combination of VLM Coding Agent sensing, planning and control tools, continuously modifying the robot pipeline by executing feedback, and combining VLA strategies to complete the task.
Here is the correct spelling.
Modern discussions about scaling revolve around increasing training data, the size of neural networks, and computation. However, the early research tradition of symbol AI took a complementary path: expanding the knowledge, rules, and procedures available to intelligent systems. Both perspectives emphasize different forms of growth: model training on the one hand, and explicit executable structures on the other. As the multi-modal LLM's ability to write and modify code increases, code-as-policy builds a bridge to these two traditions.
Our BEHAVIOR-1K program uses hierarchical Physical Recursive Self-Improvement (HiPhys-RSI) to explore this direction. In the freeze of the VLM mainstream, Agent uses classic perception, planning and control methods to build a robotic pipeline, then use feedback and video diagnostic failures and modify the code. Knowledge builds in the form of repeatable procedures, tool constraints and conditional behaviors.
1 Quotes
Scaling has become the core organizational principle of modern AI. In its current context, it usually refers to increasing the amount of trained data, the scale of neural networks, and the computation of input training and reasoning. Its fundamental goal is to gain increasingly universal capabilities through mass learning. The early research tradition of symbol AI emphasizes a complementary form of growth: the accumulation and organization of structured knowledge. Intelligent behavior is supported by the representation of rules, calculators, programs, and other feasible system reasoning and execution. From this perspective, capacity can be increased by expanding system knowledge, executable operations, and combining these operations. These studies traditionally emphasize the differences between Scaling: the ability to learn models, and structured executable knowledge.
This problem is of particular concern in the field of robotics, which already has a large number of effective computational methods in place. Geometric reasoning, task and motion planning, SLAM, perception and feedback control can all provide powerful solutions within their respective scope. However, applying these methods to different tasks and environments often requires a large amount of artificial engineering development and analysis. A pipeline that is effective in one scenario may require different capturing strategies, adjusted sequences of actions, or additional recovery procedures in another. Therefore, the challenge lies not only in the ability of individual modules, but also in how to combine and adjust these modules according to the task required.
Coding Agents provide a way to automate part of their work through Agent-driven recursive self-improvement, RSI: they can check for feedback and video execution, diagnose failures, modify procedures, and test modified behavior. After multiple iterations, useful modifications can be maintained as replicable tools and conditional behaviors, transforming the experience of a specific task into structured knowledge that can support subsequent tasks.
This view extends the meaning of the generalization of robotic systems. In addition to implementing fixed strategies in unfamiliar scenarios, the system can also build and adjust solutions using existing tools. Capacity growth may result from the accumulation and refinement of this structured knowledge if the programs generated can be replicated across tasks and environments. This echoes a tradition of symbol AI: to compile the experience of solving problems into replicable rules and algorithms, thereby reducing the need for rethinking in subsequent scenarios. As these knowledge become more abundant, organizations become more complete, parts of the existing structure that can be processed in each new task will also increase, and parts that need to be rebuilt or modified will gradually decrease.
We explored this problem through solutions to the BEHAVIOR-1K challenge. Where long-term household tasks require diversified interactions, coordinated sequences of action, and adaptation to physical state changes, this provides a challenging environment for studying how executable structures support complex robotic behavior. Our program HiPhys-RSI uses a generic coding agent, assembles robotic components into task pipelines, and improves them through Agent-driven RSI. Here, layered improvements include the use of high-level tools to coordinate, the improvement of implementation of underlying tools, and the post-training of VLAs using pipeline-generated trajectories.
Our framework will also link executable program improvement to basic robot model training. We will combine the trajectory of executable pipeline generation with the demonstrations provided by benchmarks to fine-tune the VLA to integrate the learning strategies acquired during pipeline construction and improvement. The generated strategies can be selected along with the well-built pipeline to infer time routing mechanisms. This provides the basis for future joint evolution between executive structures and learning strategies: improved strategies can expand upon subsequent pipeline construction, while further improved capability pipelines can also generate more training experiences.
2 Related work
2.1 Special tools in the field of mature robotics
The field of robotics has developed a number of methods for solving different computational problems and providing replicable basic components for embedded systems. Geometric reasoning methods are used to describe spatial relationships, evaluate collisions and gaps. Perceptive methods are used to extract useful information from sensor observations through localized character extraction, correspondence estimation, and point cloud mapping. Task and motion planning methods are used through task decomposition, motion reasoning, and sequential parameter sampling to link discrete task decisions to geometric feasibility in continuous space.
Together, these methods build a deep algorithmic foundation, but using them requires appropriate inputs, assumptions and interfaces. Our work builds on this foundation to investigate how the General Coding Agent selects, combines and adjusts robotic tools, builds executable task pipelines, and automates the extension of these tools to the engineering work required by different tasks and environments.
A summary of the framework presented in Figure 1.
HiPhys-RSI workflow (left):Coding Agent with a frozen VLM main stem, which gains task experience through tool combination, rollout analysis and cross-task experience and iterates a perfect executable robot pipeline on training instances. Optional artificial corrections can provide additional guidance in the construction process.
The following outline is provided as an overview of and topical guide to strategy:The agent uses the policy card to select a pipeline or VLA strategy for each new task, and the selected component performs the task autonomously.
The result (right):Our framework and mission scores of the best VLA baselines for each task, grouped by task category.
The process in the diagram includes:
- Experience assignments:VLM interacts with the environment, gathering mission experience, including mission observation and interaction trajectories.
- Pipeline design:It combines autonomous robot pipelines, including components for perception, planning, inverse-motion (IK), navigation, and control.
- It's the first time I've seen this.Observe rollout and evaluate robot behavior, analyze feedback and diagnose failure.
- It's time to change pipelines and strategies:Module modification and pipeline verification, pipeline improvement and VLA post-training.
When the score rises below the threshold, corrections can be introduced in the circuit. The completed pipeline can be built to perform tasks independently.
2.2 Agentic Robotics
Language Model Agent extends the model's capabilities through tool use, environmental interaction and feedback, including reasoning and action by mistake, learning to call APIs, and reflection through language. Executable programs provide a unified operating space and enable the accumulation of replicable skills. Software engineering Agent further demonstrates the ability to store checks, program editing, and execution-based modeling, laying the foundation for Agent's self-evolution through improving programs and skills.
In the field of robotics, Agent incorporates environmental feedback into planning and generates combined perception, planning, and control processes. Robot harness through task decomposition, routing, and memory-based coupling, supports capability-based coordination of heterogeneous strategies; and works to combine frozen VLAs with parse-based basic movements and execution of memory. Li provides geometric perception capabilities through queryable tools, and distills execution trajectories into smaller models to connect tool-mediated interaction and learning strategies.
Recent work has further extended these Agent methods to the self-evolution of robots. Li et al. integrated data collection, learning and execution of strategies; Xiao et al. used Coding Agent to improve strategies and training infrastructure through physical experiments. Wen et al. organized skills in a layered manner, positioned failures, and verified modifications before replication. Liang et al. viewed the supporting system as an evolving strategy, converting execution trajectories into verified context management, persistent memory and layered skill updates. Zhang et al.
These studies explored capacity building through program modification, skill accumulation, system evolution and strategic learning, but did not examine the scalability of structured tool use, nor fully compare it to end-to-end approaches such as VLA. Our work aims to fill this gap: studying how the General Coding Agent uses mature robotic tools to build and improve executable pipelines, and systematically compare these pipelines with learning-derived action models.
3 Formalisation of the issue
We studied how the General Coding Agent could build an autonomous robot pipeline based on task descriptions, demonstrations and replicable robot tools.
A pipeline construction problem is defined as:
These include specifying tasks and their conditions of success; a set of reference examples; a robotic environment; a callable toolkit, in which tools implement classic methods of navigation, perception, planning, and control. The pipeline previously constructed can also be incorporated into the toolkit as a replicable callable toolkit. The input, output, and time range of these tools may vary.
The environment provides observation and receives robot actions. The reference sample consists of video and trajectory of previously collected robots completing corresponding tasks. For each task, the commands indicate the initial set of environment configurations to be used to gain mission experience and perfect the pipeline.
Given, a parametrically frozen agent constructs an executable pipeline. The agent has a frozen VLM mainstream for interpreting visual evidence. The pipeline defines the combinations of tools, information flows, and conditional execution logic based on observations and tool execution results. The pipeline generated by the coordinated tool calls, responds to environmental feedback, and performs autonomous tasks. Its execution process is noted as:
It records the execution trajectory and mission outcomes. In reasoning, the Agent uses the accumulated information capability to select an existing pipeline or VLA strategy to complete a new mission.
The construction of the pipeline is carried out by repeatedly executing and perfecting the above.
Among them, the initial environment configuration is specified. The resulting video, execution log, tool output and task results together guide the next round of modifications:
Among them, the execution of feedback is combined with the visual analysis of the freezing VLM. The parameters of the coding agent and VLM remain fixed throughout the process; improvements are encoded in the executable processes of the pipeline and their associated structural knowledge.
The goal is to build a pipeline that maximizes the expected task score:
It represents the set of executable pipelines constructed using available tools, measuring the performance of the execution process. The expected value takes into account changes in environmental configuration and execution results. During the construction process, the system estimates performance on a set of fixed training instances to guide improvement.
4 Ways
Our framework enables the freezing VLM-based Coding Agent to build an autonomous robot pipeline through four stages: task experience acquisition, design pipeline analysis, rollout analysis, and iterative improvement. The agent uses demonstration and interaction on a set of mission training instances to provide the basis for the design, assemble the available robot tools into executable programs, and use the executable feedback to improve those programs. Throughout the process, the freezing VLM executive is responsible for helping interpret visual evidence.
4.1 Gaining mission experience
For each task, we sampled 10 instances from the BEHAVIOR-1K training environment, forming a set that remained fixed throughout the process of experience acquisition, rollout analysis and pipeline improvement.
VLM-based Coding Agent first checks task descriptions and reference samples, then directly controls tool calls and robot interactions on a small number of instances in the collection. During this exploration phase, the Agent interprets the results of observations and tool executions, selects follow-up operations, and attempts to complete the task. These interactions provide first-hand evidence to determine which tools are needed, how to combine them, and which observational conditions indicate successful completion of the task.
The resulting experience records the dependencies between inputs and outputs and operations of the tool, as well as the intermediate states associated with the progress of the task. These experiences, together with the reference demonstration, help the Agent to implement the task-defined success conditions as observable results and identify which links need to be perceived, coordinated or feedback.
Subsequently, the Agent will use the tools that are explored in each training instance to transform the decision making into executable structured procedures, building autonomously operated pipelines based on the interaction trajectories collected.
4.2 Executable pipeline design
Based on the task experience gained from it, the Coding Agent combines the tools into executable programs and builds an initial pipeline. The Agent translates the operations discovered during the exploration and their dependencies into tool calls, information flows, and conditional execution logic.
In order to coordinate the various interfaces and time-range tools, the Agent writes the necessary support codes for preparing input, interpreting output, maintaining execution status, and transferring controls between modules. Observable success conditions established during exploration provide a basis for checking intermediate progress and completion of subtasks.
The ultimately generated pipeline integrates classical robot modules into the same program. For example, the perceptual output can provide parameters to the planner and trajectories generated by the controller executive planner. The program specifies the timing of each component's call, and how to advance the follow-up execution based on observations and tool execution results.
After the pipeline construction is completed, it is executed autonomously in accordance with the pre-written logic and coordinated with the tools directly responsible for the exploration process by the Agent. During each rollout, the pipeline responds to environmental feedback through the verification mechanisms and conditional branches that have been implemented.
During the build process, the Coding Agent then analyzes the rollout on the training instance set and modifies the program between the two executions, separating the agent-driven pipeline build from the robot's autonomous execution.
4.3 Rollout Analysis and Failure Diagnosis
In the second iteration, the pipeline is executed under the initial configuration and obtained. The resulting video, execution log, tool output and task results are integrated as feedback.
Program-level feedback reveals what operations were called and what results they returned; VLM-based visual analysis assesses corresponding robotic behavior and environmental changes. These complementary signals help distinguish whether the tool was successful in calling and whether the physical task was actually successful.
The agent will compare the observed rollout with the expected task progress to determine where the execution begins to deviate and which pipeline components need to be modified. The failure may be due to incorrect tool input, incompatible assumptions between modules, improper execution order, or inadequate handling of the failure to execute an action.
Visual evidence also helps to diagnose failures that are difficult to detect due to log execution alone. For example, an operation, although completed, does not bring the object to its intended state; or a motion planner cannot find a path due to small obstacles.
The purpose of this analysis is to link the observed failures to possible procedural changes. Agents not only rely on task scores, but also use existing execution evidence to determine what should be changed and why.
4.4 Improvement of pipelines
Using the task description, current pipeline and rollout feedback as a condition, the coding agent created the following version of the program:
Modifications can involve tool selection, input and output changes, execution sequences, tool configuration parameters or condition logic. Subsequently, updated pipelines are evaluated to determine whether these modifications have improved task performance. By repeatedly executing and analyzing, Agents can improve the coordination of existing capabilities while keeping the Coding Agent and VLM parameters fixed.
To show the pipeline's average assessment score above. We define the improvement of a change as:
The improvement process is terminated when the score increases the number of consecutive specified times below the threshold, or when the predetermined computing budget or interaction budget runs out. The improvement threshold and wait window are superparameters related to the task, with the default values being changed 10 consecutive times and adjusted according to the length and difficulty of the task respectively.
In the process of construction and improvement, a selectable artificial loop correction can provide additional guidance for pipeline design and modification. After construction is completed, the pipeline is executed autonomously according to its programming logic.
4.5 Cross-mission experience and pipeline reuse
Built on a set of training instances for each task, executable and structured knowledge is generated to support subsequent tasks. Executable programs retain a combination of tools that have already been implemented; structured knowledge records relevant operating conditions, observed failure patterns, and lessons learned from previous modifications.
These resources enable the Coding Agent to leverage existing experience when designing new pipelines. For a new task, the Agent can select and adjust previously constructed pipelines, using them together with the original robotic tools as callable components.
This supports layered combinations: a pipeline can coordinate multiple replicable programs, and each program can call on more underlying tools. Repeatable components are evaluated on a set of training instances for new tasks, where executive feedback on the set will guide the necessary adaptations.
Throughout the process, the Coding Agent and its supporting VLM remain frozen. Improvements in the executable logic and structural knowledge encoded in the pipeline allow the accumulated experience to improve the way in which existing robotic capabilities are coordinated across missions.
4.6 Trajectory mitigation of VLAs generated by Agent
The trajectory generated by the Agent provides a pathway for transferring the experience gained through HiPhys-RSI to the robotic base model. As the executable pipeline is continually improved through interaction, its rollout records the operation strategy, coordinated sequences of actions, and responses to execution failures.
We converted these rollouts into a sequence of observational puzzles coupled with task instructions and combined them with benchmark demonstrations for VLA post-training for specific tasks, see Section 5 for details.
This will link structured Scaling to data-based Scaling: the strategies and recovery behaviours formed by perfecting executable programs will generate observational feedback trajectories used to complement primitive examples. These trajectories allow the VLA to access other ways of completing tasks and dealing with execution difficulties, providing a pathway to integrate the experience accumulated in explicit programs into the learning strategy parameters.
4.7 Dynamic routing when reasoning
In order to support the routing of reasoning, as shown in Figure 1, we allocate a policy card for each available VLA strategy and built-in pipeline. Each card outlines the capabilities and usage requirements of the corresponding components, and their historical success rate distribution across a set of training instances, in the context of different tasks and under execution conditions.
For well-constructed pipelines, the cards also contain records of borderline cases collected when constructing and improving on these sets, including patterns of observed failures, and unreliable conditions of execution. These cards provide a basis for Coding Agent to judge the applicability and limitations of each component.
For each input task instance, the Agent compares the context and requirements with the policy card to select the appropriate pipeline or VLA strategy. The routing process takes into account the historical performance of the component under the relevant conditions, the recorded border case, and whether its functionality is appropriate for the current task.
The selected components then complete the task according to their own execution mechanisms; the tool coordination within the pipeline is controlled by their programming logic.
The policy card will be updated based on new execution results and observed border case dynamics in the training instance set, enabling subsequent routing decisions to leverage the accumulated experience.
5 VLA Model: Architecture and Training Setup
We adopted the first BEHAVIOR Challenge in 2025 as a foundation model. The program, based on pre-trained weight, continues training by using the BEHAVIOR model to adapt the model to mobile operations tasks in home environments. Its major modifications include: the use of relevant noise in Flow Matching, the use of a learning mix of attention between the visual director and the Action Expert, and the prediction of task phases as conditions for the execution of the action.
We adapted this foundation model to 100 tasks for the current benchmark and evaluated three training variants.
Meta-1-Epoch。We used aggregated demonstration datasets covering all 100 missions for a complete epoch of joint training of the base model. Training using all-available benchmarks for each task. Training demonstration enables a single shared strategy to learn navigation and operational behavior across missions. The resulting checkpoints are used directly to evaluate all tasks while serving as a follow-up to the common initialization model for post-training specific tasks.
Per-Task Post-Training。We initialize each task-specific strategy from the same Meta-1-Epoch checkpoint and continue training using the corresponding task data. The data set combines the training demonstrations provided by the BEHAVIOR-1K benchmark with the additional trajectories generated by the Agent-driven pipeline to migrate the behavior formed by pipeline improvements to the learning strategy, as described in Section 4.6.
This process generates a separate strategy for each task: both retaining the initial capabilities acquired by the joint training and adapting to specific task execution patterns and operational requirements. When evaluating, we select strategies that match the target task. We also add 3D spatial information, such as depth and geometric information relative to robots, to the strategy training.
Meta-2-Epochs。We continued to train together on the same aggregated benchmark sample data set, bringing the total training duration to two epochs. This variable is used to evaluate the effect of increasing multi-tasking training, not doing task-specific adaptations, and not using pipeline-generated trajectory expansion data.
These VLA variants serve both as learning-style strategy candidates, along with a well-constructed pipeline to engage in reasoning routing; and as a contrasting baseline for using VLA alone to explore the effects of extended multi-task training and post-task specialization training. In these, the post-task specialization training setup also incorporates Agent-generated experiences into VLA training, linking pipeline improvement with improvements in learning-style strategies.
6 Results and Discussions
6.1 Comparison of overall performance with VLA baselines
Figure 2 shows the performance of each of the eight tasks.We compared the three VLA training variants to the final program. Each column indicates the score of a task, the task ID is marked on the outer edge of the chart, and the task is divided into eight AH classes according to the diagram. See Appendix A for detailed category definitions and classification criteria.
Coding Agent makes Scaling of structured knowledge possible.In almost all missions, our Agent system scores are higher than the VLA-only variants, with enhancements covering all eight task categories, as shown in Figure 2. These extensive enhancements are consistent with the benefits of blending explicit information processing, task-level reasoning and dedicated action execution with hybrid systems.
Each phase plays a different role: Classical perception and SLAM modules convert observations into structured scenario information; in the pipeline construction and improvement process, VLM-based Agents use this information to formulate execution strategies and specify the goals and constraints encoded into the pipeline; and downstream dedicated executors convert these decisions into robotic movements through their own planning and control methods.
For example, path mapping based on availability supports object search and navigation, feedback control to achieve precise alignment, task and motion mapping to handle logically structured classification tasks, and code-as-policy programs perform repeated crawling and placement operations. This division connects scene understanding to task-level decision-making and physical execution, enabling complementary components to work in sync through explicit information flow.
To achieve these gains, the ability to select the right ones, coordinate their execution, and perfect their concrete realization through environmental feedback is required. Therefore, the Scaling of a system depends on the ability to efficiently complete these engineering tasks across tasks. These processes traditionally require a large amount of artificial engineering input, limiting the scalability of systems built on structured robotic knowledge.
The general, VLM-based Coding Agent completes most of its work automatically through the RSI closed loop, converting the execution of feedback into modified implementation, constraint and recovery behavior. By easing this engineering bottleneck, the Agent provides a pathway for expanding the use of structured knowledge by building and perfecting executable pipelines.
Increased VLA training can improve task scores, but the skill gap still exists.As shown in Figure 2, in the evaluated training configuration, the benefits of extending unit training from one epoch to two epochs are relatively limited.
In contrast, additional task-specific training using a mix of benchmark samples and pipeline-generated trajectory data for post-training of each task can improve individual task scores while generally maintaining a relative advantage and disadvantage distribution between different types of tasks. These benefits reflect the common role of post-task training and data enhancement.
In the VLA variants evaluated, the performance of general placement and partial object deformation tasks was relatively strong; mobile retrieval, matching placement and precision operation remained challenging; and the performance of device operations was uneven.
We assume that these persistent gaps partly reflect the limitations of the evaluated model on long-range coordination and specific task operation capabilities. Specifically, mobile retrieval performance is weak, possibly due to the difficulty of the model in supporting the continuous decision-making needed to map, search and explore. Challenges in matching placement and precision operation may reflect its lack of feedback control required to align precise objects.
These requirements may explain why additional training can improve the performance of some tasks without fundamentally changing the overall distribution of advantages and disadvantages across different task categories.
6.2 Scaling of structured knowledge
Figure 3 The pipeline improves progress and skills formation.
- Left: Overall tasks scored with changes in the number of iterations improved, showing an initial rapid improvement, followed by convergence, and subsequent improvements of smaller magnitude.
- Middle: Distribution of the number of iterations required for different tasks.
- Right: The frequency of use of various techniques in the pipeline being constructed, the area of the bubble and the frequency of use are proportional.
Graphical skills include visual searching, navigation, grabbing, placing, joint movement operations, device operation, contact cleaning, cutting, dropping, repositioning, connecting and spraying objects.
The trend of structured knowledge scaling.The improvement curve on the left side of Figure 3 shows that Scaling of structured knowledge includes an initial capability combination stage, followed by a more demanding strategy discovery stage.
Early iterations can lead to rapid improvements, as many direct failures can be solved by modifying tool configurations, executing sequences and coordinating modules when overall strategy is fundamentally appropriate.
As these problems are addressed in various tasks, overall performance improvements gradually diminish and further improvements focus on unresolved difficult cases. Subsequent gains come from more stable operating strategies discovered through RSI continuous exploration; these strategies are not present in the training data or sample samples provided.
These findings altered the way the system handles tasks, allowing it to overcome persistent failures under the original strategy. Therefore, the initial platform period does not necessarily represent a fixed capacity limit: it may signal further progress that requires exploring other solutions. Improvements are stored in executable code so that the knowledge gained during the interaction process can accumulate continuously, while the underlying model remains frozen.
Extensive task coverage emerged earlier; the input required for remaining tasks increased significantly.In the middle of Figure 3, the overall distribution of iteration frequencies is concentrated in lower numerical values while having a longer right tail. This suggests that the cost of building a pipeline is relatively low for most tasks, but that a few tasks require much higher inputs.
This model highlights the balance in Scaling: when a system tries to cover increasingly difficult tasks, the rate at which costs of improvement increase may be much faster than the number of tasks that have been solved. Therefore, if a uniform iterative budget is used for all tasks, unnecessary work may be put into earlier stable tasks while stopping those that need to be explored further prematurely.
Depending on the observed progress, adjustment of the allocation of budgets may be more conducive to balancing the costs of covering a wide range of tasks with solving remaining difficult cases.
Shared skills support broad coverage, exclusive skill expansion capability boundaries.The frequency of use of skills shown in Figure 3 on the right reveals why improvements to some components can have a wide impact, while other components, although used less often, remain a necessary condition for expanding the scope of the task.
Navigation, visual search, crawling, and placement occur repeatedly because all household tasks require finding, approaching, obtaining, and re-placing objects. Repeated use of these skills also creates multiple opportunities for accumulation of errors, making their reliability critical to the success of long-range tasks.
Specialized skills such as joint movement operation, device operation, cutting and casting occur less frequently, but may determine whether a specific task can be completed. Therefore, scaling requires both improved sharing capabilities and solutions to bottlenecks in specialized skills.
Figure 4 Improves the number of iterations of pipelines for different task categories.The box represents the 25th to 75th percentile, the lines and numerical values in the box represent the median, and the lines represent the 10th to 90th percentile.
There are differences in the needs for improvement for different task categories.The distribution of categories in Figure 4 reveals different sources of improved workload.
In general, average placement requires the most improvement, reflecting even the usual clutching and placement procedures, which must be adapted to the various geometric shapes of objects, supporting surfaces and release conditions.
The use of tools also requires substantial improvements, as successful execution requires that the grip, direction and movement of the tool be coordinated with the expected impact of the contact.
The mean of fine operations is relatively low, but the upper end is particularly long: existing programs are sufficient to complete some of the tasks, while strict alignment and contact constraints make improving other tasks more difficult.
The distribution of materials also varies greatly, due to the different geometric shapes of the container and the different movements required to control material transfer.
The distribution of device operations is relatively concentrated and the typical iteration frequency is also low, because once the relevant institutions are identified, their interaction can usually be achieved through limited range of repeatable movements.
These models show that the need for improvement depends on the diversity of physical interactions across categories and their sensitivity to errors.
6.3 Generalization of hidden test sets
Whether the pipeline constructed for testing can be generalized to examples used in the development and reporting phases. As shown in Figure 5, the Q scores for most tasks in the hidden evaluation decreased, but the majority decreased to less than 0.2.
Some tasks maintain similar performance or score higher, indicating that the built-in program can migrate across instances.
The greatest reduction in tasks depends mainly on object search, whose success rate is significantly influenced by the object's random initial location. If the object is located in an obscured or difficult-to-reach location, search times can be significantly increased, causing robots to not be able to complete tasks within their budget.
Some tasks have a low word count, and some tasks use binary scores, so a disproportionate sharp drop in scores can occur if one condition is not met. Each task has only 10 test instances, and the differences in reporting are influenced by sample fluctuations.
Thus, the observed gaps reflect the sensitivity of the system to the configuration of instances, the roughness of the grades, and the fluctuations caused by the evaluation of finite samples. Overall, the results suggest that the system has the ability to generalize across instances.
Figure 5: Differences in Q ratios between hidden assessment and reporting results for each task.The column diagram shows, and is arranged in descending order. Positive values represent higher concealed evaluation scores and negative values represent lower concealed evaluation scores.
6.4 Failure pattern analysis
Failure to move from operating mistakes to executing bottlenecks.The failure distribution in Figure 6 shows that the failure of the VLA baseline is highly focused on grabbing and placing; in our approach, failure is more focused on grabbing and overtaking.
There has been a decrease in the proportion of placement, joint motion operations and object loss, which is consistent with the fact that explicit geometry constraints and dedicated execution programs can handle these interactions more reliably.
Grabbing is still the main failure pattern of both approaches, highlighting the difficulty of establishing stable contact when the geometry of objects, their accessibility and the degree of surrounding turmoil change.
Figure 6 compares our approach to the failure pattern of the VLA baseline.The column head represents the percentage associated with each failure pattern, and the numbers in the column represent the frequency ranking of the failure pattern in the corresponding method. A detailed description of each failure pattern can be found in Appendix B, and the failure pattern divided by task can be found in Appendix C.
Physical constraints restrict further improvement.The prominence of overtime and navigation failures reflects the cost of finding and approaching a target object. These failures often occur simultaneously when robots cannot find a target object, or when path planning fails repeatedly in a chaotic environment.
Long-term searches, repeated attempts at planning and inefficient mobility consume execution budgets, resulting in insufficient time left for subsequent operations. These failures reflect the limitations of current perception, search and motion planning strategies on visibility, physical space surplus and execution budget limits.
For example, target objects may be hidden behind larger objects in the corners of the cabinet, although repeated searches can still not be seen from the point of view that the robot can reach. Even if the target is found, the robot may need to navigate between small spaces of furniture, or allow the mechanical arm to pass through a narrow gap to reach the target. In this case, small errors in the reconstruction of the scene or in the robot's positioning may cause the path originally possible to collide with the surrounding environment.
For such situations, changing search sequences or repeatedly adjusting planning trajectories will have little effect on improving visibility or increasing physical space surpluses. Under current strategies, these constraints may lead to repeated failed attempts by the system, depleting the remaining execution budget.
6.5 Case studies: Unconventional strategies in an agent-driven pipeline
In addition to the overall performance, we also examined how pipeline construction and improvement generated operational strategies that were difficult to implement with adaptation to the constraints of the scenarios. Figure 7 shows six cases involving alternate contact points, preparation movements, execution sequences and geometry of placement relationships.
These strategies were developed through Agent proposals and improvements under artificial guidance, showing how the accumulated executive knowledge translates into specific robot behaviour.
The method of grabbing depends on the approximate orientation of the object.In Cook_broccolini's Mission 96, the robot grabbed the rim of the handle near its side when the handle was facing outwards or sideways. It raised the level of the handle and moved it to the stove to avoid bypassing the kitchen island in order to get closer to the handle.
Similarly, in carrying_out_garden_furniture (mission 74) walls or nearby furniture may prevent the robot from approaching the single-wheel handle. The robot approaches from the side of the wheel, grabs the front of the wheel with both hands, confirms that it has been lifted, and then moves the single-wheel to the gate.
Both strategies are based on the choice of approachable contact positions for objects in the scene.
Change the state of the object so that it can be operated.In the spray_for_bugs (task 38), a spray that crawls directly upright is prone to collision and may cause the spray that crawls to be at an angle that makes it uncomfortable to start. The robot first pushes the spray to the side, making it easier to crawle from the right direction and adjust it to a direction that makes it easy to start.
In stacking_wood (task 58) parallel placement can cause a piece of wood to roll or slip from below its supporting wood. The robot then moves the wood in transport to the vertical direction of the supporting wood and releases it slightly closer to its position. One end is mounted on the supporting wood and the other end falls to the ground, using gravity and ground support to maintain contact, reducing the risk of rolling.
These behaviors make subsequent interactions or placement more reliable by changing the position of objects and supporting geometric relationships.
Adjust the order of movement and change the mode of transportation.In packing_meal_for_delivery (task 81) the robot first puts the three empty paper bags directly into the box and then puts the burgers in the paper bags. This sequence uses the boxes to support the paper bags during the packing process, saving the need to support the paper bags one at a time when putting the burgers in.
In installing_a_scanner (task 92) the robot first opens it when the scanner is on the desktop, checks if the button has turned green, and then uses a slanted closed palm to move along the side of the dashboard, pushing the scanner to the laptop. When the scanner is too far away or in an unsuitable position to push, the pipeline uses a crawler transfer process.
These strategies simplify the execution of tasks by changing the timing of the execution of an operation, as well as the way the animal is moved.
Figure 7 shows unconventional strategies in the Agent-driven pipeline.Representative execution cases show the unconventional operating strategies found during pipeline construction and improvement.
Overall, these cases demonstrate how pipeline improvements will record practical adjustments in grabbing selection, object state, and motion coordination. Programs generated use scene geometry and physical support to solve execution difficulties and store these adjustments in code so that tasks can be executed autonomously.
6.6 Case studies: Benchmark Hacking or Reward Shaping?
Physical Agent self-improvement has raised concerns about benchmark hacking, but the boundary between using vulnerability assessment and response reward shaping is not always clear.
We use official evaluators and set the same task execution budget for different methods; hidden test instances are inaccessible throughout the development process. These controls support consistent comparisons and allow us to assess the system's ability to generalize scenarios outside development instances.
Two examples in Figure 8 illustrate the blurred boundaries between benchmark hacking and optimization for specific rewards.
In task 58 (stacking_wood), Agent stacks timber into three groups of two separate timbers instead of stacking them into a pile. This is a relatively clear case of benchmark hacking.
Task 48 (canning_food) was more controversial: the agent directly sliced the food into the refrigerator, gaining some scores, but not completing the goal of transferring the food to the bowl. The evaluator correctly recorded the progress of the task not yet completed, and local slicing may also be an effective intermediate step. However, the same behavior may also reflect that the system optimizes for easily accessible scores without continuing to advance to complete the task.
These two examples together illustrate why benchmark hacking is difficult to distinguish from reward optimization: both are responses to set goals, but they differ in how well they match the true intent of the task, may be significantly bypassing the goal, or may be reasonable but unfinished progress.
This ambiguity is not unique to the physical RSI; it reflects a long-discussed reward design problem in the field of RL: strategies may optimize setting incorrect rewards without achieving the expected goal. As Amodei and others have discussed, an agent indicator that is associated with the expected goal in normal behaviour may lose this correlation with strength optimization.
Similarly, in terms of how benchmarks are implemented, it seems reasonable enough that success quotes and partial rewards may no longer be sufficient when RSI explores the broader strategic space. These observations suggest that benchmark evaluation standards can evolve alongside systems that are optimized for them.
The shortcuts found can help improve the assessment of the criteria: revealing where termination conditions failed to accurately reflect the expected outcome, or where partial rewards encouraged strategies prior to the completion of the stay.
In perfecting these standards, it is also necessary to maintain reasonable alternatives, since deviation from the demonstration itself is not sufficient to prove the existence of hacking. Therefore, improving task-solving capabilities and improving evaluation standards can be mutually beneficial: stronger system exposes gaps in task definitions and better standards guide subsequent improvements towards the expected task goal.
In addition to better reconstructing and modeling real-world environments and interactions in simulations, and to help optimize and evaluate strategies closer to real deployment conditions, World Labs' work on the real-to-sim-to-real pipeline explores this direction.
Figure 8 shows potential benchmark hacking in the pipeline built by Agent.Representative execution cases show strategies that take advantage of insufficiently defined success conditions, or prioritize partial scores rather than completing tasks.
In addition to perfecting the definition of what constitutes a success, another valuable direction is the ability of the evaluation strategy to independently judge when it has been successful. Even if the success criteria are well defined, there may still be some capacity gaps that have not been measured if the evaluator tells the system at the same time when the criteria have been met.
Therefore, future benchmarks can be considered as an additional evaluation dimension of phase-conversion detection and completion status checks. For example, simulator status judgments can be retained for scoring while allowing the system to self-judge task progress and termination timing.
At the same time, assessing accuracy and latency is also valuable, as repeatedly questioning large closed-source models, even if judging correctly, can delay execution. A specialized study by our team explores this direction: distilling execution status assessments into a small model with a running frequency greater than 1.7 Hz, enabling frequent checking of task progress and phase conversion.
Incorporating such assessments with the results of the task can make a more comprehensive judgement as to whether a system can both achieve the expected objectives and independently determine when to move to the next stage or stop.
7 Conclusions
In this report, we explored how Scaling can accomplish robot tasks through structured, executable knowledge. We built and perfected an autonomous pipeline using a freezing VLM-based Coding Agent using HiPhys-RSI.
Our hybrid system combines pipelines constructed from classical perception, planning and control modules, and reasoning time routing mechanisms for selecting between existing pipeline and VLA strategies. By systematically comparing these with three VLA variants covering different training durations and task-specific adaptive settings, we found that in the BEHAVIOR-1K task category we evaluated, our approach demonstrated a broad performance advantage.
These findings support structured knowledge accumulation as a complementary direction for physically intelligent scaling, while coding agents can automatically complete some of the engineering work required to combine and adapt robotic capabilities.
Our analysis suggests that structured scaling moves from rapid improvement through a combination of capabilities to the exploration of more demanding alternative strategies. A wide range of task coverage will emerge at a relatively early stage, and disproportionate improvement work will be required in difficult cases.
Failure mode analysis suggests that Agent systems can solve many of the coordination and operation difficulties, but the improvement alone remains limited in visibility, physical accessibility, and execution budgets. Case studies further demonstrate how HiPhys-RSI gains unconventional strategies by changing contact location selection, object state, and movement order.
Future work should bridge the sim-to-real gap through a real-to-sim-to-real loop: using the observation and execution failures of the physical world to calibrate simulations, improve pipelines and strategies under changing conditions that are consistent with the real world, and verify these improvements on real robots. Subsequently, further simulation updates, code modifications and strategy adaptations can be guided by deployment differences, and HiPhys-RSI can be extended to real-world continuous improvements.
Additionally
A Definition of the task category
We have divided them into eight categories according to the main competence requirements of the mission.
A. Mobile retrieval:Search, locate and retrieve objects, including objects transported through rooms, doors or narrow passageways.
B. Generally placed:Moving objects to surfaces or into larger containers does not require specific matching or precise alignment.
C. Matching placement:Depending on the identity of the object, the type or the assignment, the appropriate target location for each object is selected. For example, bottles, cans and paper are classified into the corresponding boxes.
D. Detailed operation:For operations such as inserting, stacking, hanging or connecting, carefully control the position and orientation of the object, such as installing the camera on a tripod.
E. Tools used:Operating hand tools apply to targets, including brushing, cleaning, spraying, vacuuming or lighting.
F. Material allocation:An inclined container is a container in which the contents are transferred to another container or to a target object, such as pouring nutrients into a container.
G. Equipment operation:Operating mechanical devices, buttons or electrical appliances to alter the condition of equipment or objects, such as starting a washing machine, or using a stove to cook food.
H. Object deformation:Producing and tracking the shape of an object after processing or splitting, such as cutting an onion, or cutting a piece of wood in half.
B Definition of failure mode
We will execute failures in ten patterns.
How to cope with failure:It is not possible to establish a stable grip during the pick-up process, including failed claws closure or failure to achieve the required grip posture.
Putting the error:The object is placed or inserted into the wrong position, direction or depth, and fails to meet the required spatial relationship.
Lost objects:After the object has been seized, the object accidentally falls or is lost, such as during transportation or operation.
Failure of joint movement operation:Failure to interact with objects with moving joints, such as failure to open or close doors, drawers or lids as required.
Failed navigation:Robots are unable to locate or reach a target object or work area, including search failures, blocked routes, or unable to reach an operational location.
It's too late.Exhaustion of the available time budget prior to completion of the task may be caused by prolonged retesting, inefficient movement, or stagnation in execution.
Physical disturbances:When a robot or surrounding object collides, sticks, postures become unstable, or falls to prevent execution.
Planning mistakes:Inappropriate sequences of actions or decisions impede the progress of tasks, such as omitting pre-set steps, executing actions in the wrong order, or continuing to adopt ineffective plans.
System error:Where there is evidence to support it, the execution of interruptions will be attributed to supporting software or evaluation environments, including running errors, simulation crashes, API call failures, or simulation shots.
It's not the same as being wrong.A misunderstanding of an object's identity, location, geometric shape or state leads to an inappropriate choice of goals or actions.
These categories are not mutually exclusive: a single rollout may result in multiple failure patterns; overtime may also occur at the same time as delayed failure.
C Failure patterns by task
Figure 9 shows the failure patterns of each task.The columns in the graph correspond to task IDs, each row corresponds to a failure category. Deep color indicates the frequency of failure patterns in each task, and the deeper the color indicates the higher the percentage.
The vertical axis contains the following categories of failures:
- Catching the Failure
- It's too late.
- Failure of navigation
- Physical interference
- Putting the Mistakes
- Understanding the Mistakes
- Lost objects
- Mistakes in planning
- Joint movement fails
- System errors
The horizontal axis is the task ID, the range is 0×99;; the color gradient is 0%×100%.
