Hanging a Mark Cup on the cup shelf, the success of the action is often buried in the first contact.
Holding the cup handle, the cup may be stable, but the hanger loses its entrance position; after holding the cup body, it is necessary to align the cup handle with the hanger correctly.
A suitable seizure needs to leave space to complete the task for subsequent actions.
Such requirements relate to the availability of objects (affordance): for the current task, which area of the object can support which interactions. This is also required for the process of generating robot-operated data based on the target access location.
Figure 1 places the dynamic contrast with the hanging task. The four outcome windows in the chart play a general crawling and AffordSim execution process respectively; the functional areas associated with the task simultaneously constrain crawling selection and follow-on alignment.
To address this problem, a team of Prof. Lanzhou Sims from the Institute of Artificial Intelligence and Robotics at Xi'an University of Transportation, the National Focus Laboratory on Humanized Mixed-Intelligence, proposed AffordSim, which would generate open vocabulary 3D Availability Prediction to access simulated trajectories. The system is based on natural language descriptions, organizing scenarios and tasks, positioning functional areas on the surface of objects, and then combining motion plans to generate executable robot operation demonstrations.
The article links to:https://arxiv.org/abs/2604.11674
The main page of the project:https://affordsim-platform.github.io/AffordSim.io/
The code repository:https://github.com/AffordSim-Platform/AffordSim
At the same time, the study constructed evaluation benchmarks covering 50 tasks, 7 operating skills, 5 mechanical arms, and supported an asset base containing more than 500 rigid bodies and articulated objects. For tasks requiring precise selection of functional areas, AffordSim reduced the need for manually specified contact locations per object and provided an experimental platform for assessing how to use this data for strategies.
Background and Challenges
To prepare an operational demonstration for a robot, the first thing to do is to determine how it contacts an object. Existing simulation data generation processes often use two types of methods: manually to adjust positioning for different objects and tasks, or to call a general crawling estimator to select the candidate posture by crawling for stability.
Artificial regulation can encode a subtle task experience, but it may need to be re-adjusted every time an object or an interaction is added. The general crawling estimator reduces this part of the work, without considering what to do after grabbing the object. For example, the contact location of a cup for transportation may prevent it from being hung on the cup shelf.
Adding semantic region filtering to the general scraping results can exclude some of the inappropriate candidates; however, if the estimator does not initially generate a candidate in the target region, subsequent filtering will not compensate for the missing scraping. AffordSim therefore requires the task to directly determine from where the robot attempts to contact the object before placing the functional region prediction in the candidate generation.
Contribution One: From task description to three-dimensional functional area to executable trajectory
AffordSim connects scene organization, availability prediction and sports planning.
First, the visual language model analyzes scenarios and task descriptions respectively. Scenario analysis extracts objects, properties, and spatial relationships, selects objects from the USD asset base, and assembles scenarios; task analysis breaks down targets into operational steps to generate functional region queries. For example, in a downstream task, the system distinguishes between containers held and reception containers to generate queries for different interactive roles.
Subsequently, the open-source 3D Availability Model receives object point clouds and queries to predict which surface points are related to the current interaction. Here, point clouds are used that are supported by simulated assets, so the system is able to restrict contact locations in 3D geometry.
Figure 2 of AffordSim's data generation process. Language descriptions translate from scenario and task analysis into functional area queries; 3D prediction results are bound to pick up candidates, and motion planning further checks whether candidates are reachable or collision-prone.
The first step is to determine which region to try to grab from: the system extracts representative contact points in the predicted region, combining local surface methods to generate candidate postures in different approach directions. The second step is to determine which candidate the mechanical arm can execute: call a cuRobo to plan the candidate's non-collision approach trajectory, and rate the length of the path by joint space paths.
The scoring rules are straightforward: when planning successfully and without collisions, the candidate scores as the path length counts; when it is not possible to find a feasible path, the score is zero. The system selects the highest scoring candidate, duplicates the corresponding approach trajectory, then clamps the claws shut, lifts and subsequent task movements, forming a complete demonstration.
This design simultaneously examines functional adaptation and the feasibility of mechanical arm execution, enabling the projected functional area to actually participate in trajectory generation.
Contribution 2 Assessment of available sexual perception operations with 50 tasks
AffordSim's benchmark covers crawling, placement, switching, pulling, hanging, dropping, stacking, and timing a combination of seven skills. Tasks can be run on five mechanical arms: Franka, Kinova, Sawyer, UR5e, and xArm7.
Figure 350 shows the dynamics of the missions, covering multiple object interactions, operation sequences and mechanical arms, used to evaluate trajectory acquisition and strategy execution.
The benchmark is further divided into three levels of difficulty according to functional area constraints and operation sequence requirements: Easy contains 10 tasks, with fewer restrictions on contact areas; Medium contains 30 tasks, with contact areas required according to object function selection; Hard contains 10 tasks, with further introduction of long sequence dependence or more stringent alignment and stability requirements.
This division allows the experiment to observe how the gap between general crawling, human experience and availability prediction changes as the task becomes increasingly dependent on the correct contact position. For multi-stage tasks, benchmarks also statistics the success rate of each stage to position failures occurring in crawling, alignment or follow-up operations.
Contributors 3 Replace real-life scenarios and support changing camera views
After the trajectory can be executed in simulation, it is also necessary to consider the differences between training images and real environments. AffordSim focuses on introducing real-world scenario background replacement with cross-camera perspective rendering in domain randomization: replacing simulation backgrounds with real-world scenarios such as coffeehouses, libraries, and factory workstations, with the premise of retaining robots and operational objects, to provide a different environmental appearance for operational demonstrations.
The system starts from 10 to 20 photos of each scene, using Deepth Anything 3 to restore depth to the camera position, and then builds 3D Gaussian Splatting.
The background can be switched with the camera's perspective: within the scope of reconstructed coverage, changing the position and direction of the virtual camera's observation, rendering the background according to the corresponding camera parameters, and synthesizing with simulated robots and operating objects from the same perspective. Thus, the same operating process can be observed from multiple camera angles, and the perspective of the background changes as well.
Figure 4 Conjoined variation of the real background with the camera angle. The three lines show the corner of the coffee shop, the library and the factory workstations respectively, with the camera angle of −90°, −60°, −30°; each line of images is taken from a synchronous camera image from the same operating trajectory and synthesized with the reconstructed background of the corresponding angle.
In addition to background replacement, the system also supports randomization of object positions, surface textures, lighting and image noise, so that training data simultaneously covers the environment, perspective and appearance of objects. Research further uses these simulation samples for the deployment of zero samples of real robots to test the migration capabilities of generating data.
The highlights of the experiment
The stronger the functional region constraints, the more obvious the benefits of availability guidance. In the 50 mission trajectory collection experiments, each method uses the same scenario and motion planning settings, with each mission running 100 times, and the object's initial positioning is randomized using a 30 cm horizontal position window with a 60° deviation angle range. The following table gives the average trajectory collection success rates for each difficulty level reported in the paper.
Trajectory retrieval success rate in Table 1. Artificial adjustment refers to the setting of contact positions and posture rules for objects; AnyGrasp+Filter adds task-related area filtering to general crawling candidates. It measures whether a demonstration trajectory will be retrieved successfully, rather than the success rate of execution of a learning strategy.
On Medium and Hard, AffordSim improved by 34 and 48 percentage points, respectively, compared to AnyGrasp+Filter. Compared to the AI-adjusted baseline, it achieved a recovery success rate of about 93 percent and 89 percent, respectively: 79 percent/85 percent and 64 percent/72 percent, respectively.
Gains were observed on all five mechanical arms. In cross-body experiments,ffordSim had an average collection success rate of 76% on 50 missions, 83% higher than AnyGrasp+Filter on each mechanical arm, 32% higher than AnyGrasp+Filter.
Data collection can be used for strategic training, and complex operations still have room for improvement. The study compared BC, Diffusion Policy, ACT, VLA-adapter andπ0.5 in 20 representative tasks, each using 300 samples. The overall simulation success rate of π0.5 and VLA-adapter was 55% and 54%, respectively; in the Hard task, it was 24% and 23%, respectively.
The real mechanical arm has confirmed the feasibility of zero-sample migration. The study will deploy the strategy using only the AffordSim operating demonstration training to the real Franka FR3, covering two placements, one dump and one hanging task, with no real robot data minimization 10 times per test.
The four real Franka FR3 missions in Figure 5 are arranged in a 2×2 order: left up for the banana plate, right up for the cup rack, left down for pouring the ingredients into the bowl, and right down for the Mark Cup hanging.
Table 2 zero samples simulate real migration success rates. 10 tests per mission; the accurate average value of the VLA-adapter is 22.5%, and the paper form is reported as 23% as a whole.
True machine results show that simulation data can support the direct migration of these tasks, while also presenting clear differences in difficulty: tasks involving precise attachments are still more difficult. Combined with simulation experiments, it is possible to continue tracking functional regional predictions, catch-up alignments and long-time sequences to execute their respective errors.
Significance and direction of research
AffordSim uses the task-related contact area as an explicit condition in data generation, connecting natural language goals, three-dimensional geometry, and mobility feasibility. It provides a pathway to reducing manual adjustment of object-to-object contact rules, as well as through a unified benchmark to assess the actual impact of robot operations through an understanding of availability.
Current results are also limited by availability model coverage.
For example, contact configurations in some suspended tasks are less common in model training data, and the collection performance is still less than man-made adjustment; transparent objects, new geometries, and blurry functional areas are also challenging.
