Can Models Perceive 3D Pose Adjustments? The First Interactive Projection-Alignment Benchmark Tests

  It's the first time in the history of the world.

After identifying an object, another angle can be identified and further imagined as it moves, all of which involve spatial perception. Human beings need to represent the shape and direction of the object in their minds, understand the relationship between different points of view, and deduce three-dimensional geometry from two-dimensional projection.

When a visual language model enters an interactive scene, this understanding also needs to fall into action. A rotation changes the outline, proportions, and shadowing relationships of the object; the model needs to predict the change, then observe the execution results, and decide how to adjust the next step.

There is a focus on this in existing benchmarks. Static benchmarks typically evaluate spatial reasoning by asking questions on a fixed image, lacking action execution and closed-loop feedback; many interactive benchmarks focus on task planning, tool calling, or path search. These tasks can examine the ability to interact, but it is difficult to specially test whether models can understand three-dimensional directions from visual cues and turn this understanding into precise rotation.

This is why we are here.Can visual language models translate two-dimensional observation into a three-dimensional understanding of precise action?

This requires models to associate the geometry and gesture of objects, predict the effects of rotation on projection, and continually modify new observations after action. For the intelligent body that needs to explore the environment, understand geometry and act, these judgments connect visual perception to real control, which is precisely the ability the industry wants to observe further through assessment.

For this purpose, a team from the University of Xiamen and the South China University of Technology has proposedRoTA-BenchThis ability is evaluated through interactive 3D-to-2D projection alignment.

Link to the paper：https://arxiv.org/abs/2610.17665

Code and data sets：https://github.com/MAC-AutoML/RoTA-Bench

On the InternetDemo：

Hugging Face：

Below is an interactive interface of a real-world mission in the RoTA-Bench. The image on the left is a T-shirt-shaped target outline, on the left below is the current projection of the object, and on the right is a view of the object with a deep, bright, dark view. The mission requires a selection of rotational movements to gradually align the current projection with the target outline.

RoTA-Bench is an example of a real-world task. The image is in the top left of the target outline, the bottom left is the current projection, and the right side is a deep, bright, dark assisted view; the bottom button is used to select the direction and angle of rotation.

This step should be turned 30°, or only 5°? The rules are intuitive, but each step requires linking a two-dimensional image to a three-dimensional gesture.

In the experiment, GPT-5.6-Sol had a 17% success rate on 100 test missions; 400 missions completed by 20 undergraduates together were all successful.

This gap is worth exploring further: at what stage did the model encounter difficulties? Does the outline look more like the goal, and does that mean that the three-dimensional gesture is also more correct?

For the previous T-shirt example, a left-to-left turn may change the two-sided outline, a upward turn may change the overall proportions; the same object may change in one direction, or a shadow may appear. The model needs to connect the projection in front of the eye to the three-dimensional posture behind it to determine which axis, in which direction, and at which angles should be turned.

In the RoTA-Bench, the model sees three pieces of information at each step: the target outline, the current projection, and a deep, bright and dark auxiliary view using the same perspective as the current projection.

The model was then developed from24 discrete rotation movementsIn the middle, choose the next step: three rotating axes, two opposite directions, paired.1°、5°、10°、30°After each execution, the environment repaints the image, and the model continues to adjust the history of the updated observations and interactions.

This requires a more substantial turn, and space is reserved for fine adjustment. When the direction is much worse, larger steps can be chosen; when approaching the target, smaller angles can be tried.

When the current projection and target outline converge, the IoU reaches0.90The higher the number, the closer the projection. The maximum 48 decisions are allowed, and there are also rules for active stops and stagnation terminations.

The researchers restricted interaction to pure rotation in order to distinguish three-dimensional reasoning from parallelism, grip, and contact friction, focusing on "how to control posture based on visual evidence".

If the usual three-dimensional objects are used directly, the model can be judged using familiar shapes.

In order to reduce these semantic shortcuts, researchers constructed three-dimensional objects within a spherical volume constraint from two-dimensional vector images. The generated objects were able to reproduce the prototype in standard positive projection and retain detail.

Then, with a series of reversible discrete rotations, the object is reversed to the initial posture. Since each step can be reversed, it is possible to rotate back to the target posture by reversal, ensuring that the task can be solved from the construction process.

On this basis, researchers further searched for and verified reference paths to select test tasks according to the reference path length layer. Reference paths used here for task screening and analysis are not guaranteed to be the shortest solution, nor will they be provided to the tested model.

Finally, the RoTA-Bench included674 missionsof them.100 missions comprise a set of freeze testsThe remaining 574 are reserved for further training and research.

The first half of the image shows three-dimensional objects generated from two-dimensional icons, initialized through reversible rotation, verified reference paths, and selected test tasks; the second half shows "Observer selects rotation to update the image".」It's a closed-circuit review.

The researchers evaluated nine visual language models, two intelligent-body configurations, and invited 20 undergraduate students to participate in human experiments.

On 100 test missions, the GPT-5.6-Sol success rate was17%In a human experiment, each participant completed a block containing 20 tasks; the 20 participants collectively covered the entire test set four times, for a total of 20 tasks.400 missions run, all successful。

Humans and models use the same tasks, visual observation, spinning operation spaces, decision-making budgets, and success criteria. Humans submit actions through buttons, models submit actions through structured output, and are eventually mapped to the same spinning operation.

RoTA-Bench overview (document diagram 1) The image on the left is a reference path length distribution, in the middle is a T-shirt task example, and on the right is a test set success rate and final IoU. Human results summarize four runs of each test task.

The average IoU of human trajectories increases overall as the movement progresses, eventually exceeding the successful threshold; the model trajectory enters the platform period earlier, with most stopping at around 0.50 times 0.63.

Average projection of IoU with changes in the number of rotational steps performed (Table 3). The result is based on the same set of 100 test tasks, including 400 human runs; the null-line marks a success threshold of 0.90.

To understand the gap, it is necessary to go back to each step: choose the wrong direction, choose the right angle, and see if the projection changes are consistent with the three-dimensional gesture.

The paper first examined the cumulative surface observation coverage: GPT-5.6-Sol is calculated as:79.58%Human beings are83.07%The overall coverage of the two is close, but it may be different depending on which regions and how these observations are used.

As a result, the researchers further divided each spin into two options:The direction, i.e. the axis of rotation and the positive and negative directions; the angle, i.e. this step changes significantly.

If there is at least one legitimate angle along one direction that reduces the three-dimensional gesture error, mark this direction as "effective direction".

GPT-5.6-Sol The proportion of effective directions selected is58.75%Human beings are83.91%The corresponding value of the uniform random selection is 53.78%. This indicates that selecting a rotational direction that can improve posture is already a problem facing the model.

Look at the angle. Even if the correct direction is chosen, too many or too few turns can affect the alignment. Researchers compared the angle of model selection with the legal angle that can minimize gestural error. GPT-5.6-Sol's average angle is less than 0.6.49°Human beings are2.99°The proportions of the best legal angle chosen are:61.80%and71.82%。

Diagnosis of orientation and angle selection, data extracted from the paper table 1. Angle-related indicators are calculated only when the selected direction is valid;」This is not a guarantee that the actual angle of selection will improve the posture.

Directional and angular errors together constitute a gap at the action level. Separating the two can be further judged: whether the model finds the appropriate direction of rotation, and whether fine adjustments can be made along this direction.

The researchers also analyzed the correlation between rotational angles and pre-action projection IoU, and three-dimensional gesture errors. For GPT-5.6-Sol, angles are more closely related to projection IoU; human selection angles are more closely related to both. This is consistent with the relative patterns of most models in the paper, suggesting that the model's angle selection may be more associated with visible projection overlapping.

Spearman correlation coefficients between the rotation angle and the projection IoU before the action, 3D gesture error, absolute value (Figure 4). Blue represents the IoU angle correlation, orange represents the gesture error angle correlation; the error line is mapped to the absolute value after the original 95% bootstrap interval. The diagram shows the strong correlation and does not directly reveal the judgement mechanism within the model.

After rotation, the current outline overlaps more with the target, which seems to be a good signal. However, different postures of three-dimensional objects may produce similar projections, and local improvements in two-dimensional images are not always synchronous with reductions in three-dimensional posture errors.

To compare this phenomenon, the researchers found humans on the trajectory of GPT-5.6-Sol.211, paired with the same gestureIt's a great way to observe the movements of each of them.

In these pairs, the model has19.4%The movements that improve the overlap of the outline, but alter the alignment of the posture; the proportion of human responses is10.9%Conversely, humans are also more likely to temporarily reduce projection overlap, but reduce 3D gesture errors by 33.2% and models by 23.7%.

211 for the results of the movements in the matching state (Figure 5). The left-hand side of the image shows the ratio of four types of movements: UC simultaneously improves IoU and gesture,DCOnly improved posture, UF only improved IoU and DF both varied; the number of rotations before the movement was summed up on the right, the surface coverage and the average change brought about by the movement.

It is therefore possible to distinguish between "looking more like" and "gesture closer". Here the current gesture is matched, and previous observations and history of movement may still be different; these results describe the differences between human and model movements in a shared state.

Eventually, the task is still projected IoU as the success standard, and 3D gesture errors are used for additional diagnosis. Due to geometric symmetry and projected overlap, success does not require objects to return to their unique gesture.

The interaction also means that the model can compare the images before and after the action: how much has just been turned, how the outline has changed, and whether the next step should be to shrink the angle.

In the same group of 30 missions, the researchers conducted an input-melting experiment on GPT-5.6-Sol.5/30After removing the history of interactions, keeping current observations and mission descriptions, the number of observed successes decreases to0/30。

There is also a significant change in the choice of action:The proportion of 30° rotations increased from 62.75% to 96.76%, and the proportion of 1° adjustments decreased from 12.18% to 0.16%.The fine tuning almost disappeared.

GPT-5.6-Sol is the result of input dissolution on the same set of 30 test tasks (Table 2). A is the complete input, B is the removal of the auxiliary view, C is the removal of history, and D is the simultaneous removal of both; the table simultaneously reports the success rate, the final IoU, the effective direction selection rate, and the proportion of movements from all angles.

In this group of experiments, the impact of historical information on "multiple angles of rotation" was more apparent than on "which direction of rotation". Previous observations and movements may have helped the model determine when to make more detailed adjustments after a larger rotation.

These results show that the model's precise controls in the measured settings are sensitive to historical interactions.

Go back to the beginning of the T-shirt. To complete the task, you need to extract clues from the target and the current image, deduce gestures, select rotations, and then modify them according to feedback. RoTA-Bench puts these processes in the same closed loop, both to see if there is a alignment at the end, and to see how each step in the middle selects direction and angle.

Experimental results show that observing, perceiving and adjusting the position of 3D objects remains a challenge for the tested model.

In the future, the controlled task could help researchers observe the problem more specifically and provide a verifiable assessment environment for subsequent model improvements.