Behavioural space benchmarks: Assessing the understanding of space by robots
"Ask, Mean, Do" is a figure of speech.
The VLA (Visual-Language-Action) model is almost entirely based on a hypothesis: the space understanding of the pre-trained VLM protagonist will automatically migrate to the behavior of robots.
The strategy may simply remember the familiar layout of "which object is where, who is to be captured" without understanding the spatial relationship "second from the left number"; or it may choose the target but fail to execute.
The Stanford Lee Flight team built BEHAVIOR-Spatial on top of BEHAVIOR-1K, breaking it down into three layers of evaluation of the same scene:
Space questions can't answer "which is the biggest"), point-level goals can't refer to "which is the biggest"), robot actions can't really get enough of it, sharing the same sets of objects and spatial relationships.
The results of the experiment showed that the baseline strategy had a soft success rate of 41.6% in all new scenario configurations, but in a trained layout, in other words, it dropped to 16.1% and 70% of the rotation was wrong enough: the strategy relied on the "scenario-action" link, not the instructions themselves.
Behavior-Spatial will link visual Q&A, understanding, targeting, positioning, and executing actions to test whether VLM's spatial capabilities can predict VLA mission performance.It's a great way to get started.
Why Mission Success Is Not Enough
When a robot "stretches its hand to the largest plate" and succeeds, you don't know if it really understands the "largest" plate, or because the plate is exactly where it's always been.
Q: Understand: only the model can answer the question of space;
Ground: See if it can drop the "target" to a specific point/region;
Act: See if it can actually be executed.
While these three levels may contradict each other, the answer is that they do not represent precision (the choice of the subject and the positioning are two capacities) and that precision does not represent to be done (the execution is another level of difficulty).
BEHAVIOR-SPACIAL puts these three things back into the same set of scenes, the same set of objects, the same set of spatial relationships, so that they can correspond one by one.
▲Chart of BEHAVIOR-SPACIAL data set showing left: coverage direction, distance, size matching task VQA, pointing to action examples, navigation showing three contextual perspectives and current perspectives; right: spatial classification, VQA numbers, pointing/intermediate representation and task coverage.It's a great way to get started.
Data set: three-tiered suite + diagnostic evaluation protocol
The data is divided into three parts:
BEHAVIOR-Spatial-VQA ((3,256 space problems, 49 activities, divided into 1,572 general + mission related 1,684, including single and multi-tasking);
BEHAVIOR-Spatial-IR: 14 manned missions, 1,240 targets pointed to queries;
BEHAVIOR-Spatial-Action: 14 manoeuvres + 10 navigation missions)
All are automatically generated from the BEHAVIOR-1K demonstration using simulated geometry + elemental data, with visibility checks and relationship marginal filtering to control biases.
▲Graphic data build pipelines. Demonstration observation and simulation geometry support common VQA and select space tasks; task instances generate task-level VQA; point to labels and VLA data sets; select standard guiding screening, and test samples for each set.It's a great way to get started.
There are two types of evaluation protocols:
Held-out: Unseen new scene configuration + Unseen instruction variable, to measure "generalization";
Prompt-probe: The training layout is inactivated, replacing the instruction with another legitimate target and testing "if you don't listen".
The action indicators are also very subtle: soft success (a proxy indicator with the right goal intent), strict success (a true accomplishment of a task), failure rate (success of an unspecified goal).
Soft/Hard/False targets are three sets of targets, separating right and wrong altogether.
Models and experiments: key findings of several anti-intuitives
VLA uses the π0.5 architecture, replaces the Qwen3.5-4B mainframe, freezes VLM+ from the initial training stream matching movements; also measures LoRA adaptation, DROID pre-training initialization, and three oracle targets pointing to interfaces (point IR, point IR+ head, attention chart IR).
▲Drawings of the BEHAVIOR-Spatial-VLA model architecture. Various task views share Qwen 3.5-4B mainframe networks with flow matching action experts; only manipulation tasks are available with optional real-valued target positioning inputs.It's a great way to get started.
VLM space VQA: single-engine still going, multiple crashes
Of the 14 VLMs ((5 closed-source + 9 open-source), the best single pump is the closed-source GPT-6 Astra ((82.1%), with the strongest open-source Molmo2-8B at only 51.8%..
Many open-source models fall directly below 25% of the random conjecture on directional questions, while the skeleton extracted from the VLA is generally lower.
Explain that the general VQA rating is high and not equivalent to mission-level space perception.
▲Table of VLM accuracy on BEHAVIOR-Spatial-VQA.© Deep Blue and IntelligentCompiled
Action is more of a disaster zone: in other words, success rates go up and down.
Freeze the VLM baseline: leave the consolidated success rate at 41.6%, strictly 27.9%, and miss target at 38.2%;
However, the prompt-probe success rate was only 16.1%, strictly 10.0%, and the error rate dropped to 70.0%.
It is often not the goal of new instructions, but tied to the training and the scene. "Can't see the layout generalized" and "Listen in the familiar layout" are two separate craters.
The former is still able to collide, the latter almost completely overwhelmed.
More importantly, there are two subsequent sets of controls:
LoRA adaptation/DROID pre-training: almost useless.
prompt-probe soft success 16.1→17.9%, 16.6%, strict success 9.6%/8.9%, and even without the super-baseline 10.0% straightening of the spine can not cure the disease of "disobedience to instructions";
Oracle's goal is to see the shadows.
Point IR pulled the soft success rate to 60.3%, the probe pulled to 38.0%, and the error rate dropped from 70% to 43.9%. The problem is not "don't execute", but "don't know where to go at all".
▲Tables of all strategy variables are the success rate of the two assessment sets. +Soft = Soft Success Rate; S = Strict Success Rate; W = Proportion of submissions to unspecified objects.It's a great way to get started.
Perception-action relevance: Indicators associated with strategies are "generation", not "response".
The paper uses Spearman to quantify VQA/IR scores and the relationship between strategic success (Table 3).
For the two strongest strategies (point IR and attention chart IR), accuracy is more closely related to success than answer accuracy (ρ=0.701 vs. 0.560, 0.753 vs. 0.608).
In other words, the real predictor of robot behaviour is "the point at which the VLM generates" (location) rather than "the option it has to select" (Q&A).
This correlation varies by strategy, with the accuracy of directional questions in navigation missions and the success of 10 navigation missions being almost irrelevant (16.7%-40%, weakly correlated).
▲Spearman correlation coefficients between VQA or poor IR accuracy and tactical success rates on 14 manoeuvring missions.It's a great way to get started.
The code is a strategy: the code that is written will also be "lost".
A code-as-a-strategy agent (which uses language models to write Python to control robots) was also measured.
The closed-source author software model has a success rate of 50x66%, while the open-source author has only 37x39%; however, the open-source model 83x87% reversal is "moving toward the target but never interacting", with a strict success rate of 5.1%/5.3%, while the closed-source Gemini/GPT-6/Fable can reach 33x51%.
Explaining that "soft success" is just an intention agent, there is a big gap between the actual completion of the task and the same indicator, and the "unfinished" ratio behind different agents varies greatly.
▲Table code is the performance of strategic intelligence. +Soft = soft success rate; S = strict success rate; W = the proportion of bits submitted to unspecified objects.It's a great way to get started.
Summary
BEHAVIOR-Spatial re-examined an industry-defaulted phrase, and the result was that "VLM space capability → VLA behavior" was an unimaginable path.
The signals it gives are clear: first, changing targets is a common phenomenon, and re-engineering (LoRA, DROID) can't be saved, and the real use is to "indicate clearly"; second, it can predict behaviour by "positioning" rather than "question", as the VLA that generates the query, rather than scrolling the selection points.
Of course, the simulation of verification, privilege marking, and the intention to represent these borders are also a reminder that this benchmark is more like a "physical report" that identifies the real question of whether space understands whether it is in action, but the answer is far from it.
Related research: Lifeifei team's new work: the first physical-spatial intelligence benchmark to close the perception-action cycle
Edited by Dr. Little Monkey
I'm not sure what I'm doing.
Ref：
BEHAVIOR-Spatial Benchmark: Evaluating Spatial Understanding for Robotics
Link to the paper:https://arxiv.org/pdf/2610.34051
