From "Show the end to the robot".
To "Tell the robot how to walk".
"Go straight along the corridor, pass the glass wall on the left, turn left at the corner, and continue to stop at the wall".
When it comes to guiding people, this is often enough. But for a robot, it needs to adapt information such as "glass wall", "left turn" and "wall edge" to the immediate environment and then transform it into continuous movement and rotation.
Existing target-condition visual navigation methods often use a terminal photo to tell the robot where to go. This setting is not necessarily convenient when used in practice, as users do not necessarily have photos of the destination, but are usually able to describe targets and references along the way using language.
Around this need, we proposed the LCVN language-conditioned visual navigation task, built a supporting dataset, and compared two routes that combined world models with navigation decisions: one learning strategy in a model-imagined environment, and the other using a unified model to combine action and imagery predictions.
The paper:
The code is:https://github.com/UWMILab/LCVN
What are you doing here?
The data collection:https://huggingface.co/datasets/fly1113/LCVN
The competition:
The scoreboard:
The mission is set:
Just look at the starting image and the language instructions and see the full navigation route after the forecast.
The LCVN mission input has only a first-person image of the starting point and a language instruction. The model needs to predict the complete sequence of movements (including plane displacement and directional changes) based on both inputs until the end point described in the instruction is reached.
There is a key setting here: there is no real-world feedback for correcting errors during the prediction process. The model can only continue to evolve based on its predicted state and ultimately evaluate the results on real robot trajectories.
Therefore, the LCVN focuses on the ability to predict off-line trajectories. Even if a step goes wrong, subsequent images and actions can follow deviations, which makes the collaboration between language, environmental prediction and route planning extremely important.
The data collection:
The same route, three instructional styles
The LCVN data set contains 39016 trajectories, 117048 trajectory instructions that have been manually verified, trajectories from five datasets: Go Stanford, ReCon, SCAND, HuRoN and TartanDrive. Among these, the first four data sources are used for training and field evaluation, while the cross-sectional scenarios of TartanDrive are used to test the generalization capabilities of the model in unfamiliar environments.
These three instructional styles also raise a real question: Is it better to give a robot direction in detail or to provide clear references more effectively?
Two routes:
How to Combine Imagination and Decision Making
1. LCVN-WM + LCVN-AC: Learning to move in an imagined environment
The first route was completed in collaboration with the World Model and Strategic Model.
The world model LCVN-WM first extracts image characteristics, then combines historical state, action and language instructions to predict subsequent scenario changes. The language instructions enter the spread model through cross-attention, allowing the prediction process to continue with the goals and routes in the reference instructions.
On this basis, the decision-making model LCVN-AC learns strategies entirely in the subterranean space of the world model: it selects moves, then the world model predicts the changes brought about by the moves, and then rewards based on the closeness of the predicted state to the expert trajectory. During training, the model also learns to infer potential plans from the current state and instructions and align with the plans extracted from the full expert trajectory.
The world model is thus directly involved in strategic learning, which predicts future states that will affect how actions are optimized.
2. LCVN-Uni: Combine predictive action and image in the same model
The second route LCVN-Uni is based on the multi-modal self-return model of Anole-7B, encoding images, language, and decentralized actions as a unified token sequence.
LCVN-Uni will place the next action with the next set of observations in the same joint prediction, and together optimize the action with the image prediction goal. The images generated will then be input as the next step, pushing the entire trajectory forward.
The difference between the two routes is clear: the former trains another strategic model in the world model, while the latter allows the same model to learn motion and visual prediction together.
The results of the experiment:
Predictions are more coherent and generalized, with a focus on each.
The experiments evaluated the navigation success rate, trajectory errors, and image quality at the same time. The mission success here refers to the distance between the predicted endpoint and the actual target that is less than the average step length of the data source.
The LCVN-WM + LCVN-AC success rate was 43%, the LCVN-Uni success rate was 42%, and the NWM + LCVN-AC combination with language conditions was 29%.
In the unfamiliar TartanDrive environment, the advantage of LCVN-Uni is even more apparent: it has a success rate of 27%, up from 19% for LCVN-WM + LCVN-AC. In the mixed test set, the two routes reached 36% and 34% respectively.
LCVN-WM is better at maintaining structural and temporal consistency, with SSIM@8 reaching 0.293 when continuously predicting eight steps; LCVN-Uni is 0.218, but performs better on the DreamSim indicator.
These results show that visual prediction and navigation skills need to be examined separately, and looking at a single indicator is not enough to judge the overall capability of the model.
How does language affect navigation and future image prediction?
To investigate the impact of language guidance on navigation and world models, we compared the performance of adding and removing language conditions in the world model. The results showed that in the absence of instruction guidance, the navigation success rate for both routes and the sequential screen prediction quality decreased. Among them, the success rate for system verification sets in domains decreased from 43% to 35% after the language conditions of LCVN-WM were removed.
The following illustrates this difference. For the instruction "Go forward along the corridor until you see the orange wall on the right", both models are better able to retain the scene clues associated with the target in continuous predictions after the language conditions are added. Language here effectively assumes the role of target description, so that the predicted state of the model is consistent with the destination and provides a more stable basis for subsequent actions.
What are the other issues that need to be addressed?
The paper provides an example of a failure: LCVN-Uni judges the direction of rotation but does not correctly generate the corridors and target white doors that should appear after the rotation, and subsequent actions therefore lose a reliable visual basis.
While LCVN provides a common testing platform to explore how to combine imagination and decision-making, real robot closed-loop verification remains to be done. The current instruction also covers only three predefined styles of English expression.
Introduction by the author
The paper is a collaboration between researchers at the University of Washington, the National University of Singapore, Drexel University, Clemson University, and the Microsoft Research Institute. Yifei Dong, co-authored by Fengyi Wu, co-authored by Zhi-Qi Cheng, co-authored by communications. The work focuses on visual navigation guided by language, and the combination of world models with body decision-making.https://roboworld2026.github.io/track1
