
"The world model can predict the future, why is the robot still "unable to move"?
LG-DKGP: Let the world model learn action continuity. The average success rate across five simulated tasks is 87.07%, which is 28.41 percentage points higher than LeWM.
Introduction
Give the robot a current picture, and then give it a target picture: Can it let it find a series of actions on its own to turn the world in front of it into the target?
Visual world models are trying to answer this question. It can "rehearse" the consequences of different actions and then choose an option closer to the goal. However, being able to predict the outcome of an action does not mean being able to propose a coherent action.
When pushing an object, the next step needs to continue the current direction of movement; when the robotic arm switches from approaching the object to grabbing, it needs to change the control method in time. Actions must maintain a necessary continuity but also change at the right moment. Without such a time structure, even with world models to assist assessment, planning may still take long detours and miss the target.
To this end, the DeepTouch-Lab team of Beijing University of Posts and Telecommunications proposed LG-DKGP (Locality-Gated Deep-Kernel Gaussian Process): learning how to continue actions from expert demonstrations to provide more reasonable candidate actions for world model planning. Among the five simulation tasks, LG-DKGP achieved an average macro success rate of 87.07%, which was 28.41 percentage points higher than LeWM’s 58.66%.
01.
Question: The world model "thinks", why may the robot still "not do well"?
World model planning usually generates candidate actions first, then predicts the state after execution, and finally selects the solution closest to the goal. There is a gap in the division of labor: the world model answers "what will happen if this is done", and the planner also needs to answer "what should we do next".
Taking PushT as an example, the robot needs to use a circular pusher to push a T-shaped object to a specified position and orientation. In the comparison of the same starting point and the same target shown in the paper, LeWM turned many times near the object and failed to complete the task within 50 steps; LG-DKGP approached and pushed the object along a more coherent path, reaching the goal in the 24th step.


The key is not to always minimize the difference between adjacent actions, but to retain the necessary stage switches such as turning, contact, and grabbing while continuing the intention of the existing action.
02.
Inspiration: What happens if you disrupt the order of expert actions?
We conducted a simple experiment: retaining all action values in the expert demonstration, only randomly shuffling the order of actions within the same trajectory, and then comparing changes in adjacent actions.
In the five environments of PushT, Cube, Caddy, Scene2 and RoboTwin, the adjacent changes of expert actions arranged in the original order are significantly smaller than after shuffling the order; the 95% bootstrap sampling confidence intervals corresponding to the differences in the five tasks do not include zero.
This means that expert demonstrations not only provide action samples of “what to do” but also provide a temporal structure of “what to do in what order”.
Just like writing an article: knowing each word does not mean being able to organize them into smooth sentences. For robots, choosing the right action is the first step, and it is equally important to connect the action to the context.
03.
Core method: Equip the world model with an action prior that understands "context"
Method framework diagram

LG-DKGP learns an action prior in addition to the frozen visual world model: it first predicts the next reasonable action based on the current and target images and the history of actions that have been performed; then the world model determines which candidates are closer to the target. The whole method consists of three links.
3.1 Look at “what you just did” and “what you were doing before” at the same time
Recent movements reflect current movement trends, and earlier movements may reveal the task phase: is the arm approaching the object, or has it already started carrying it?
LG-DKGP designed a locality gated history encoder. Local attention is biased toward recent actions, while global attention retains longer-term cues; a learnable gating mechanism combines current and target image features to dynamically fuse the two pieces of information.
In this way, each new plan can be based on the context of actions that have already been performed, rather than re-guessing the next step.
3.2 The neural network gives the main direction, and the Gaussian process fills in the time details
After the historical information encoding is completed, the neural network first predicts the main direction of the future action sequence; **Deep-Kernel GP** further learns the residuals that the neural network cannot explain and the correlation between different future time positions.
Predicted mean of future actions = Neural network action continuation + GP residual correction
The neural network is responsible for complex non-linear action patterns, and the GP is responsible for structured corrections with time dependence. The combination of the two makes the generated action sequence closer to the behavioral rhythm in expert demonstrations.
3.3 First propose reasonable candidates, and then let the world model find the target
LG-DKGP does not directly perform the prediction mean, but performs a cross-entropy search (CEM) around it. The initial exploration scale of the search comes from the statistics of changes in adjacent actions of experts in the training set, so candidate actions have both a reasonable starting point and room for further exploration.
The frozen world model predicts potential states after the execution of candidate actions, and CEM iteratively filters solutions based on the distance between the predicted state and the target state.
The action prior is responsible for proposing actions worth trying, and the world model is responsible for evaluating whether these actions can promote the achievement of the task.
In the main experiment, 25 actions are planned each time and a maximum of 50 steps are executed; CEM evaluates 300 candidates in each round, with a total of 30 iterations. After executing an action segment, the robot re-observes the environment, updates the actual action history, and then starts the next round of planning.
04.
Training: Let the model learn to "continue writing" expert actions
Action priors are obtained through two-stage training.
In the first stage, an expert future action supervision history encoder and a neural network mean head are used to learn the relationship between the current image, the target image, and the history of executed actions and future actions.
In the second stage, GP residuals are added to jointly optimize the history encoder, neural network and GP parameters to further fit the time-related structure of expert actions.
The expert's future actions during training are used as supervision; during online planning, only the current and target images and the history of executed actions are used a priori. The visual world model remains frozen during both training and planning.
05.
Experiment: Five simulation tasks, the average macro success rate is 87.07%

We evaluate LG-DKGP on PushT, OGBench Cube, LIBERO Caddy, LIBERO Scene2 and RoboTwin, and compare with LeWM, iCEM, PRISM, INTACT.
| method | Average success rate of five-tasks |
|---|---|
| LeWM | 58.66% |
| iCEM | 59.33% |
| PRISM | 61.66% |
| INTACT | 71.03% |
| LG-DKGP | 87.07% |
The success rates of LG-DKGP on five tasks are PushT 97.60%, Cube 98.80%, Caddy 83.32%, Scene2 76.96%, and RoboTwin 78.66%, among which four tasks achieve the highest average success rate. On Scene2, INTACT is 77.53% and LG-DKGP is 76.96%.
The most striking difference is for RoboTwin: LeWM has a success rate of 10.91% and LG-DKGP reaches 78.66%. This shows that under the same visual goal planning framework, the way candidate actions are organized and search initialized can significantly change the control results.
The macro average success rate here refers to the equally weighted average of the success rates of the five tasks. PushT and Cube are evaluated based on native task success criteria, while Caddy, Scene2 and RoboTwin include local goal achievement determinations defined in the paper.
06.
Why does it work? Ablation experiments tell us the answer
Ablation experiment diagram
First, GP residuals provide effective temporal structure correction. After removing the GP residual and retaining other planning settings, the average macro success rate dropped by 3.80 percentage points.
Second, local and global history need to be dynamically combined. Using only local attention, the success rate drops by 2.08 percentage points; using only global attention, the success rate drops by 7.58 percentage points. Recent movement trends and longer task phase information can affect the next action.
Third, the search scale needs to match the expert’s actions. Using a fixed scale, the average macro success rate is 74.92%; after switching to a scale obtained by statistics of expert action changes, it increases to 87.07%. On RoboTwin, this change increased the success rate from 18.88% to 78.66%.
Fourth, reasonable action priors and goal-oriented search complement each other. When performing the prior mean directly, the success rate of PushT is 58.40%; after adding CEM search, it reaches 97.60%. Caddy improved from 60.82% to 83.32%. Action priors provide a good starting point, and world model search is responsible for further selection.
There is another phenomenon worth studying: removing the visual conditions in the priori, the overall success rate drops by 8.86 percentage points, but the performance on Caddy, Scene2 and RoboTwin has actually improved. This suggests that the optimal fusion of visual targets and action history is task-dependent.
07.
Beyond success rate: Are the generated actions more expert-like?
The goal of action continuity is to preserve the action transition structure in expert demonstrations. To this end, the paper constructs features based on action increments of different time spans, and uses indicators such as MMD² to compare planned actions and expert actions.
In the scene-target query pairing experiment, the proportion of LG-DKGP closer to the expert action structure than LeWM reached: PushT 83.6%, Cube 90.0%, Caddy 99.5%, Scene2 98.2%, RoboTwin 100%.
The study also found that simply repeating the expert's first move 25 times resulted in lower MMD² on some tasks. This shows that a single distribution distance is not enough to describe the quality of the action: to judge whether the generated action truly conforms to the expert's behavioral structure, it is also necessary to combine the action change curve and the success rate of the closed-loop task.
08.
Technical Enlightenment: The bottleneck of the world model is not just in the "world"
This work reveals two distinct capabilities in world model planning: predicting what consequences an action will have, and proposing what sequences of actions are worth predicting.
Even if the world model is frozen and only the continuity structure and search initialization of candidate actions are improved, the control success rate can be significantly improved.
At the same time, expert demonstrations provide two types of information that can be used: one is the time dependence of the evolution of actions with task stages, and the other is the typical scale of action changes. The former helps predict the next action, and the latter helps determine the search scope.
Therefore, to improve world model planning, we can not only optimize the predictor, but also start from the action proposal mechanism itself.
09.
Next step: longer tasks, lower overhead, more realistic execution
Existing experiments mainly focus on the simulation environment and the 25-step action prediction window. As the target distance exceeds the prediction window, the planning success rate will decrease; CEM requires 300 × 30 candidate evaluations for each planning, which also brings computational overhead to real-time control.
The focus of the next step is to make the history length, planning window and re-planning frequency adaptively adjust with the task stage, reduce the cost of world model evaluation, and test the effect of expert action continuity in real manipulators and longer time series tasks.
10.
Summary: Let the next step follow the previous step
LG-DKGP converts the action continuity in expert demonstrations into learnable action priors, and then hands it over to the frozen world model and goal-oriented search to determine which action is worth executing.
The world model must not only be able to see the future, but also be able to accommodate the past.
Paper information
title: Rethinking Actions: Continuity-Aware Priors for World-Model Planning
author: Shaobo Yang, Zichen Ai, Ziyang Ma, Qian Yang, Bin Fang
paper: https://arxiv.org/abs/2611.16645
Open source code: https://github.com/oliverstack-byte/LG-DKGP








