Multi-modal large-scale systems are being asked to work across domains, inter-tasks: understanding spatial relationships, planning body movements, understanding the timing of videos, determining whether an object will fall, etc. These tasks fall into different domains, and research also has a variety of scales, benchmarks, and approaches, and methods that work in one field rarely move to another.
However, all of these tasks involve how the world will change with the action, which is what everyone is talking about. The failures recorded in all the downstream areas are the lack of the ability to model the world.
What does world modeling mean? Is it an ability to measure independently, rather than a superimposition of spatial, physical, physical, and time sequences? If it can be measured, is it possible to learn the current model?
The team from Northwestern, CMU, UNC Chapel Hill, and Amazon explored these issues one at a time in their paper WOVEN: Weaving Visual World Modeling into Multimodal LLMs: defining this capability independently, then measuring it independently, teaching it, finally testing what can be learned about cross-tasking replication, and what kind of training data can be used to replicate it.
The paper:
The main page of the project:https://woven-ai.github.io
The code is:https://github.com/mll-lab-nu/WOVEN
The data:
Everyone says the world is modeled, what does it mean?
The term world model is widely used, with different definitions in different works. The team narrowed it down to the most central layer: state transfers. The current state s, an action a, followed by state s′, is an estimate of p (s′, s′, a).
Multiple tasks, such as space, body, physics, and time sequence, are unified under this structure to perform different queries for the same triad group. Knowing s and a for s′, is the result of predictive action; knowing s and s′ for a, is the judgment of what happened; changing a to ask what would happen, is counterfactual; ordering a series of intermediate states, is time sequence.
Compared to the post-hoc interpretation, the team chose a more rigorous sequence of experiments: First, there was the ability to define and measure without relying on any downstream task, and then there was the lack of a worm model.
Second, modeling the world as a controllable experiment.
Real video cannot do this: change how an action will be done in the same scene, no one has shot it; controlled disruption of the scene under the same action, no one has done it. The team's practice is to use the video generation model as a disruption engine: give a set of initial images and a description of the action that occurs after it is generated; change several movements from the same set of movements to get a real controlling counterfactual trajectory.
Starting from the same initial image, perform four different moves and get four different results:
Three, what's missing from this model?
The team measured 38 front-end models, with the strongest GPT-5.4 responses at 65.8%, compared to 92.3% in humans. The scores themselves were more interesting, the shape of the gap being the same in all model families; for example, the weakest of the adjacent time sequences judging the cluster categories, stopping at 25% and 28%, close to a four-way level, and the same set of models doing the same time sequencing can be up to 63%; changing the object's color (external disturbance) models are basically unaffected, changing the object's position or toward (geometrical disturbance) results falling to 30%.
This cannot be solved by scale alone. On the Qwen 2.5 VL scale from 3B to 72B, the apparent robustness is 31 points per ten-fold parameter, while the geometric robustness is only 9 points; the same 7B model, with an apparent disruption of 50.5%, a geometric disruption of 23.5%, is unlearned.
The team also found some other interesting conclusions, such as the order in which the model mastered the principles of physics on a scale consistent with the order in which the babies learned: first the lines were gravity and inertia, then inward concentration, persistence and containment, then support and solidity, and finally the collision, which was consistent across the model family; 38 models averaged 31.4% of collisions and were always offline.
The shared failure distribution pattern predicts that this is a training problem, not the bias of which model.
Fourth, can this ability be learned, can it be learned, can it be learned effectively, and can it help with downstream tasks?
The world's modeling capabilities themselves can be independently learned. The team trained on WOVEN with SFT+RL models of various sizes and families, including Qwen 2.5-VL-3B, with an accuracy range from 26.4% to 89.3% in domains, and 88.2% in unseen scenarios, and 92.3% not far from humans; 7B, 32B, and Gemma 3 were the same.
The whole set of training enhances 12 independent external measurement benchmarks that span multiple fields of physical understanding, time-order understanding, physical operations, and spatial reasoning; if the migration results are combined according to different controlled sub-sets of the same world modeling capability, the number of enhanced external independent measurement benchmarks increases to 22, although each controlled sub-set has only 2000 data. These zero-shots move up to a maximum of 27.3 points.
Five, world modeling is basically a skill set, or a stack of tasks.
There are 22 benchmarks, and will it be just that different parts of WOVEN each serve similar tasks, and together they seem to cover a wide range of common underlying capabilities?
The first is that different forms of supervision sources need to be able to practice the same capabilities. The team did a detailed analysis of the training results. The 11 controlled sub-sets, each of which elevated the type of action it had never trained before (e.g. data training with camera movements only, object operation questions +39.8, navigation questions +26.9).
The second is that the downstream revenue needs do come from this capability -- rather than the superimposition of various forms of supervised source training effects. The 11 subset of the migration curve two at 26 parameters is completely correlated; the more a subset improves the world modeling capability, the greater the average increase on the external parameters, r = 0.86.[4] The static perception and general video QA parameters are not consistent with any training, so the overall performance of the simulation model is not stronger.[5] The team did a harder set of controls: they constructed a virtual world simulation, in which the interaction of motion and results follows a fixed but erroneous relationship (e.g. a left-turn always leads to seeing all objects on the right side), maintaining the same set of parameters (e.g. a set of statistical modeling and video QA parameters) at the same time, so the simulation model is not consistent with any training conditions.
Finally, the team made a more direct causal intervention. Fixed training budgets, replacing a substantial proportion of the downstream task's own training data (30% to 50%) with WOVEN data, after training in corresponding downstream measurements, the accuracy of the results can be maintained substantially, and higher than the control of simply removing that part of the data (and therefore not in-domain data). Transfer oversight replaced that part of the downstream task that was originally to be done from its own data physics.
Using WOVEN to replace part of the training data of the downstream task itself (blue line), as opposed to directly removing that part of the data (three downstream tasks are SAT, ActionEQA, CLEVRER, across three different domains).
Six, how to train it is generally useful for the downstream: learning is more important than seeing.
The team further explored a more useful question: Since world modeling can be directly trained and can help with a variety of downstream tasks, what kind of transfer oversight should be selected in advance to train this underlying capability of world modeling in order to get the most out of it if specific downstream tasks are to be upgraded?
The intuitive answer is to challenge the pictorial: scene, action, field, etc. However, surprisingly, the experimental results are not. After removing the common migration component, the correlation between the subset of the same reasoning operation and the downstream gains taught is much higher than between the subset of the only shared action type (−0.22), while the latter is no different from the subset that doesn't share anything (−0.23).
In other words, it's not a matter of who the data will help, it's a matter of what the reasoning is for the transfer group.
In addition, the team summarized other conclusions: to be robust, prioritize greater supervision of changes in visual states; direct supervision of time-ordered tasks and object-driven tasks; no apparent migration between active and passive physics driven operations; the ability beyond state transfers, such as static perception, cannot benefit as downstream tasks from state-driven supervision, requiring alternate supervision.
However, these formulas were refined from the results already available, so the team did a further foreseeable verification: first according to formula formula training data, then according to seven parameters that were not used at all in formula introduction: MMSI-Bench, VLM4D, PhysBench, SPAR-Bench, MVBench, IntPhys 2, CameraBench, and up to eight randomized varieties. All five groups of directions and formula predictions: in the subjects, the mix of formula problems is significantly higher than in other formula types; in the dynamic physics problems, such as formula predictions, a more compact set of dynamic physics-dominant mixtures.
Two correlations of the 11 subset migration curves. Left: the original correlations are all positive; middle: grouping by type of action after removing the common component, not seeing the structure; right: grouping by type of reasoning, the structure appears.
Seven, what's in it for you?
To extract the process of this paper, the process is as follows: to explore whether world modeling is an independent of downstream tasks, first dismantle its variables and allow each element to make its own variations to provide an intervention experimental infrastructure; then use cross-domain testing, error control and causal intervention to eliminate surface correlations; isolate substrate capability scales and scales with rigorous causal intervention and statistical analysis; advance verification of the proposed training formula with unprecedented tasks.
For training systems, this means that world modeling can be organized by reasoned operations rather than by the types of tasks, domains, actions or scenarios.
Of course, migration is not the same as unconditional migration. Time sequence tasks, subject-driven tasks, static perception, are places where transfer supervision is difficult to directly help or limited to help, and the paper reports these boundaries and positive results together. The formula is therefore a specific rule that guides data selection, rather than a sentence.
About the team
This paper is from a collaborative team from Northwestern MLL Lab and CMU, UNC, Amazon, which includes Zheyu Fan, Yue Zhang, Mingkai Deng, Kangrui Wang, Qineng Wang, Canyu Chen, Jie Hao, Xing Fan, Chenlei Guo, Eric Xing, Mohit Bansal, Manling Li.