Going into the era of physical AI is a key step in building a model of a polymorphic base that can truly unify language and vision. However, language is essentially an orderly Token sequence, while vision is a continuous high-dimensional distribution, both of which have a natural gap in information structure. How to unify these two distinct distributions within a model has been a hotly debated topic.
Currently, the mainstream solution consists mainly of two solutions. One is a fully discrete route (such as Chameleon, emu3), which also quantifies images into discrete tokens, which together with language can be modeled as self-regression. However, this method is limited to visual Tokenizer, where high-frequency details of images are very easily lost during the quantification process. The other is a mixed discrete route (such as TransfusionJan, usFlow), where text diffuses and images diffuse continuously. Although avoiding the quantization bottleneck, a model can maintain two sets of distinct target functions and sampling processes at the same time.
If there is a compromise between the two, is there a third solution?
Recently, a team from the University of Science and Technology of China, Wang Xinggang, in collaboration with institutions such as the Horizontal Robot, proposed Multimodal Flow, giving the third path: full continuity. Under this architecture, language and vision respectively remain in the original continuous embedded space, but share the same Flow Matching goal. This means that the model does not have to be forced to translate visual into language, nor does it have to maintain two systems that are cracked, and only uses a single Flow model to unify the distribution of the two modules. Experiments have shown that Multimodal Flow not only exhibits a strong multimodal generalization capability, but also efficiently migrates knowledge learned during the mixed pre-training phase to the task; under the same budget, it is more capable of overcompensing and disseminating mixed and fully distributed data in the same budget.
Multimodal flow: unified flow modeling of language and vision in embedded spaces
Address of the paper:https://arxiv.org/pdf/2609.40362
Open source code:https://github.com/hustvl/Multimodal-Flow
The main page of the project:https://hustvl.github.io/Multimodal-Flow
What information does the 1.6B model show?
The team exemplified the main model MF-1, with only 1.6B parameters, and was pre-trained from scratch, consuming only 150B Tokens. Compared to the current trillion-dollar Token model, this amount of data is quite limited, but the results are very prominent.
In terms of text graphics, the MF-1 scored 0.821 on the GenEval integrated score, and the DPG-Bench long pointer coordinate score was 83.44. Especially in GenEval's Counting (counting) dimension, it reached 0.73, significantly ahead of similarly sized unified models. In terms of multi-modal understanding, its VQAv2 scored 72.6, MMBench scored 67.2, POPE scored 86.1, three average scores reached 75.3, and even surpassed the bulk model that was initially matched with some pre-trained LLM.
More importantly, under controlled contrast with the same data, the same parametric budget, the same optimized configuration, Multimodal Flow has surpassed Transfusion's hybrid architecture and Chameleon's fully discrete architecture in various metrics such as GenEval, GQA, VQAv2, MMBench, etc.
How can we put language and vision into the same flowing hole?
Since the results are so good, how exactly does this whole series of architectures work?
Since the language and visual structure were different, the team simply left them in their original form. The text was divided into each of the eight tokens, and the frozen T5-small encoder became a continuous text superblock; the images were transferred to the frozen SigLIP2 encoder, which extracted 16×16 spatial grid characteristics and turned them into a visual superblock. These superblocks were organized into a multi-module sequence in the order set by the task, and sent directly to the shared Chunk-Causal Flow spine network.
The design of this backbone network is very clever. In the attention mechanism, it uses the Chunk-Causal disguise: when the model predicts the k-superblock, all the superblocks in front are fully visible, the current superblock is modeled together, while the future superblock is concealed. This both ensures causality and allows full interaction of information within the current block. At the same time, the network uses a mixed strategy in computing: attention-sharing to promote cross-modal interaction between language and vision; while the FFN layer remains independent because the statistical differences between language and vision are so large that sharing would be appropriate instead.
In training and reasoning, the architecture is extremely efficient. In training, the model can simultaneously predict multiple target superblocks, with each superblock sampling the flow time step independently; while in reasoning, it is generated in the order of blocks, and the completed superblock KV cache is directly replicated, greatly accelerating the generation process.
A very elegant result of this design is that as long as the sequence and target direction of the superblock is changed, the same set of network parameters can switch tasks seamlessly. Whether it is pure text generation, text graphics, graphics, or VQA, it can be uniquely expressed as a conditional generation of the superblock sequence.
Mixed Pre-Training: The Baseline Value of a Fully Continuous Architecture
The team trained three-dimensional models of 0.6B, 1.2B, and 1.6B, observing their evolutionary trajectories under the same mixed-pre-training protocol (Figure 4). The results showed a healthy initial expansion trend: as the model evolved, the number of trainings increased, the target losses of Flow Matching and the confusion of language models decreased, while the GenEval and CLIPScore indicators of graphical biology rose steadily.
Looking back at the migration, they compared the two models of a randomly initiated direct downstream tuner and a mixed pre-training tuner. The result was remarkable: GenEval jumped from 0.527 to 0.821, SEED-Bench jumped from 31.6 to 62.4, MMBench from 36.0 to 67.2.
This set of data strongly demonstrates that, under the goal of unified Flow Matching, the model can efficiently migrate text, images, and graphics learned during the mixed pre-training phase into downstream tasks. Multimodal Flow is not only a multi-mode generator, but also has great potential as an excellent multi-mode pre-training base.
Two discoveries in the smelting experiment
In the process of verifying the design rationality, the team conducted a number of meticulous melting experiments, two of which were very noteworthy.
First, what kind of visual representation does the model need? The team compared semantic features such as SigLIP2, DINOv2, and FLUX.2 VAE, SD-VAE sub-variables, and primitive pixels. The result was that, under the same training budget, SigLIP2 and DINOv2 performed better overall in generating and describing tasks, while the VAE sub-variables that were geared toward rebuilding fell short. This provided an important hint to future multi-modal generation: for unified lifetime models, semantic features tend to be more important than rebuilding reliability of single-purpose pixels.
Second, the same CFG can work in both directions. Traditionally, Classifier-Free Guidance (CFG) is used mainly in visual generation. However, thanks to the unified Flow Matching goal, the same CFG mechanism in Multimodal Flow can work on both text and images simultaneously. Experiments have shown that the best Guidance Scale for graphing tasks is around 3 and graphing tasks around 5.
A simpler model for a longer future
Multimodal Flow and MF-1 show one possibility: we don't need to force visual translation into language, and we don't need to maintain two separate generating targets. In continuous embedded spaces, language and vision can be modeled using the same dynamic systems while maintaining their own structural characteristics.
The current 1.6B model and 150B Tokens are just a starting point. The authors also point out in their conclusion that the future may just open up the imagination of extending the framework to longer cross-sequences of graphics, video and other structured models, consistently unified multi-mode models.
© THE
