One of the menu items in the paper, which seems to need only one simple addition: a Funghi pizza for €6.90 and a Prosciutto pizza for €7.90, what is the total price?
The LLaVA-1.5 not only read the wrong price, but also added an empty calzone, which ended up giving 26.40 euros; the DGP read the two correct prices, which amounted to 14.80 euros.
The problem may not only arise in the "reasoning" link. The model may not have obtained sufficient reliable visual evidence to match the problem before it began to calculate.
Many current multilingual models take only the characteristics of the last layer of the Transformer (ViT) or the second layer of the Transformer (LLM) and send them to the large language models (LLM) once after the projection. After that, no matter how many layers the language model has, it starts with this fixed vision: recognition of small letters, judging colors, understanding movements and analyzing the entire picture, sharing the same visual input.
But the same picture faces different problems, and the visual granularity required is not the same. Reading menu prices depend on local textures and text details, while understanding the historical significance of a picture depends more on higher-level semantics. Since different layers of ViT code for different granularities of information, why keep only one layer before entering an LLM?
A research team from Northwestern Industrial University and the Hong Kong University of Technology proposed the DGP (Dynamic Gated Pathways, Dynamic Gateway Pathways): allowing each LLM layer to access all of the ViT layers, and the contextual and semantic status of the language layer based on the current image editing problem.
In a nutshell: DGP is not simply averaging more layers of visual information, but rather transforming "which layer of visual information, which layer of language model should be connected to" into a learning, routing problem that changes with the language layer, tasks and dynamics of the sample.
In LLaVA-1.5-7B, DGP crossed the baseline in all 11 indicators reported in the paper, with SQA-I up 4.2 points, MM-Vet up 3.3 points, and VizWiz and LLaVA-Wild up 3.1 points. The method achieved consistent gains in the same way in Qwen3-VL-2B and LLaVA-NeXT.
Where to Connect: Boosting MLLMs via Dynamic Gated Pathways across ALL ViT and LLM Layers
Research team: Northwestern Industrial University, Hong Kong University of Technology
Link to the paper:https://arxiv.org/abs/2611.06844
The code repository:https://github.com/YingyingYan1/DGP
Key contributions:
DGP is not a visual encoder.
It's the "connection" of visual information.
The core contribution of this work can be summarized as three points.
First, move from "Which ViT layer to choose" to "Which LLM layer to connect to".
Past discussions have often focused on the use of the last layer, multiple intermediate layers, or the pre-fusion of multi-layer features. DGP redefines the problem as where to connect: visual representations have levels, linguistic representations have levels, and the connection between the two layers really needs to be designed.
Second, the predefined connection is extended to the dynamic routing that covers the entire ViTTMLLM layer pair.
DGP incorporates all visual layers and all language layers into the candidate connectivity space. Each LLM layer is no longer fixed to a specific visual layer, but is assigned different weight to all ViT layers according to its current multimodal semantic state. Here, all-to-all refers to all "layer pairs" that can participate in the connection; the actual route uses Softmax overload, not two connections between visual tokens, nor does it retain only a few hard routes.
Third, not only does it verify the "points" but it also analyzes "how exactly the model is layered".
The dilution experiments showed that, in the current setting, the direct average of the entire visual layer would drop significantly, and the learned dynamic weight was an important source of gain. Further visualization showed that routes would vary with different problems of LLM depth, task types, specific samples and even on the same image, providing explanatory evidence for performance improvement.
1. Why would it be a bottleneck to just look at a chart at the beginning?
ViT does not repeatedly encode the same information in each layer. After visualizing the attention of different ViT layers, the paper found that the shallow layers were more focused on edges, textures, and local structures; as the network deepened, attention gradually focused on fewer semantic areas.
Figure 1: Distribution of attention in different layers of the same image in ViT.
This level of division means that different levels have value:
The shallow layer retains more local clues such as edges, colors, textures;
The intermediate layer is better suited to express the overall relationship between the structure of the object and the local silicon;
Deep is more geared towards abstract semantics and global concepts.
However, mainstream MLLMs often take only a fixed ViT layer, project it as a visual token, and inject it into the LLM input once. After that, visual information can only be transmitted step-by-step with the hidden state in the language model.
Multi-layered visual characterization methods have been developed mainly along two routes:
Before integration, after input: compressing multiple layers of ViT into a representation on the side of the visual encoder and then sending it to LLM. It adds a source of information but still allows all LLM layers to share the same result of integration.
Pre-defined cross-layer connectivity: connecting several visual layers to a designated language layer. It begins to consider the layer structure on both sides, but the connectivity location or candidate visual layer is usually still pre-determined manually.
The difference with DGP is that instead of deciding which layers are worth connecting, the model opens up the entire ViT layer pair and lets the model learn the connection intensity based on the current semantic state.
Figure 2: From fixed single-layer, encoder-side fusion and predefined connections, to full-layer dynamic connections in DGP.
2. How does DGP make each LLM level a "review" visual level on demand?
The overall process of DGP can be broken down into three actions: first determine what is needed, then select the corresponding visual layer, and finally write the result of the integration back to the current language layer.
Figure 3: DGP framework. Each LLM layer generates a set of routing weights for the entire ViT layer and updates the visual token through gate control residuals.
2.1 Start a "visual query" using the current state of the language layer
DGP inserts a learning glimpse token before the first answer token after the image and the question. Due to the causal structure of the return of attention, the location allows the previous image and context to be seen on the text; while it is hidden in different LLM layers and records the multi-modal semantics that the layer has already formed.
Therefore, DGP uses a glimpse of the hidden state of each language layer as a query signal. The visual side takes the [CLS] representation of each ViT layer as a layer summary, which is mapped to the LLM cache space as a key and value.
Intuitively, the glimpse token expresses "what this current language layer is looking for" and the [CLS] abstract for each ViT layer expresses "what this visual layer offers".
2.2 Weight allocation for all visual layer dynamics
The query and visual layers abstract first enter a multi-headed cross-attention module shared by all LLM layers and are characterized by cross-modal routing; each LLM layer then uses its own separate MLP routing to generate weights that cover the entire ViT layer through Softmax.
The design retains two characteristics:
Shared cross-attention provides a unified visual layer comparison interface;
Layer-specific routers allow different layers of LLM to form different visual preferences.
Thus, even when faced with the same graph, different problems will generate different routes; even when images and problems are the same, different LLM layers can choose different combinations of visual layers. The "dynamic" here refers to the fact that the weight will change with the sample of the graph problem and the LLM layer; the current implementation of the paper will not recalculate routes on each new token generated in the answer.
2.3 Converge in the same spatial position and rewrite with the door control residue
After weighing the routing, DGP adds the characteristics of each ViT layer at the same patch location and:
The convergence patch of the current language layer = Σ(This language layer is assigned to the weight of the fourth ViT layer × the corresponding patch of the fourth ViT layer)
This combines different semantic graphs without disrupting the spatial correspondence between the patches. The result of the merger is then passed through a projector that is exclusive to the current LLM layer and rewritten in a residual way by means of a learning scale gateway token.
The entire process can be summarized as follows:
Language layer state generates query → Dynamic overload on the entire visual layer → Location-based fusion patch → Door control injects the current LLM layer
As a result, visual information is no longer present only once at the input end, but is continuously updated at the level of participation in the LLM.
2.4 Use the rule of thumb to prevent premature routing "only one layer".
Opening up all candidate connections does not mean that the model will use them automatically. If the initial training routes are rapidly concentrated in a few layers of vision, the DGP may still degenerate into an approximate fixed connection.
To this end, the paper incorporates the batch rule at the pre-training stage, maximizing the average route distribution distribution of each LLM layer on a batch, and encouraging exploration of all the different visual layers. This restricts the batch average distribution and does not force each sample to use all layers uniformly, so individual problems can still form a clear preference.
The normal coefficient for the pre-training phase is 0.01; it is turned off after the instruction is tuned, allowing the route to be further specialized to the visual granularity required for specific tasks.
3. All 11 indicators of LLaVA-1.5-7B have been upgraded,
In the meantime, there are two additional frameworks for returning profits.
The team first experimented with LLaVA-1.5 on two scales, Vicuna-7B and Vicuna-13B. The evaluations covered tasks such as GQA, SQA-I, VQAv2, VizWiz, SEED, LLaVA-Wild, MM-Vet, MMBench, and POPE, involving general visual questionnaires, particle sensing, instruction followed by hallucinatory evaluation.
Table 1: Comparison of DGP with mainstream MLLM and multi-layered visual feature utilization methods.
Under the 7B configuration, the DGP is upgraded relative to LLaVA-1.5 in all 11 indicators in the table:
SQA-I:66.8 → 71.0, an increase of 4.2 points;
MM-Vet:30.5 → 33.8, up by 3.3 points;
VizWiz: 50.0 → 53.1, up by 3.1 points;
LLaVA-Wild: 63.4 → 66.5, up by 3.1 points.
Compared to the 7B multilayer visual characterization methodology in the same table, DGP scored the highest reported scores in six indicators: GQA, SQA-I, SEED, LLaVA-Wild, MM-Vet and MMBench-CN.
Under the 13B configuration, DGP exceeded the LLaVA-1.5 baseline in 10 of the 11 indicators. VizWiz improved by 3.8 points, MMBench improved by 2.4 points and GQA improved by 2.2 points.
Changing the visual encoders, language backbone and resolution strategies, DGP is still working.
In order to test whether DGP applies only to LLaVA-1.5, the research team added it to two larger differences:
Qwen3-VL-2B：SigLIP2-L + Qwen3-1.7B；
LLaVA-NeXT:CLIP-L + Vicuna-7B, and high-resolution cuts using AnyRes.
Table 2: DGP results on Qwen3-VL-2B and LLaVA-NeXT
All four indicators in the two framework reports received improvements. For example, Qwen3-VL-2B's SQA-I was upgraded from 85.3 to 86.9; LLaVA-NeXT's SQA-I was upgraded from 69.5 to 72.3; MMBench was upgraded from 66.5 to 68.7.
How many layers are there and how much does it cost?
Table 3: Batch size is a parameter for 1 hour, generating FLOPs with peak visibility.
On the 7B model, the DGP increased the parameters by 5.29% and the FLOPs by 4.07% with peak storage essentially unchanged; on the 13B model, the parameters increased by 4.27%, the measured value of the FLOPs was essentially flat, and the peak storage increased by 4.13%.
From scores to specific answers: see first, it's possible to answer yes
Figure 4: Mathematical problems, OCR and comparisons of cases in particle understanding.
In the cases presented in the paper, LLaVA-1.5 misses menu prices, prepares dishes that do not exist, or is unable to correctly associate text, movements and expressions with the problem.
In the menu item, read EUR 6.90 and EUR 7.90, and get the correct total price of EUR 14.80.
Identify "single foot jump after ringing fingers" in a multi-step motion chart;
In classic cat expression packages, the contrast between personality emotion and cat theology is explained more fully.
These examples cannot alone prove that all the errors come from visual connectivity, but they do intuitively show a key link: more reliable visual evidence of the relevant problems can reduce the chances of subsequent reasoning arising from the error.
4. Key Meltdown: More layers of vision do not equal more effective information
Does DGP benefit from dynamic routing, or just because the model sees more ViT layers?
The team set up a direct contrast: instead of studying the routing weights, they averaged all 24 ViT layers evenly. The result was not only no improvement, but significantly lower than the single-layer baseline in most indicators.
Table 4: Average convergence, dissolving of dynamic gate control routes and components of the silicon standard.
MM-Vet decreased from 30.5 on the baseline to 25.3;
MMBench decreased from 64.3 to 57.6;
LLaVA-Wild dropped from 63.4 to 56.3.
The reason for this is not difficult to understand: low, middle and deep layers are characterized by different semantic grains, and overlapping without differences can confuse complementary information into noise.
With the addition of dynamic gate control routing, MM-Vet and MMBench bounced back to 32.4 and 65.6 respectively; with the addition of the aluminum rule, further to 33.8 and 66.7, SQA-I also increased from 70.5 to 71.0.
This group of mergers provides a more powerful explanation than "using more layers to make it stronger":
In the current LLaVA-1.5-7B setting, simply increasing or averaging the visual layer is not enough to improve performance; the learned layer weight is an important source of gain.
5. Route visualization: the same diagram,
If the problem changes, the visual layer of model selection will also change.
If DGP is indeed on-demand routing, then the weight learned should not always be concentrated in the last ViT layer, nor should it degrade to a uniform average.
Figure 5: Different levels of LLM, routing weights for evaluation tasks and sample responses.
The results of the visualization are consistent with this expectation:
Different LLM layers have different preferences. Medium ViT layers are more often activated, but low and deep layers are also selected by specific language layers. Representative curves typically have multiple peaks, indicating that the model combines multiple visual particles, rather than betting on single layers.
Different task preferences differ. For the 21st LLM layer of fixed observation, LLaVA-Wild, VizWiz and SQA-I will form a different visual layer weight distribution.
Different samples of the same task are also different. Routing differences do not only remain at the data set level, but are further refined into individual inputs.
More intuitive evidence comes from the "fixed image, just change the problem" control experiment.
Figure 6: When the same image faces different problems, the DGP changes the visual layer weight.
When a question asks for color, attributes, or small-area details, the route is more oriented towards retaining the shallow or shallow-middle characteristics of the texture and local structure; when the question is directed to the identity of the painting, its historical significance, the meaning of the scene, or the understanding of the action, the emphasis is shifted to the middle deep or the deep.
The differences in routes are ultimately reflected in the spatial characteristics of the merger.
Figure 7: Fixed use of CLIP 23 layer, average of all 24 layers, and integration characteristics generated by DGP in different LLM layers.
The fixed single-layer spatial response is rougher, with more interference in the background; the uniform response is still more dispersed on average, although multi-layer information is added. In contrast, the DGP characteristics shown in the paper have a clearer subjective outline, and local structures such as the edge of the column, building boundaries are better preserved. LLM sections 1, 18 and 28 do not have the same convergence results, further explaining that the path does not collapse into a fixed template.
6. Applicable boundaries: This is a prospective one
The connection principle, but not the ultimate answer
DGP's current main validation remains focused on the 7B/13B configuration of LLaVA-1.5, and extends to Qwen3-VL-2B and LLaVA-NeXT. For language models over 30B, broader visual encoders and larger multi-modal content, the paper has not yet completed system validation.
Additionally, full-layer dynamic connectivity will introduce additional parameters and computation. The growth in existing experiments is relatively limited, but its expansion efficiency is still worth further research as the number of model layers and visual resolution continues to grow.
Thus, the more accurate conclusion is not "DGP has solved the multi-modal vision bottleneck", but rather: the paper provides evidence that, through cross-architectural experiments, fusion and routing analysis, in models and data settings that have already been tested, the connectivity of the visual cube language significantly affects the performance of multiple benchmarks, a design dimension that is worth continuing to verify.
What is the meaning of this verse?
In the past, multi-modal models often used visual encoders as a one-time input of information: selecting a layer, generating a set of visual tokens, and then handing over all subsequent understanding to LLM. DGP connects visual cryptography from a fixed channel, transforming a set of dynamic paths controlled by the language layer state.
If ViT's multi-layered characteristics are viewed as an abstractly arranged database, the traditional method is more like copying one page before entering the LLM, allowing all language layers to be read repeatedly; whereas DGP allows each language layer to go back to the page they need in the database based on the current problem.
The revelation of this work is therefore more than a new module: the bottleneck of multi-modal models, not necessarily because the visual encoder is not always strong enough, data is not enough or language models are not large enough, or it may come from too rigid "connections" between the two models.
What is really important is not to make the model look more unmistakable, but to make it in the right language, with the right visual detail, to see the information that is really relevant to the current problem.
