Scaling up is a hallmark of big models and has led to the leap in language model technology in recent years. However, in reality, large language models architecture, as its most essential computational basis, has also continued to evolve in recent years. GQA, MLA, the introduction of various rare attention modules, or linear attention networks such as Mamba, Gateway DeltaNet, DeepSeek's fine-grained MoE, and many tokens, prediction and Looped architectures, have introduced new architectural improvements on the basis of previous models.
Recently, "Father of LSTM" Jürgen Schmidhuber, in collaboration with researchers from institutions such as the University of Hong Kong, KAUST, Huawei, Chinese University of Hong Kong, and Beijing University, published an in-depth technical overview of large-scale language model architecture research.
Towards the Next-Generation Architectures of Large Language Models: A Survey
The main page of the project:https://surveyllmarch.github.io/
GitHub：
From 1913 onwards
The evolution of language models spans almost a hundred years. In 1913, Markov calculated the sequence of vowels and consonants in Pushkin's epigraph, first using data to illustrate the condition of language. In 1948, Shannon viewed language as a random source. Later, the n-gram model in speech recognition, the loop network (1990), the LSTM model (1991) were proposed successively. To date, the Transformer architecture supports the entire field of large models.
The longer time span allows us to see the cross-evolution of architectural technology. For example, in the Fast Weight Programmers proposed by Schmidhuber in 1991, there is an additive expansion update rule, which today can be referred to as the non-negative linear Transformer curve. While the 2020 focus on nucleation linearity is mathematically equivalent to adding it to the integration. In addition, the multiplication gate control of the LSTM was moved to the front-end network by the GLU, forming the SwiGLU network that is common today, while the central LSTM maintains a poorly flowing additive path, and has a substantial connection to the later Residual Connections.
Details of the model architecture
The article first summarizes the different components of Transformer based on their respective architectural studies. It covers the following aspects: efficient attention mechanisms with KV cache management, linear language models, hybrid experts (MoE), activation functions, integration with residual connections, location coding and terminology. In addition, the article discusses the computational allocation of generation time and discusses topics such as multi-token prediction, adaptation and loop depth.
Overview of classification trees with chapter numbers in brackets
The basic module introduces the various components of the classic Transformer architecture. The current major language model is essentially the same architecture, as its core structure is based on the single-code Transformer network (i.e. decoder-only Transformer). The decoder-only Transformer includes modules such as the standard Pre-RMSNorm, SwiGLU and RoPE.
KV compression method (separated from overview table 2), grouped by mechanism of action, ordered by first year of publication.
High Efficiency Attention with KV caches a chapter, which breaks down KV caches into token heads, key focus heads, key dimensional heads, and key layers. These four dimensions are classified. Rare attention mostly falls on token dimensions, such as StreamingLLM, H2O, SepLLM, and trainable block caches such as NSA, MoBA. While MQA, GQA share on the head, MLA compress in feature dimensions, and MiniCache share across layers.
Comparison of the uniform forms of the representative linear attention model (see Table 4)
The first chapter of the linear language model introduces different linear language models from the current RNN, Fast Weight Programmers, and State Space models. Although there are different technical origins, these linear models can be written in the same form. The article thus summarizes and contrasts the core linear arithmetic of today's mainstream linear models using two different uniform forms.
The MoE chapter traces its evolution along three intertwined lines: introducing the load balancing and stability techniques required for tight gate control FFN and its large-scale training, introducing the layout of fine-grain specialists and sharing specialists, and finally introducing system-level variables that are redesigned by specialists under communication and storage constraints.
Activation function chapter 1 discusses the evolution from Post-LN to Pre-LN, RMSNorm, and discusses the lattice effect between gradient disappearance and represents contraction. Location coding chapter covers relative position codes such as absolute position codes, RoPE and ALiBi, as well as context extension methods such as YaRN, and also discusses models of non-expressive position codes such as NoPE.
The final two chapters focus on computational allocation at generation. One chapter predicts how future tokens will be modeled, dividing the method into mutually independent parallel predictors (such as Medusa) and sequential predictors that retain a causal chain (such as MTP modules in DeepSeek-V3) and discussing the validation and acceptance of draft speculative decoding separately.
A combination of open source models
Chapter 14 comprehensively ranks all mainstream open source models from LLaMA, Qwen, to DeepSeek, Kimi, GLM, MiniMax, and includes four common architectural design combinations: standard compact models, rare expert models with compressed attention mechanisms (e.g., DeepSeek-V2/V3), mixed architectures that intersect attention layers with linear layers (e.g., Jamba, Qwen3-Next, Kimi Linear), and models with multi-token predictability.
Architectural configuration contrast of representative MoE large models (selected from overview table 14)
The scale of the open-source model corresponds to Humanity's Last Exam scores, and the size of the model marks increases with the parameters (see MoE models see activation parameters).
Conclusions
Starting with the standard decoder (left), the review divides recent structural changes into four directions.
One, the standard decoder-only Transformer is still the reference of almost all new designs, and the new work largely retains this framework, modifying only some of its modules or modes of operation, and can be categorized into four directions: 1) Flexible and conditional processing: avoiding that each token is treated the same as MoE; early withdrawal and looping depth algorithms; 2) refined change dynamics: keeping the computing path unchanged, improving activation functions, regulation and residues; 3) selectivity and flexible management status: compressing, filtering, attention modules for looping functions, improving or replacing the original KV caching mechanism; 4) Looka mechanism and delayed submission: three predictive submissions and decoding, such as multiple token architectures and predictive coding.
The default practice of the standard decoder-only Transformer, as well as recent changes to the four-dimensional, ten-dimensional architecture
The article argues that the next generation of architectures, characterized by an increasingly arbitrary allocation of computations, capacities, states and predictions over tokens, layers, and time, rather than a single module replacing attention or a tight frontal network. In which predictive field, loop depth, and state size are still lacking in scaling law studies equivalent to MoE, is a problem to be solved.
In addition, how conditional computation and cyclical computation can maintain training stability after scale expansion, how evaluation can simultaneously reflect the balance between expressiveness, efficiency and stability, are also to be addressed. These problems require architectural design, continuous collaboration of optimization methods and systems engineering. The authors hope that this summary will provide a structured design space map for this purpose.
The article is attached to the project's homepage, as well as the GitHub open source project, inviting the community to collaborate.
Introduction by the author
The first co-author is Yao Teng, Guoxuan Chen of the University of Hong Kong, and Yimeng Chen of KAUST and Han Shi of Huawei.
