Agent began to perform continuous tasks, with new changes in the needs of the model.
As the task chain lengthens, the computational costs and long-text capabilities of each step of the model affect the final user experience and cost, which makes the importance of the Flash file model stronger: they focus more on finding a balance between capacity, speed and cost to take on those high-frequency and continuous tasks.
On November 5th, the Baidu Value System officially released the new generation of the Core Model Value System 5.5 Flash Base (ERNIE 5.5 Flash Base) and opened the weight.
This release is a rare MoE language model, with a total parameter of 280B, activating 14B per token, which completes pre-training from zero on 33.5T tokens, with the longest training context being 128K, and extends beyond the minimization and reasoning phase to the longer context of 1M tokens.
The model is open-weighted and supports open-source ecosystems in communities such as vLLM, SGLang and others.
HF FP8:
HF BF16:
SGLang's cookbook:
Vllm's cookbook:
The Center has released a "Base" model, which has full universal capabilities that can be used as a starting point for all subsequent training, on the basis of which field adaptation, instructional mitigation, reinforcement learning, and even agent-facing behavior can be shaped. This means that developers can decide for themselves the direction of the next phase of the model.
The model's performance and architecture were even more of a concern. This time, Baidu made stronger underlying capabilities with limited activation parameters and a caching budget.
Performance of SOTA at the same level
Currently, AI applications are about to enter the "Super Agent" era, and in the context of high-frequency, long-term autonomous operation of the token consumption, the Flash-level model will become a really widely used ranking. For the model itself, the Agency needs to frequently read bulk environmental events, the biggest bottleneck being the long context Pre-fill and KV Cache cache occupancy, which is also the focus of the 5.5 Flash Base design.
Baidu's goal is to make every active parameter, every byte of KV cache, play a bigger role. Based on this idea, its team has made significant technological innovations in specific components such as attention, MoE, and future forecasting.
The model has 43 layers of Transformer, Hidden Size 4096, and 64 attention heads. Except for the first layer, all layers of the front-end network have been replaced with MoE blocks. Its pre-training uses a mixture of Muon optimizer and FP8 accuracy, which is gradually expanded to 128K in training, and further expanded to 1M in the reasoning and fine-tuning stages.
In comparison to the official equivalent model, the 5.5 Flash Base is the overall leader in knowledge, reasoning, mathematics, code, multilingualism, and context, with rivals including DeepSeek-V4-Flash Base, Hy3-preview-Base, and MiMo-V2.5-Base.
From the individual contrast with DeepSeek-V4-Flash-Base (parameters 284B, activation 13B), it is clear that the new Centauri model has advantages in terms of discipline and reasoning, with code capability leading the largest segment: HumanEval + is 82.3 to 69.5, LiveCodeBench is 40.1 to 31.9, and FullStackBench is 54.6 to 43.3. However, Centauri 5.5 Flash Base is slightly different in small ratings such as ChineseSimpleQA.
The results of the context evaluation also confirmed the architectural efficiency of the 5.5 Flash Base, with the overall soundness of the context benchmark in six dimensions, and the design of "a few full layers plus a majority of compressed layers" without sacrificing the long-text capability of the model.
Behind this scoreboard is a significant improvement and innovation in model architecture by the literature team.
Mixed scant attention, 1/7 Full context access
In terms of architectural efficiency, the 5.5 Flash Base uses mixed scarcity attention to reduce the cost of reasoning.
Transformer keeps the KV cache for each layer of the entire history token when processing the long text, which is a major source of reasoning costs as the linear growth of the context length increases. Baidu's insight on this is: not every layer needs to see historical details as clearly.
5.5 Flash Base uses a mix of MLA and HCA: only one of the seven layers retains token-level access to the entire context, with multiple potential attention (MLA), the remaining six layers use gravity compression attention (HCA), and each of the 128 tokens is compressed into a cache entry, with the focus computed only on the compression entry plus the most recent small uncompressed window.
In long-term context reasoning, the MLA layer is responsible for page-by-page refinement, maintaining access to each original token, while the HCA layer focuses only on chapter summaries, maintaining a rough view of the entire history, and going back through a few layers when details are needed. Because the HCA layer only requires a small portion of the KV cache on the entire context layer, the cache expense of the entire model is significantly reduced.
The six-layer MLA is the only way to access long-distance history by token, and if you do tight attention in these layers, they will swallow up most of the computation under the long text. However, the MLA layer introduces scant attention (DSA) by assigning a lightweight index for each query to the historical location, so that the core attention is focused only on the top-k position calculations with the highest score, and the single query expense no longer increases with the context, while the pre-training loss is balanced with tight attention.
When the KV cache budget is limited, the existing CSA + HCA method is to use a lot of compression scarcity attention (CSA) before using four times the historic compression and then selecting scarcity, while on the Baidu 5.5 Flash Base, the budget is focused on a few MLA layers that retain all of the tokens.
Within the attention layer, there are also two improvements to the VC 5.5 Flash Base. Low-level gate controls use the low-level query latent to calculate outputs generated within the layer. Since the latent dimension is only a quarter of the cache state, the gate projection parameter is reduced by 75%, and the melting effect is improved.
Virtual Head Attention has been received by NeurIPS 2026.
Multi-router MoE, more accurate at the same cost
Attention determines how each token reads history, and it is the responsibility of the MoE layer to hand it over to who handles it.
The goal of the MoE (mixed expert model) architecture is to pursue the density of intelligence, while the MoE 5.5 Flash Base is designed in depth for the MoE layer, solving the "arbitrary" choice problem.
The traditional MoE workflow is that each token enters the MoE layer and is scored with only one router and then through a hard top-k selection expert. The problem with this mechanism is that scoring is only slightly related to the actual usefulness of the expert, especially in the middle depth, and because the selection expert does not get the gradient of the token, once the error is difficult to correct, the error reinforces itself.
Multi-Router allows multiple independently initialized sigmoid routers to be rated simultaneously, turning "one router says count" into "multi-router comprehensive ratings". These routers vary in preference and can be selected as long as they are strongly viewed by one router, or viewed by several at the same time, and single misjudgments no longer determine the outcome, while the number of activated FLOPs is exactly the same as the volume of communications as the single router, with no additional cost in reasoning.
By configuration, each MoE block contains 512 routing experts, 10 at a time, plus one shared expert, load balancing bias correction with no ancillary loss; tokens are first projected from 4096 D to 2048 Vine spaces, and experts are computed and mapped back to the native space in the hidden space, reducing the amount of communication with expert FLOPs by about half. Stable LatentMoE design (RMSNorm plus SiTU-GLU activation) allows experts to maintain a stable scale of output over 43 layers, with each expert's routing weight multiplied by their own learning parameters.
ECHO, even the MTP module is saved
The third innovation in Flash Base 5.5 has shifted its focus to a more fundamental question: what goals should the model use to learn?
Multi-Token Prediction (MTP) was proposed by Meta Systems in 2024, and was widely used in the industry after the adoption of DeepSeek V3, allowing models to learn multiple tokens behind the predictions while training to plan for more intensive monitoring of signal training models. It can also be used in conjunction with draft models of speculative decoding when reasoning, first guessing multiple tokens, and then verifying batch acceptance to accelerate generation.
But the problem with conventional MTP is that it is attached to a set of dedicated modules outside the mainstream Transformer to predict more distant tokens, with multi-step losses mainly overseeing these outputs, with the mainstream layers still serving only the next token. In other words, long-term planning capabilities are not directly trained.
ECHO does not add the MTP Transformer layer separately, but instead chooses to let the master handle the final layer of its own token prediction at the same time, and the multi-step loss thus directly affects the parameters of the current token's generation.
This design added an advantage to the reasoning phase: the final layers of the project can serve directly as a speculative decoding built-in draft model, with little to no additional parameters added. It is noteworthy that this layer also exhibits an emerging outputs: it only learns to predict multiple tokens during pre-training, but as a draft model, it can continuously draft multiple tokens back and forth, with higher acceptance rates than outputs of MTP modules trained in the same way.
From the three main improvements, it can be seen that Baidu's thrust point is forward-looking, as Agent starts to run autonomously on high frequency, long-term, and the competing models are focusing more on efficiency, which is the main line that has been determined by the Central 5.5 Flash Base.
More structural improvements, data and infrastructure
There are also several component-level updates. In residual connections, fluid-bound hyperconnection (mHC) replaces the standard residual structure, with each layer reading and writing 4 parallel residual flows through a constrained mixed matrix, to extend the residual bandwidth between layers at almost zero cost, in line with the hyperconnection idea of recent years in the industry.
On the output side, MultiMax for a single softmax temperature can only be introduced in a difficulty between scarcity and multi-punctuality, both to compress the long tail of unreasonable tokens and to preserve the diversity of reasonable candidates. Training expenses can be neglected without increasing visibility.
In terms of annotation, the new BPE word is about 200K, and the numbers are taken from right to left, with a set of three segments, such as 1,234,567 being cut into 1,234,567, making the numbers more symmetrical and more symmetrical to arithmetic and numerical density texts.
On the data side, Baidu focuses on building three directions. Web data combines multilingual crawling, high-credit extraction and model-driven quality filtering; code data beyond static code, including software evolution data such as commit, issue, pull request, code review, automated testing, and CI-verified repair, pointing to warehouse-level reasoning and Agent software engineering, with reference to Base positioning; long-text data includes natural length documents, sequences that interfere with text spellings, and synthetic detection and multiple-printerview data.
On the built-in side, Baidu allows distributed Muon optimizers to be efficient at thousands of cards. Muon uses incremental conversion updates, changing communication patterns. Baidu changes the Dense parameters from interconnection to the same card path, and merges with MoE communications into single-collection communications, while balancing the state of the optimizer between nodes to avoid excessive storage pressure at a few nodes.
In addition, Baidu has proposed an algorithm for large head dim FlashMask-4, based on CuTe-DSL that supports SM90 and SM100 architectures, supporting qk's first 576 dimensions, v's first 512 dimensions. The SM100 has been upgraded to a forward performance of 175%, upgraded to a reverse performance of 74%. The algorithm is now open source in PaddlePaddle's GitHub warehouse.
The infrastructure capacity behind these models is the result of years of continuous investment in deep learning frameworks and large-scale training.
Overall, on VC 5.5 Flash Base, Baidu used a full-scale pre-training to verify the added benefits of numerous technical improvements under the same architecture. Real deployment will be the next step in testing this paradigm.
Base is just the beginning.
Opening a Base model also means giving more space for further development to developers, and to some extent, Open Base requires more depth than Open Chat. The weight of the pre-training terminal is equivalent to the ability to directly display the color.
Based on open base weight, the basic model can be incorporated into the developer's secondary training stream to grow a richer ecosystem. A high-quality base that can be directly replicated means that enterprise-level developers do not have to start from scratch to train models to serve the local language, culture and industrial landscape.
In the 5.5 Flash Base model, from long-text attention reading, to MoE routing expert calls, to ECHO's attempts to make the mainstream learn to predict the future more directly, the design around actual use also provides support for more valuable tasks. In the future, under the same computational resource conditions, AI has the opportunity to support longer task chains, more rounds of trials and feedback, thus achieving unprecedented applications.
The official version of Civic 5.5 Flash is expected to be released soon. On this multi-player race track, the weight open is just a ticket, can not sit on the card table, and to see if the official version can fulfill the potential of the base seat. For users, the arrival of Civic means more options, the competition for the future Flash file will be even fiercer, and there will be cheaper and more powerful models.
It's time for the official launch.
© THE
