The winner of this super-smart race is not trained, but the first to run through RSI pre-training.
The head said: "Pre-training and post-training is not a quantitative thing: computing, experience, engineering, they are all above a quantitative level. But unfortunately, it is also the most important thing, it is the primary basis for our processing of intelligence.
A classroom, and a $4 billion company.
Over the past year, "RSI" (Recursive Self-Improvement) has become almost the hottest word in the big model circle: let AI itself come up with improvements, execute itself, self-assess itself, then continue to improve on top of the results, cycle back and forth, and change more and more.
A US-based startup, Recursive, whose founding team comes from Google, Meta, and OpenAI, called it a "$4 billion project". Two weeks ago, they released a zero-training model, Flint.
In another classroom in Shanghai, the door number is 801, which is also the name of the model.
We are the only one that has released all the technical details.
Today, the Shanghai Institute of Innovation and Technology, together with New Order Intelligence, released the first full public pre-training RSI technical report, base model (801-30B), high-quality SFT data.
Not just one water test: less than three months into use, the model delivered in the classroom stood in the first row of open-source seats of the same size.
Technical report:
Models and data:https://github.com/GAIR-NLP/801-30B
801 ((red) in comparison with the open-source model: the upper row is the base model evaluation, the lower row is the SFT after evaluation ((mark †) in the evaluation of the complete post-training version of the comparison model released to the manufacturer)
Why do you need to pre-train with the RSI?
The answer is: RSI ends not after training, but in pre-training.
Pre-training determines the limits of all abilities
The knowledge and reasoning skills of a large model are largely formed during the pre-training phase. Post-training is done to "stimulate" these abilities, and it is difficult to create something that is not given by pre-training. Pre-training is therefore the source of the entire AI capability chain: all agents, applications and products are built on it.
The RSI is really "recurring" only after entering pre-training.
The most fundamental idea of the RSI is that AI can improve the process of making the next generation of AI, while the stronger next generation of AI can also continue to improve the next generation. Once this loop is turned around, progress is replicatively accelerated. But if the RSI only stays on post-training, it improves only "to repair an existing model; the fundamental process of making a model is still entirely dependent on humans. The loop is not really closed".
It's only when RSI enters pre-training that AI becomes involved in "manufacturing AI" itself for the first time. This is the most critical and difficult step on the path to self-improvement from "repair" to "building a house".
It will change the way the base model is produced.
Today, training a front-end base is one of the most scarce capabilities in the world: it requires large-scale, top-notch teams, years of accumulated but never publicly trampled experience, months of time and mass-calculation. The number of institutional clusters that can be done globally. RSI entering pre-training means that this highly talented, human-powered set of processes has begun to become AI-driven, accumulable, and replicable.
For the first time, experience can be profitable rather than lost.
The most valuable asset of pre-training is "Which roads don't work, why don't they work". In the past, these experiences existed in the minds of a few people, and people don't walk away, and this is the deepest shield in the laboratory at the forefront. We immerse every attempt, every failure and every reason for failure in a shared set of "memories" so that the next round stands directly on the shoulders of the next round.
It's particularly critical for those with limited computing power.
In a single opportunity pre-training, every GPU hour can't be wasted. Making every training, every failure turn into a cumulative experience is essentially improving the intelligence output of unit computing. For those with limited computing power, this is not just a technical route, it's a must-have.
The end point of the RSI is not training, but pre-training: five levels of meaning from the bottom up; post-training RSI is repair and pre-training RSI is a method of improving house construction.
So important, why are only two in the world doing it publicly?
There is no shortage of attempts at RSI. However, existing public work has largely focused on areas beyond the "manufacturing model": optimizing agent frameworks, writing GPUs, automating post-training, or simulating pre-training agent experiments with small models ranging from a few minutes to a few hours.
RSI track map: the closer to the "manufacturing model", the less public work; the full front-end pre-training column has only two points.
Difficulty, not on a quantitative scale
Because pre-training RSI is not a "more difficult post-training RSI", but something else.
After training for a round of RSI experiments, it usually takes only a few to dozens of GPU hours; while a full pre-training takes hundreds of thousands of GPU hours and lasts for several months.
There is only one chance: post-training can be repeated dozens and hundreds of times; complete pre-training, almost all teams can only run once. This is why the official technical report is titled: "No Second Run".
There is no standard answer: after training a clear score can be optimized; the real goal of pre-training is how much potential is left behind in post-training, which cannot be read from any single score.
Post-training RSI is like trying a dish in the kitchen: one is bad, one is done again.
Pre-training RSI is like a chip chip chip: once is months and a huge investment, with no second chance.
It's not just mathematics that's difficult, it's the whole production process.
Prior to the start of pre-training, the data had to be created from scratch: the original web pages, documents and code warehouses, through 6 data pipelines, more than 30 processing stages, eventually forming 1,053 data subsets. During the training, the training infrastructure replaced more than 130 versions, with superparameters involving 106 fields.
Therefore, the pre-training RSI needs to be improved not by a single model, but by the entire production line, and the entire organization that operates this production line.
How we do it: break the big circle and break it into small circles that can close.
The classical RSI is a big cycle: proposing improvements, executing, evaluating, updating; getting to pre-training, "execution" takes months and can't be repeated; "evaluation" doesn't have reliable scores, and the big cycle can't close.
Our approach is not to force the closure of the whole big cycle, but to break the pre-training into three categories of small cycles that can be quickly closed and then link them together with a set of shared memories.
Modulation cycle: From abnormal signals in training indicators and model outputs, all the way to infrastructure code, training configuration or some data source, repair and verification.
Potential Cycle: Not only does it look at how many points the model scored today, but it also measures how much training potential it has left to determine whether the training is still progressing and whether the formula needs to be adjusted.
Experience quantification: the process of cleaning up data, discovering websites, crawling over past "sensory" decisions into a process with clear metrics and agents that can be iterated.
Production memory: Every attempt, every failure and every reason for failure is recorded. Every subsequent proposal is based on "what has already been ruled out" rather than starting from scratch.
The big circles of the classic RSI cannot be closed in front-end pre-training (left); we divide them into three categories, six small circles that can be closed, and share a set of tissue memories (right).
Breakdown one: a load balancing problem hidden on the 15th floor
In a test run, all the aggregate indicators looked normal, but the agent found that the expert load imbalance indicators at layers 15 and 18 of the model were rising, while the aggregate data completely covered it up. The agent cross-pared over 130 code versions and 106 hyperlinked digital segments, ultimately locking the problem into a change in the load balancing punishment, and its interaction with the state of a particular parameter optimizer.
Breakdown 2: Why would a model "back" an API key?
Enter a word "deepseek", the model spits out a string of characters that resemble API keys, and then repeats the same proxy configuration indefinitely; ask it what to eat for dinner, and it starts with a selection of "Does this sentence make sense?"
Three types of anomalous output patterns, and the process by which agents trace them back to specific data pipeline links
We have solved five of these cross-team questions in about a month; in our past experience, without this infrastructure, each could take months.
These corrections, along with improvements to the data, have led to formal training. Under the same 1T tokens training amount, the average score on the 22 benchmarks ranges from 47.78 for the initial version test, to 51.96 for the modified version test, and to 56.83 for the formal training.
Two rehearsals and formal training: each turn improves on the previous one
It's not a water test, it's a first-tier base.
The first attempts at many new approaches were based on proof-of-concept: running on a small model and proving that "it can be done".801 is different. It is a complete training model directly to the strongest open-source base in the labeling industry, and the results stand. We have three levels of comparison, with conditions ranging from fairest to most unfavorable for us: 1) base to base (2) with the same SFT data (3) and only lightweight SFTs 801 against the complete post-training model.
First level: base to base, climb the first ladder
Based on 13 basic capability ratings, 801's base model averaged 73.13, which is comparable to NVIDIA's most recent release Nemotron-3.5-Lightning-30B-A3B, which led Qwen3-30B-A3B by almost 7 points, second only to Qwen3.5-35B-A3B.
Second layer: Same SFT data, gaps are drawn
When each base model was compared with exactly the same data, 801's 13 items averaged 81.34, led by Nemotron-3.5 ((79.88) by about 1.5 points, led by Qwen3 ((78.27) by about 3 points, and still lagged behind the more Qwen3.5 parameters. In context and knowledge class evaluations, 801 ranked first in both AA-LCR v1.1 ((61.0) and AA-Omniscience ((31.3).
Level 3: Lightweight SFTs only and complete post-training models
This layer is most unfavorable to us: the 801 only makes lightweight SFTs of 6B tokens, no reinforcement learning, and the data volume is only a tenth or even a few hundredths of each manufacturer's post-training; and the opponents are the official release versions of the complete post-training process. Even so, in the SWE-bench Verified, SWE-bench Pro, CyberGym and KernelBench four agents rated the 801 an average of 58.4, higher than the K2-36B-A4B of this comparison: 45.3), Qwen3.6-35B-A3B: 44.6) and Gemma-4-26B-A4B36.1).
I'm going to face Flint.
Only 801 of lightweight SFTs, the complete post-training version of Flint on KernelBench and AA-Omniscience, are significantly ahead, with a gap in the other two. Considering the disparity in both resources and post-training inputs, the results themselves illustrate a lot.
A new team, for three months, has only made a base model of lightweight SFTs and has been able to deal directly with the manufacturer's models after complete post-training.
What we published and how to compare
At the heart of this release is the pre-training itself, so we are open source on the base model, as well as on the lightweight SFT data for measuring post-training potential, as well as the full technical report.
Q: Why do we only send the base model and not the chat model?
A: This time the focus is on pre-training. The base model is the outcome of pre-training; the full post-training version will be released at the next stage.
Q: What is a lightweight SFT for?
A: It's a "meter" used to measure how much post-training potential a base model has, not a finished chat model. We're open-source and the community can use it to evaluate any base model.
Q: Is it fair to compare 801 and other chat models directly, right?
A: It's not entirely fair, which is equivalent to comparing a newly-covered furnace with a finished finished room. We suggest comparing base to base, or on the same SFT data; comparing with a complete post-training model, only as a reference to potential.
Post-training RSI is the starting point and the end point of pre-training.
All post-training RSIs have a premise: first there is a strong enough base model, and this is exactly what pre-training will produce.
Bringing RSI into pre-training means that AI is beginning to improve the process of making AI itself. This is not a re-branding, but a change in the way the baseline model is produced. When this process can self-improve, each generation of baselines will be the starting point for the next generation.
"Pre-training is a once-in-a-lifetime opportunity. It's time to improve, not just a model, but the whole organization".
What's going on?
This path requires more computing skills, more talent and more peers. Welcome to downloading models and data, reproducing, comparing, taping; and welcome computing and industry partners to work with us to make this turnaround faster.
© THE
