Dr. Zhang Chang, founder and chief scientist of Norizon AI, focuses on continuous learning, machine learning, machine learning, big model architecture, and reasoning. Formerly a research scientist at Facebook AI, now Meta AI, responsible for the development of large-scale machine learning systems for billions of users. Currently an assistant professor, researcher, PhD student at Beijing University's Institute of Artificial Intelligence, the first national youth talent program overseas. He is the chairman of top academic conferences such as NeurIPS, ICML, ICLR.
This is a remarkable achievement.
Researchers at MIT CSAIL published a blog titled Sparks of Weight-Space Programming. The most eye-catching is a set of numbers: the GPT-6 Astra did not use gradient training, only modified Qwen3.5-0.8B by about 0.016% of its weight, bringing the MATH-500 score from 7.9% to 48.1%.
They call it Weight-Space Programming (WSP): allowing a large enough general model to read the model, analyze the model, and then manually rewrite its weight.
As a scholar studying hypernetwork and model continuous learning, I am excited about this direction.
Let's start by talking about ourselves. We are an AI startup company that I founded, and we only do one thing: after we get the big models out of the factory, we can learn from the margins like people do.
A common way to do this today is to extend the context, or to retrieve a new version of a set of data to "recreate". We chose another path: to directly change the model parameters and write new experiences into the model itself.
The tool used is called a supernetwork. Simply put, it is used to directly generate the parameters of another network using one network. Giving it a new material to complete the writing without having to repeatedly train it. It is more like installing a "hybrid" on the model: the context is a short-term memory that can be cleared by hand, something that is written into the parameters will remain, and the next task can be used.
So when I saw WSP, my first reaction was kind: we were pursuing the same vision, not relying on re-training, so that the model continued to grow in use. The means were different: WSP relied on general big model field reasoning, temporary design and editing; we trained a model specifically responsible for updating beforehand. One based on the field play, one based on prior learning, and the two are likely to complement each other.
And that's why I'm even more curious about this question: Will Astra's on-site weight change really help the model absorb the new knowledge, new rules and human feedback it encounters after deployment and use it steadily in the future?
Around this problem, we did a series of complementary experiments. The results both gave us reason to continue exploring and reminded us of several different kinds of "model evolution".
What's the baseline next to 48.1%?
The main disadvantage of Qwen3.5-0.8B is that it is easy to get caught up in repeat cycles after starting the thinking pattern, going back and forth until the token runs out. Astra's intervention reduces the cycle and gives the model more opportunity to speak. The author himself makes it clear that repair cycles and increasing mathematical ability are two things.
So what do we do with a very simple contrast: not change the weight, just turn off the thinking pattern?
With the same set of MATH-500 tests, the same sampling and rating set, the accuracy was 48.95% after the Qwen 3.5-0.8B thought pattern was turned off without any editing.
The first two lines are from the original authors' experiments and the third line is from our independent tests. Qwen's official model card originally set 0.8B as the default non-thinking mode, suggesting that it is more likely to circulate in the thinking mode.
This controlling changes the problem itself. For a model that circles to deplete the token's upper limit, weighted editing did bring significant improvements; however, it seems to have no additional benefits compared to the almost zero-cost approach of "off thinking".
I don't think this will erase the value of WSP. Being able to locate and intervene in an internal behavioural mechanism is itself a demonstration of Astra's reasoning ability: it first discovers that the failure is largely due to re-thinking, then comparing the above context of normal progression to being trapped in a loop, extracting the "repeating direction" from the internal representation, finally finding which neurons are writing signals in that direction, and making directed modifications to the two layers of MLP.
It is also worth noting that weight is no longer just a set of numbers that can move slowly down a ladder, but can be read, understood, and directly rewritten by another model, such as reading text. In the past, we wanted to make the model stronger, but there was only one way: draw data, run training; now there is a second one: let the model read another model and then write it.
The real question is, how far can this chain extend? In the face of more complex, more widely distributed knowledge and capabilities in an unfamiliar model, can Astra still not find a corresponding mechanism and judge the benefits and side effects of a modification beforehand? The WSP provides valuable preliminary evidence, but not enough to show that it has established a stable, movable "parameter bias behavior" mapping.
Can changing the parameters make the model learn something new?
We used Qwen 3.8-27B as a common starting point to compare three states: unupdated models, Astra-edited after-weighted models, and models of new knowledge internalised through our supernetwork.
Testing was limited by Astra's call limits and execution speeds, and the scale was smaller: 10 samples were drawn from SQuAD, CharacterBench, and lingOLY, each examining knowledge internalisation, role and style adaptation, and skill internalisation respectively. Linking the materials required, the two methods of updating the same material; removing the material in response to see if the model can only respond to updated parameters.
Knowledge: amplifying the signal without changing the concept
On SQuAD, the supernetwork updated models were able to answer questions without the raw material, and neither group was successful.
There is one specimen that can illustrate the problem in particular. The material is Cestida: only two species are known to swim with body fluctuations and a clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed clawed
Astra's diagnosis was vague: it contrasted the material text with the activation differences described by worms in the MLP layer, selecting neurons that were more sensitive to the material and magnifying their output by 1.5 and 2 times, respectively. The weight did change, the wording did change, but the concept remained the same.
Enlarging a signal related to the material does not establish this new association with the "Cestida finger-belted mermaid".
Style: Write in parameters, not characters
On CharacterBench, after supernetwork updates, the model output clearly reflects the character and style requirements, while the Astra set does not.
The material requires the model to play Evolto: an arrogant, greedy, anti-Semite who uses questioning to emphasize a sense of superiority. Astra's practice is to calculate the activation gap of a certain layer under both the "role material" and the "materialless" states, and use it as a modification to write a single layer of weight. Writing was successful, and the weight stresses did indeed change, and the neutrality of arithmetic was unaffected.
However, the opening of the model reveals a flaw: self-introduction or "common questions", being questioned by the general leader or responding modestly, being asked to distribute the most precious things free of charge to everyone, or quick promises. Writing the same vector in a layer, it is not possible to evoke the same personality in all new questions, in all generating locations.
Skills: Use the rules to learn
LingOLY examines skills: give a model a short learning material in a foreign language, require it to learn the grammar and vocabulary rules in it, and then do translation and reasoning questions. In 10 questions, the unupdated model answers 1 question, Astra edits 1 question, and supernetization internalises 7 questions.
Astra's practice and the previous two, for example, yielded an example: Astra wrote a modification to the 43rd layer MLP, but it did not modify the test subject it set itself. The material required that "some change" be translated as returns, and the pre-edit and post-edit model produced money. After only trying this one candidate, Astra ultimately abandoned the modification and retained the original weight.
Learning a rule means being able to correctly apply it to unseen topics, different sentences, and every generating position. A local activation bias may allow the output near the topics to rely on the material, but this coverage is not guaranteed.
In three cases, Kar was in the same place.
Failure of knowledge, style, skills, and the three types of tasks all point to the same reason: Astra is a classic knowledge editing tool used when editing weights, such as activating, extracting directions, scaling neurons, playing a patch. This tool deals with single-point failures, such as suppressing repetitive cycles; but to turn a new material into a stable callable knowledge, personality, or rule, accuracy and updated parameters are far from being enough.
It can find material-related activations but cannot find the causal direction in which the final behaviour changes according to the material; it can change the weight but cannot control exactly what the model will say after the change. The tool itself is not strong enough, which is the most direct cause of these failures.
This set of experiments is enough to remind us that the internalization of knowledge, behavioural adaptation and skills learned in large models are not currently within the scope of Astra's real-world editing power.
To give an example, let's use the general large-scale model to give the model "a look at the disease", like asking a general practitioner to look at any disease with experience: sometimes it does cure the disease, such as inhibition of repeated cycles; but the dosage is not accurate, and the side effects are difficult to predict, and change a disease must be tried again.
In order to consistently write new knowledge, new styles, new rules into the parameters, it takes a specifically trained means to do this. You know where to change, how much to change, and how to change the model.
Let the agents organize, let the supernetwork write down.
Complementation of the two can occur in the same workflow: from the general large model as an editor, to the supernetwork as the writing tool in its hands.
Specifically, when faced with a new material, the agent can decide what content is worth internalizing, how to organize and sort these texts, in what order they are delivered to the supernetwork; after writing, run a round of evaluations to see if the new information can't be summoned, if the original capabilities are damaged, and then adjust the material organization or writing strategy accordingly, and try again.
The supernetwork is only responsible for the most precise of these steps: accurately matching text to the parametric space. Our supernetwork is end-to-end trained to learn "what kind of text, what kind of parametric changes correspond to a large number of samples".
Combined, the real-world reasoning capabilities that WSP demonstrates will not be wasted, and the more the agent is organized and the more accurate writing tools are useful, which is why we expect WSP to continue to evolve.
Back to the beginning of the figure: 0.016% weight change to 48.1%, WSP's "reading models, making diagnoses, changing weights" is a perfect link, which is an amazing step.
But changing weights is not the same as learning new skills. In these sets of experiments, the parameters do change, and the knowledge, roles and rules in the material do not become something that can be called on steadily. To really keep them, it takes an end-to-end training tool to accurately match the text to the writing tools in the parametric space.
So I'd rather see WSP as an important puzzle: General Large Model is responsible for organizing model updates, and the supernetwork allows each update to fall into the parametric space.
The moment the weights can be changed is just the beginning.