TreeOPSD team posted
QbitAI is public.
To complete an in-depth study, an agent often has to repeatedly search, read a web page, and then decide what to look for next.
In this way, it may have found the key information early, but it spent dozens of rounds searching the unrelated web pages and finally still got it wrong.
If the punishment of failure is applied on average to the entire process, the correct retrieval will be followed by the punishment; conversely, a trajectory of ultimate retrieval may also be rewarded with many unnecessary operations.
FromThe team of Prof. Xi Yong from Qinghai UniversitywithWhat is it?According to the researchers,TreeOPSDIn addition to this, there are a number of ways to improve the strength training for long-range tasks.
In Qwen3-30B-A3B-Thinking-2507, the four Deep Research assessments of TreeOPSD reached an average of Exact Match.21.20%This is an increase of 16.23% from the GRPO.4.97 points。
Background: Credit allocation and policy obsolescence in long-term Agent reinforcement learning
Improvements in long-range missions are key.Credit allocationThe final answer is right and wrong, but it doesn't have to be clear which steps have worked and which have gone wrong.
This problem is particularly evident in RLVR training, which relies on verifiable endpoint rewards. The longer the task, the more intermediate decisions, and the more difficult it is for the entire trajectory to share an advantage signal to reflect the contribution of different operations.
There is also a time gap that can be easily overlooked: Rollout comes from a sampling behavior strategy, but the same amount of data is used for multiple continuous mini-batch updates. The model parameters have changed, and subsequent updates are still calculated along with the sampling credits, which may go too far.
TreeOPSD deals with these two elements, first estimating the contribution of each round of operations from the trajectory and then determining how much of the model needs to be updated depending on the current state.
Technical method: optimization of post-test distillation based on tree-based credit estimates
Tree-shaped trajectory sampling and rotational credit estimates
Sampling starts with several complete Agent tracks. The algorithm then returns to the intermediate nodes that have been accessed, generating different follow-up paths from the same predecessor.
Sharing forecasts means that branches have the same prior experience, with differences in subsequent selection. The algorithm standardizes the endpoint rewards for the same problem into trajectory advantages, then tree feedback, and combines the likelihood weighting of behavioral strategies to estimate the number of rounds of interactions.This is the first time I've been able to do this.。
The default configuration is four initial complete trajectories, each of which expands to two branches, forming a total of twelve leaf trajectories.Eight full Rollouts expected to generate a budget equivalent to GRPOThis does not mean that the actual training takes the same amount of time for both methods.
△The left side constructs a trajectory tree from the shared forehead and passes the final rewards back to the middle node; the right side converts the rounds of credit into the target distribution, adjusting the current strategy in continuous updates.Teacher's goal of building and post-testing distillation
Estimating roundabout credit does not mean that the training problem is solved. TreePO also uses tree-based credit, but during continuous updates, it still promotes the strategy of the credit weighting determined during sampling.
TreeOPSD no longer directly uses this fixed credit for each update, but instead maps it asBernoulli (Divine) teacher goal distributionThe objective here is not the probability of the next token, but the extent to which the rotation should change relative to the sampling strategy. The positive credit corresponds to a target value of more than 1/2 and negative credit to a target value of less than 1/2; the deviation range is controlled by the target radius.
The so-called "faculty curve" is based on the goal of constructing a trajectory, rather than another training of a large model. The algorithm looks like a sequence of current strategies relative to behavioural strategies rather than mapping them into student-side binary distributions, then using the Forward KL.(Forward to KL dispersion)Let's get it closer to the teacher's goal.
△Using behavioural strategies as benchmarks, calculate changes in the current strategy's interaction sequences and build target matching signals on the student side accordingly.Self-correcting strategy updates based on dynamic residues
In each subsequent mini-batch, TreeOPSD recalculates the gap between student distribution and teacher goals.
When the target has not been reached, the deficit continues to drive the update; the closer the target, the smaller the drive force. Once the target is reached, the deficit goes to zero; if the target is exceeded, the direction of the update will reverse.
The paper demonstrates that at the sampling point of the current strategy, this target matching will restore the direction of the tree-like strategy gradient. After the model is updated, the gradient weights change with the residual, not necessarily repeating the original driving force.
△Fixed credit continuously drives updates; TreeOPSD changes the direction of updates when students exceed targets based on the strength of the balance adjustment between student distribution and targets.Experimental results: multiple-choice and in-depth assessment of task performance
Experiments include Multi-Hop QA(Please answer)Deep Research uses 6102 deployment rounds in OpenResearcher, allowing up to 60 rounds of interaction. When evaluating, BrowsComp-Plus uses closed-end content retrieval; BrowsComp, GAIA, and xbench-DeepSearch are used for open web search migration testing.
The baseline includes GRPO, GSPO and TreePO.Similar sample trees, rounds of credit and optimized batch settingsThe main difference is whether to use the defined teacher goals and dynamic residual updates.
△The two model scales are the exact match results of the four assessment sets in Multi-Hop QA and Deep Research, with Avg being the average for the corresponding four.On the 4B model, TreeOPSD's Multi-Hop QA is divided by the mean44.13The average score for Deep Research increased from 12.18 to 43.37.15.45In the 30B-A3B model, the multiple-choice score was increased from 44.80 to 44.80.45.93Deep Research has been upgraded from 16.23.21.20。
Even in comparison to the TreePO with the same tree sample, the average score of 30 B-A3B models in Deep Research has risen from 17.59 to 21.20.It's up 3.61 points.This group of contrasts is more likely to reflect the role of dynamic goal matching.
△The 4B and 30B-A3B models vary with the number of training updates in multiple-choice and evaluation accuracy on the BrowseComp-Plus task.The paper also compares the performance of the 30B-A3B model on the BrowseComp-Plus by actual training time. At intervals not exceeding the total training time of the GRPO, the accuracy curve of the TreeOPSD remains above the GRPO. It also produces longer Agent interaction trajectories, but longer interactions do not equate to faster completion of tasks each time.
△The correlation between the accuracy of the training time budget and changes in the number of rounds of Agent interaction.Meltdown analysis: credit definition, target mapping and sampling configuration
In Qwen3-4B's BrowseComp-Plus fusion experiment, the pointwise credit of retaining the sub-node's own credit is better than making the sibling node a relatively centralized listwise credit.
△The pointwise and listwise credit allocations, as well as the dissolution of different teacher target mapping methods, are contrasted.How the sampling budget is divided is equally important. Under the same expectation of generating a budget, the average score of 8 independent trajectory 4B models for Deep Research is 11.28;Four initial tracks, two branches each.The number of intersections is not as good as the number of intersections that can be seen.
△Budget allocation of different complete trajectories and the number of forks, as well as the performance of missions that set the target radius.The appendix also tested Qwen3-1.7B and Llama-3.2-3B-Instruct. The multiple-choice mean of the two jumps from Exact Match increased from 25.70% to 26.88% of the GRPO, respectively, and from 31.73% to 35.58%, indicating that the improvement was not limited to the Thinking model used in the main experiment.
△Assessment results on other model skeletons and training stability experiments.Summary and prospects: Finite grain credit allocation and stable optimization of long-term agents
TreeOPSD does not set up another reward model, but instead extracts alternate credits from different follow-up results from the same predecessor and converts the credits intoGoals for updating with current policy changesThe first step is to figure out which action has been awarded and how many have been updated.
It does not require alternative training Critic, nor does it rely on round-the-clock manual process rewards or stronger external teacher models. Experiments have shown that such treatment is particularly effective in long-term Deep Research tasks. Subsequent research remains to be done on historical trajectory replication, adaptive adjustment of target radii, and forgetting issues in cross-task continuous training.
Title: Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning
The paper was written by Heming Zou, Cheems Wang, Yixiu Mao, Yun Qu, Yuyi Guo, Zeyuan Liu, Chenglin Li, Weijie Liu, Saiyong Yang, Xiangyang Ji.
It is located in the city of Tsinghua.
Link to the paper:https://arxiv.org/pdf/2610.23181
