In the field of LLM/MLLM, reinforcement fine-tuning based on verifier rewards, RFTs, such as GRPOs, show significant advantages in triggering complex reasoning abilities. Due to the characteristics of on-policy optimization, many studies have pointed out that RFTs are naturally more resistant to forgetfulness than traditional supervised fine-tuning (SFTs).
Recently, a team of professors from the University of Hong Kong (Hong Kong) systematically explored the bottlenecks of enhanced fine tuning in continuous visual learning, uncovered the trajectory-level drift agnosticism flaws hidden under the standard GRPO framework for the first time, and proposed the all-new RFT algorithm Retention-aware Policy Optimization (RaPO).
Overcoming Catastrophic Forgetting in Visual Continual Learning with Reinforcement Fine-Tuning
Admission meeting: NeurIPS 2026
Address of the paper:https://arxiv.org/abs/2605.09640
The code repository:https://github.com/LMMMEng/RaPO
First author: Ph.D. student from the University of Hong Kong.
A. Research motivation: Why is there still a problem of forgetting in standard GRPO continuing learning?
In the open environment of the real world, AI systems must be able to continuously learn new tasks in a timely manner while avoiding catastrophic forgetting of previously acquired knowledge. Traditional SFTs often require the introduction of complex parametric isolation, dynamic architecture expansion or distillation penalties, but severe conflicts of new and old knowledge are still likely to arise in rigorous, rehearsal-free data recycling.
To this end, the research team constructed a typical Rehearsal-free Few-shot Class-Incremental Learning experiment using the classic Qwen2-VL-2B-based model on the ImageNet-R dataset: The categories were divided into 10 non-overlapping subtasks, each strictly limited to providing only 5 labelled samples.
Figure 1: Left: No relay class increment set the GRPO to be entirely superior to SFT, but its own oblivion remains apparent; right: candidate trajectory to receive the same task reward, the difference in KL scattering relative to the previous task strategy, and increased as the task progresses.
The results of the experiment are shown in Figure 1 (a): Although the overall performance of the GRPO outperforms the SFT, this confirms the anti-forgetting properties of the RFT; however, as the number of tasks increases, the final accuracy of the GRPO still shows an irreversible sharp drop, suggesting that the standard RFT still fails to address the stability-plasticity imbalance in continuous learning.
To trace the deeper causes of this phenomenon, the team further tracked GRPO's microscopic behavior during strategy updates and discovered a key mechanism flaw: Trajectory-level Drift Agnosticism: In the process of GRPO optimization, a set of candidate trajectories were generated for the same prompt input, the current strategy network Actor, which generated a sample of candidate trajectories Rollouts.
Partial trajectories show low drift: at the same time as the new categories of learning, the model's reasoning patterns and characteristic spaces highly retain the historical distribution of the previous stage.
Partial trajectories show high drift: although the new task tag has just been hit, there has been a dramatic shift in its parametric space and representation distribution, completely contrary to historical knowledge.
Because the standard GRPO calculates a relative advantage based only on Task Reward, which results in the optimizer giving an approximate credit assignment to these correct but highly drifting trajectories. As the task sequence evolves, this unintelligible tangent to the drifting trajectory of the trajectory uninterruptedly allows the model to slide into the high drift parameter region, inevitably causing serious catastrophic forgetting.
The team designed GRPO variants in two opposite directions, intervening directly within the group of the rollout pairs: variable 1 retained only the positive reward for the low-drift trajectory, and variable 2 in turn retained only the positive reward for the high-drift trajectory. As shown in Figure 1 (a), the results were obvious: the former significantly mitigated forgetting and the latter significantly exacerbated forgetting.
Method 2: RaPO reshapes the strengthening of the credit allocation and scale to 1.
To address these issues, the research team proposed RaPO (Retention-aware Policy Optimization). RaPO did not introduce additional parametric modules or complex fine-tuning architectures to the multi-modal backbone network, but instead evolved from reinforced learning in Reward Shaping and numerical stability, building on two core components:
1. Retention Reward: Re-creation of boundary rewards based on orbital distribution drift
In the past, the conventional idea of countering forgetting has often included some positive elements in total Loss, such as KL loss. However, the overall KL loss is independent of the specific reward received by each Rollout, and it cannot distinguish between candidate trajectories within the same group that are similar to the quality of the task but have different degrees of forgetting.
The formula calculates trajectory-level drift: the current task strategy is calculated by token for each candidate rollout compared to the odds of the previous task strategy, and the larger the average of the entire trajectory, the greater the drift, indicating the way the trajectory is generated as it deviates from the previous task model. Previous variation experiments have shown that keeping Task Reward trajectories high and drifting small can effectively counteract forgetfulness, so the first step in RaPO is to precisely trajectory-level drift.
The formula defines Retention Reward: the smaller the drift, the closer to 1; the larger the drift, the lower to zero. Thus, each rollout can be given a score specifically for measuring the level of retention of old knowledge in addition to the Task Reward.
The formula gives the overall reward of RaPO: Task Reward and Retention Reward. Unlike conventional GRPO, which focuses only on how to learn current tasks, in RaPO, a task can respond to a new task and maintain a low drift path to a high score. When the Task Reward of multiple paths within a set is closer, the smaller drift will receive a greater Advantage, is strengthened, and the strategy gradient will therefore continue to flow towards learning new tasks, while not forgetting the old knowledge gap.
2. Cross-Task Advantage Normalization (CTAN)
The second component of RaPO addresses the optimization shock at the task frontier: old task learning ends, most rollouts match, standard gaps in group rewards are minimal, advantages are unusually large; early on in new tasks, the rewards plummet and the gap increases, and advantages are compressed.
Figure 2: CTAN uses the EMA smooth reward standard for cross-task duration (a), the magnitude of the advantage (b) and training, evaluation rewards (c), and (d) there is no longer any tremendous vibration at the mission boundary.
For this purpose, CTAN replaces the instant batch differential with an EMA standard differential that is maintained durably across tasks:
Each step absorbs the current batch with only a very small weight (1−β) and is dominated by historical information.
After substituting the advantage formula, the molecules remain relatively ordered within the group, with the subunits no longer bouncing violently; they are saved and passed on to the next task during task switching, providing a stable advantage scale for the entire learning sequence.
Three: Experimental verification: The five major visual continuum learning benchmarks have been fully exceeded
To comprehensively measure the universality of RaPO, the research team relied on MLLM's free-form text generalization capabilities to conduct detailed contrast assessments across images, video, and intensive predictive scenarios at a rigorous 5-shot low resource setting.
1. CIL for Image Classification
Table 1: Class augmentation image classification results. RaPO's ultimate accuracy is highest and forgetting rate is lowest across four datasets and two tasks.
On the four classical benchmarks of ImageNet-R, ImageNet-A, TinyImageNet and CUB-200 (based on Qwen2-VL-2B), the performance of traditional SFTs combined with continuous learning classical constraints (such as the L2 rule, EWC, LwF) was not as good as expected, largely falling below 40% after 10 to 20 consecutive tasks. Although the standard GRPO showed some advantages (last accuracy of 74.67% on the ImageNet-R 10 mission), the forgetting rate was still 20.02%. After the introduction of RaPO, the final accuracy of the model improved significantly to 85.92%, and more importantly, the forgetting rate was pressed to 4.69% in extremely low water.
2. CIL for Object Detection
Table 2: Results of the COCO 2017 class growth target detection.
Compared to image classification, target detection requires not only semantic identification, but also spatial coordinate prediction that relies on finite particles. The team used the Qwen2-VL-7B spatial network. In the COCO 2017 dataset, the Box AP of RaPO was 19.31% and 19.12%, respectively, up from the standard GRPO of 4.67% and 4.82%, respectively, while the forgetting rate was lowered to 1.39% and 1.37% respectively. This demonstrates that RaPO's retention mechanism can also be seamlessly adapted to complex structured detection rewards calculated through Hungarian Matching.
3. Video Classification and DIL (Video Classification and DIL)
Table 3: Classification of video by category (UCF-101 / Kinetics-200)
Table 4: Field augmentation learning results (image classification + target detection)
In the UCF-101 and Kinetics-200 video classification tasks, RaPO achieved the best accuracy after long-task stream learning. In the DILs, such as DomainNet and Pascal cross-domain target detection with 6 domains, RaPO is also the overall leader; in the Pascal series of cross-domain detection tasks, RaPO achieved an average AP of 37.18%, even exceeding the SFT upper limit of using multi-domain data for joint micro-modulation (Joint-Training) of 35.40%.
4. Visualized insight: an in-depth breakdown of the anti-forgetting mechanism of thinking pathways
Figure 3: Qualitative contrast of class increment classifications. After learning the last task, the GRPO misidentified the old category several times, and the answer to the RaPO remained unchanged.
Figure 4: Qualitative contrast of class incremental detection. After a complete departmental task, the GRPO detects new and old classes of bicycles less than those learned earlier.
The paper intuitively demonstrates the evolutionary process of reasoning within the model through qualitative sampling: after experiencing complete multitasking flow training, standard GRPO was able to confuse its <think> mind-chain with prior knowledge learned by subsequent tasks in the face of earlier older task samples (e.g. by forcing the early retention characteristics to complement later learned brain-based white-headed sculpture sequences, or by completely missing previously mastered titanium bicycle titanium classes in target detection). RaPO was still able to strictly maintain the identification chains for older task objects in output, accurately capturing key localization characteristics.
Four words.
RaPO opens up a whole new perspective on parameter mitigation and continuous learning for MLLM in a streamlined environment. This work breaks the consensus of the RFT Natural Immunity Disaster Forgetting Curve, which precisely locks the key bottleneck of fluid trajectory-level drift unknowns, and reconstructes the logic of reinforced credit allocation and differential differentiation in a simple mathematical form. RaPO is able to plug seamlessly into existing PPO/GRPO training frameworks and has the potential to provide a solid algorithm base for building lifelong learning and historical experience-based adaptive fitness and visual systems.
For more theoretical proofs and experimental results, please read the original paper! The code is now open source on GitHub and is welcome for community discussion.
© THE
