In LLM RL training, generation is becoming the most time-consuming link. When model output is increasingly long, the training phase faces arithmetic constraints, and increasing the number of training cards can effectively alleviate arithmetic pressure, but the reasoning part is limited to the display bandwidth, stacking on a single logical instance and quickly hitting the scalable ceiling.
The Adaptive Scheduling Pipeline, developed jointly by the Shenzhen Institute of Big Data and Huawei, is an adaptive scheduling pipeline for multi-instance asynchronous RL training. The first thing it does is to break down generation and training, but the second thing that's really important is to use a suite of queue-perceived scheduling strategies to eliminate the three types of bubbles in multi-instance rollouts: padding, skewness, bubbles and sorting bubbles, reduce the waste of computational resources, and improve RL training performance.
It is worth mentioning that ASP is entirely based on the construction and verification of domestic production uplift computing power, using the uplift 910B3 NPU and MindSpeed-RL ecosystem. According to the public information retrieval, this is the first work received by NeurIPS, based on the national production uplift computing power model RL training acceleration system, as well as the first outcome of the uplift framework to the A-Class summit. As a leading enterprise in domestic AI chips, the uplift ecosystem thus proves that it is fully capable of doing international cutting-edge research and has won the recognition of universities in the field of AI.
As a result, the output throughput of ASP improved by a maximum of 222%, and the single-turn time decreased by a maximum of 76%. In expansionary performance, the change in turnover time after increasing the number of cards is close to linear, and there is still no apparent convergence degradation after a strategy of about 3.3 training steps lag.
The system is already running in real-world RL training tasks in the telecommunications sector, and the overall training cycle in the Baccarat Rising Group has been squeezed from more than 3 months to about 2 months, and the output field enhancement model can be directly put into operation. Currently only based on MindSpeed-RL, it will be moved to verl framework and set up open source.
The overall idea of ASP. One offline pre-processing, while targeting three types of rollout bubbles: first predicting the length, then constructing minibatches of the same length, and finally organizing the long-short mixed batches into load-balanced scheduling groups.
Address of the paper:https://github.com/EntongL/ASP/blob/main/AdaptiveSchedulingPipelineforMultiInstanceAsynchronousReinforcementLearning.pdf
Background
What became a bottleneck
RL training has become a central link in enhancing the capabilities of large models. It relies on responses and reward signals generated by the model itself to train in return for better coordination and reasoning skills.
But every time the strategy is updated, the whole process has to go through, rollout generation, reward scoring, log-prob computation, reverse propagation. As output grows longer, rollout generation becomes the dominant bottleneck, occupying a large part of the total training time.
Currently, the most mainstream practice is mixed cycling. Each round is done with reasoning, and the resulting samples are used for training. The structure is simple and the samples are close to the current strategy, but the drawbacks are obvious.
When the trainer updates the parameters, no new samples are generated, and the card is stuck. Reasoning and training repeated switching patterns, the same batches of machines have to be re-spliced. What is more troubling is that the system characteristics of the two stages are completely different. Reverse propagation is computationally limited and prefers large-scale synchronous parallels; while reasoning generation is open storage bandwidth limited, adding resources to single reasoning instances, earnings quickly reach the top.
It's only half the solution.
The asynchronous and training-driven separation of the flow lines, allowing rollout generation and training to take place simultaneously, solves part of the above problem. But separating generation and training does not automatically make multiple instances of rollout generation efficient and stable.
When multiple actors simultaneously decode different lengths of responses, the system must simultaneously address two overlapping goals. One is to maintain high utilization rates within each logic minibatch, the other is to keep the overall producer-to-consumer relationship (i.e., rollout generation and training) and training queue stable. So the real question is not just how to separate the generation and training, but how to schedule the rollout step so that the differential generation is both fast and controllable.
To achieve efficient timing, we analyzed the bubbles present during rollout and training in the current scenario and divided the bubbles identified into three categories:
1. Padding bubbles. The prompt in the same minibatch has different response lengths. At the short end, the actor receives the longest, actually useful token output, which is much smaller than the padded calculation calculated according to the longest response.
2. Skewness bubbles. The output length of LLM is long tail distribution, and a few super-lengths are generated that directly compress the size of the valid batch later in decoding.
3. Sorting bubbles. Sorting the prompt in a predicted length order to mitigate the first two problems, but introducing the third. If short minibatches are forwarded and long batches are forwarded, the generation rate will change over time, and the Training Queue is filled quickly and then slowly exhausted.
The problem is here. It's a good thing for a single minibatch to be sorted by length, but it can be harmful to the time stability of the entire flow line. To address the above three bubbles, we proposed ASP, which predicts and adjusts the sorting combination of samples through offline pre-treatment, reduces the waste of resources caused by the bubbles, and improves overall training performance.
Method overview
Three-step offline scheduling while eliminating three types of bubbles
The idea behind ASP was to convert the schedule from "load balanced algorithm details" to plug-in components in asynchronous RLs. It used the output length prediction to sort the prompt, constructed the length homologous reasoning minibatch with dynamic planning, and then mixed the length minibatch with the balanced batch-group sequencing to bring the prediction processing time of the adjacent schedule closer together.
The entire scheduling is offline pre-processing completed once before the training, using only predicted lengths, without increasing any online scheduling costs during the rollout, and the specific steps are as follows:
Step 1, output length prediction. The first target model runs a small part of the prompt, collects the prompt with the corresponding length, and minimizes a small model with the same series but fewer parameters as the target model. We use Qwen 2.5-1.5B to make the length prediction, sorted by the predicted value.
Step 2: Construct minibatches of equal length. Fixed batch sizes will still be wasted when large intervals are encountered in the sorting list. ASP allows batch sizes to float slightly near the benchmark, using dynamic planning to find the partition that makes the average utilization rate the highest and minimizes padding bubbles.
Step 3, balancing batch-group sorting. The last step is to rearrange the minibatch so that the total predicted load for each group is roughly the same. This is no longer "all short batches first, all long batches later", but rather a long-term turnaround, with the continuous scheduling group's predicted exhaustion approaching, and the sorting bubble disappearing.
To connect the timing to a backward version controlled asynchronous system
The scheduling is only half. The ASP divides the entire system into three parts. The Training Unit (TU) is responsible for reward computing, log-prob evaluation and strategy updates; one or more Generation Units (GU) each carry multiple generating actors; a Control Unit (CU) deployed at the same address is responsible for sending a well-scheduled minibatch and passing the completed sample to TU through a controlled Training Queue.
Training Queue is key to stability. It absorbs short-term fluctuations in generation time while exposing queue occupancy to CU, adjusting the pace at which CU sends samples: the samples arrive faster than TU consumes and the sends are streamlined; at the bottom of the queue, CU continues to generate actors and produce more samples.
Asynchronous rollout will also cause a second problem, with samples likely to come from models that are slightly older than the current training weight. ASP processes by limiting the number of delayed weighted versions. TU regularly exports weighted snapshots after completing the training step, converts the cost-reasoning available format, and asynchronously reloads the actor; if an actor is decoding, it first unloads the current minibatch and reloads, avoiding a complete shutdown of all generating workers.
In this way, it maintains the core benefits of understanding the benefits of coal, generates uninterrupted output, and controls the strategy to the extent that it is observable, with clear boundaries.
ASP system architecture. Blue real-time lines for data/sample streams: CU sends a good-scheduled minibatch to each GU, returns the completed sample after the actors are generated, enters the Training Queue via CU, and is consumed by TU; green real-time lines for refresh paths; Last Weights, asymmetrical loading of each GU; color dot lines for queues occupy feedback; controls send rhythm).
The results of the experiment
The ASP, based on the MindSpeed-RL implementation, uses GRPO to conduct RL training on the elevator 910B3, using a total of eight nodes, each with eight NPUs, for a total of 64 NPUs. The evaluation models are MoE's Qwen3-30B-A3B and compact Qwen3-32B, as opposed to the lifetime mixed baseline MindSpeed-RL, and the asynchronous decomposition baseline verl.
End-to-end performance
On Qwen3-30B-A3B, ASP switching time was 216s, down 76.7% on 927s for MindSpeed-RL; 536.7s for verl, down 59.9%.[3]
On Qwen3-32B, ASP switching time was 230.7s, down 75.1% on 928s for MindSpeed-RL; 615.1s for verl, down 62.5%.[3]
The increase in throughput is equally evident. ASP on Qwen3-30B-A3B reaches 58 tokens/s/NPU, the mixed baseline is 18, verl is 44, corresponding to an increase of about 3.22 times and 1.31 times respectively.
The benefits of MoE and condensation models are that the improvements come from the system design itself, i.e. the "rollout of multiple instances of decomposition coupled with coordinate perceptual scheduling", rather than the random outcome of a particular model.
Table 1 is the end-to-end conversion time of the paper original Table 3. In the brackets, the node is configured, T represents the training node, and G represents the generation node.
Scheduled dissolution
By separately dismantling the scheduling strategy, the ASP averages a reduction in turnaround time of 11%. The most noteworthy thing here is not the savings on padding. By balancing the predicted load of adjacent batch-groups, the ASP also smoothed the minibatch completion time and reduced the queue oscillation. This is important in asynchronous scenarios, as synchronous completion would compress the CU/TU passageway, triggering unstable queue behavior; while a smoother arrival process would bring the waterline closer to its own equilibrium workpoint.
Qwen3-30B-A3B) After the start of the scheduling, the fluctuations in the corresponding indicators in the queue converged significantly, with an average downtime of 11%.
Scalability
As the number of NPUs increases, the advantages of decryption frameworks become increasingly apparent, as rollouts accelerate with increased resources. As shown below, ASPs take less rollout time at each test scale, and the more resources are generated, the greater the gap. From one GU node to three or four GU nodes, the rollout time decreases from 776s to 288s, 216s, respectively, 2.69 times and 3.59 times faster, and approaches ideal linearity. This is the value of multi-instance rollouts.
Expandability: Qwen3-30B-A3B. The left graph shows the change in generation time with the total number of NPUs and the right graph shows the change in the number of NPUs that are generated. The ASP is kept lower on each scale until the bottleneck is shifted to the training side.
Retarded version restricted convergence
Asynchronous execution will hurt learning, a question that all asynchronous programs must answer. Under the configuration of 1 TU node plus 3 GU nodes, the effective off-policy of the test is approximately 3.3 training steps away. Running 30 rounds GRPO, ASP and the maze integrity platform approaching the hybrid baseline, ASP has a brief initial delay due to pure preheating; Maze correctness is also the same conclusion.
Observed in limited lag, no visible convergence degradation. Line perception scheduling, line boundary and delay weight refresh, stabilizing asynchronous flow lines in a small lag work area, while taking in the ingestion gains.
MAZE-REASONING-GRPO-V0.1). Following a lag of about 3.3 training steps, the ASP reached a platform close to the hybrid baseline, with no visible degradation.
Industrial landfall
In addition to the experiments in the paper, the ASP has been deployed in real-world telecommunications scenarios based on the Baccarat or even the Baccarat raising cluster, the model being Qwen3-235B-A30B. In this deployment, the overall training cycle was shortened from more than 3 months to about 2 months, and the training field enhancement model has been used in the telecommunications business such as knowledge querying, tool calling. Due to the confidentiality of processes, data, and deployment details, the paper only uses it as a definitive proof of practicality and does not include controlled trial comparisons.
The purpose of this set of capabilities is not only to improve the efficiency of RL training. Telecommunications are heavy assets, strong professional scenarios, models need to continuously absorb specialist data such as bulk work orders, operational knowledge and customer interviews, whereas traditional training cycles are month-on-month, and models often fail to keep up with the fast pace of business. ASP greatly compresses the training cycle to make the "fastest business model understand business models" a reality. The same idea can also be transferred to highly specialized industries such as government, electricity, finance, manufacturing, and so on.
Summary and Prospects
The question ASP wants to answer is very specific. After training and recommending an understanding of how to make multiple instances asymmetric RL run better. Generating needs to be efficient within each actor and stable on the Training Queue. ASP supplements this layer with a single offline queue sensing scheduling, predicting length, constructing homogeneous minibatch, balancing group sequencing. It brings a more practical path, making multiple instances RL training both fast and manageable.
The current prototype runs in the MindSpeed-RL ecosystem of the elevated 910B3, with limited resources and a maximum scale of 128 NPUs. The next step will be to continue testing this scheduling mechanism in larger clusters, stronger strategic drift and course learning scenarios.
References
[1] Adaptive Scheduling Pipeline for Multi-Instance Asynchronous Reinforcement Learning. Advances in Neural Information Processing Systems (NeurIPS), 2026.
Introduction by the author
Luis Henrique Zuin Ruiz: Huawei is working as an intern and a graduate student in mathematics at the Paris Polytechnic Institute, focusing on mathematical modeling and the application of AI.
Salah Chikhi: Mathematics major at the Paris Polytechnic Institute is a graduate program where the main focus of research is the combination of mathematical theory and deep learning.
Li: He is an engineer at the Shenzhen Institute of Big Data, where his main research focuses on AI infra acceleration optimization and training acceleration optimization.
Research scientist, director of the Shenzhen Institute of Big Data, whose research areas include big model acceleration, vector computation and knowledge search, graph computation, and more.
