Coding Agent is making big model reasoning face new performance challenges.
In software engineering tasks, models are required to continuously read code, generate modifications, call tools, and decide what to do next based on execution results.
These steps are often sequentially dependent, and even if the computing resources are sufficient, the waiting time in each round of Token generation continues to accumulate, affecting the performance of the entire task.
How to keep the process of generating single-barreled requests running efficiently and continuously is becoming an important direction for optimizing reasoning systems.
Recently, the IQuest Research Institute (IQuest Research) has released and open-sourceed a dedicated reasoning running optimization engine for IQuest-Q1.IQuestRT。
According to an official technical report, IQuestRT has been implemented under the single-request conditions of 8K high-performance GPUs.The test decoding throughput of 910.70 tokens/s reaches 2.94 times the vLLM reference configuration。
This performance comes from multiple underlying optimizations: GPU autonomously controls decoding cycles, dynamically determines speculative prediction depth, anticipates expert weights that may be used, and reduces synchronous waits in cross-GPU communications.
Another noteworthy detail: AI Agent is also involved in the underlying research and development of IQuestRT. Multiple Agents are simultaneously implementing code, performing performance experiments and debugging, and in just two weeks the resulting optimization has surpassed the team's previously finely tuned computational flow stream.
From the speed at which models are generated to the R&D method of reasoning engines, IQuestRT demonstrates the two technological pathways that high-performance reasoning systems are exploring.
Contrast IQuestRT single request decoding throughput with vLLM reference configuration
A single request for reasoningWhy should it be specifically optimized?For a long time, big model reasoning has often been measured by metrics such as throughput, latency, etc. However, in applications such as Coding Agent, performance optimization needs to focus further on how tasks are performed.
For example, when a user requests a model to modify a code project, the model may first retrieve the file, understand the problem, reproduce the code, call a test tool. If the test fails, it will also need to continue analyzing and modifying it based on errors.
Throughout the process, the content generated by the model continually determines the next action. For such continuous interaction tasks, the speed of each generation round affects when the next operation begins.
IQuest-Q1, which is served by IQuestRT, is a basic open-source model for the Intelligent Innovation Institute. It uses a rare hybrid MoE architecture with a total parameter of about 320B, with each token activating about 15B parameters, primarily for code generation, software engineering, reasoning, and multi-step tooling.
In the process of reasoning, IQuest-Q1 uses the recurring multi-Token prediction (MTP) mechanism. The model is repeatedly running a draft network, continuously submitting multiple candidate tokens, and then unified verification by the main model, thus increasing the number of valid tokens that can be generated in a single calculation.
However, the multi-Token forecast also brings new engineering problems.
Each round of generation requires candidate verification, sampling, cache updates and follow-up predictions, and involves the launch of a large number of GPU computing tasks, model weight reading and cross-card data exchange.
These operations may take only a few tens of microseconds at a time, but they occur repeatedly during continuous generation.
For single-request, small-scale scenarios, how to reduce the waiting time caused by scheduling, synchronization and data transfer, thus becoming an important factor affecting performance.
IQuestRT's idea is to systematically optimize the entire speculative decoding process around the specific computational structure of IQuest-Q1.
Let the GPU itself start the next round of computingIQuestRT first changed the execution relationship between GPUs and CPUs.
In some conventional logic implementations, after a round of computing is completed, the CPU needs to read the results, update the status, and launch the next round of tasks. In this process, the GPU may need to wait for the host to complete the corresponding operation.
IQuestRT will put more control logic directly on the GPU.
Candidate Token verification, sampling, cache updates, and the next round of computing start-up can all be done inside the GPU. The CPU only needs to start the first round of tasks, wait for the entire generation process to finish, and then read the results.
In this way, the host control intervals that occur repeatedly during the sequential decoding process can be shortened.
According to the paired repositioning experiments in the technical report, after eliminating the control interval on the main terminal, the calculations saved about 22 tons 25 microseconds per round; the subsequent streamlining of the execution boundary resulted in a gain of about 3 microseconds.
Contrast between traditional CPU control cycles and IQuestRT GPU autonomous decoding cycles
GPU autonomous control is also used for speculative decoding.
More candidate tokens are proposed in advance, allowing the main model to verify and accept more content, but the additional predictions themselves need to be calculated. If the expected returns of subsequent candidates are insufficient, continuing generation may increase the time taken instead.
To this end, IQuestRT has introduced a deep strategy for dynamic speculation.
Each round of the calculation fixes the first three prediction depths, and then the GPU estimates whether it is worth the additional computation costs based on information such as the generated candidate tokens and their probabilities, etc.
The process generates a maximum of seven candidate tokens.
This design enables the system to predict depth based on the current generating state and to stop additional computation when earnings are inadequate.
From reducing CPU involvement to the next step in dynamic decision computing, IQuestRT attempts to give GPUs more complete execution control over continuous decoding.
From MoE expert weight to cross-card communicationContinued Compression WaitingIn addition to GPU scheduling, IQuestRT also deeply optimizes expert routing and parameter reading for MoE models.
The MoE model will require experts to participate in the calculations depending on the input dynamic selection. For small batch reasoning, waiting for the expert weight from the standby load to the location required for the calculations can lead to significant costs.
IQuestRT employs a pre-processing strategy: before a formal routing result is generated, experts are expected to be called in and load the corresponding load into the L2 cache of the GPU.
When subsequent calculations really need these weights, some of the data is ready.
However, pre-processing itself also requires computing resources.
Through experiments, the team found that in the Sliding Window Attention Layer, pre-emption can bring benefits when the validated position is four to five; when the position increases to six to eight, the pre-emption costs outweigh the benefits.
Therefore, IQuestRT enables pre-loading only on the appropriate computing path.
Furthermore, the prediction results only determine which weights are put in cache in advance, and the actual experts involved in the calculations are still determined by the formal routing results of the model.
Another optimization direction is to reduce the startup of the independent GPU Kernel.
The research team integrated expert routing calculations that were originally performed separately into the attention-related computational processes. The first version of the routing convergence scheme saved 88.5 microseconds per round in the trajectory repositioning test.
There is also room for optimisation for cross-GPU communication.
IQuest-Q1 requires multiple GPUs to complete the calculation in an 8-way parallel environment. Each GPU exchanges data and confirms whether other devices have already written the results.
IQuestRT has designed a way to synchronize and assign NaN labels.
NaN is a special numerical marker in floating-point computing. The system fills in the NaN in the output buffer area before the GPUs complete the calculation and then covers these locations with actual data.
If there is still NaN in the result of cross-GPU mapping, it indicates that at least some of the data has not yet been written; when the result no longer contains NaN, the system can continue to process the data that is already available.
This design combines data readiness judgments with result assignments, reducing additional synchronization operations.
IQuestRT synchronizes and assigns cross-GPU data based on NaN labels
These optimizations cover computational scheduling, data loading and communication links that occur repeatedly in the reasoning process. A single improvement may save only tens of microseconds, but cumulative gains will affect the overall generation speed as the number of rounds and model layers increases.
Of course, changing the computational path may also affect the floating-point numerical results.
The technical report makes it clear that some routing convergence operations use different numerical calculation methods than the original ones; the effects of FP8 activation and regression sequence changes still need to be independently verified in combination with higher-precision reference implementations.
This is also an important issue that needs to be addressed by the underlying optimization of the reasoning system: continuous checking of numerical changes, acceptance of tokens, and final result generation while shortening computation time.
910.70 tokens/sHow is it measured?According to the official technical report, IQuestRT's primary performance tests used 400 Claude Code runways and completed 800 decoding tests under two random seeds.
The test used sample parameters recommended by IQuest-Q1: Temperature=1, top-k=20, top-p=0.95.
With 8K high-performance GPUs, 8 path volumes in parallel, and single-request conditions, IQuestRT achieves decoding throughput of 910.70 tokens/s.
Further dismantling, the average speculative decoding output per round was 3.24 valid tokens, which took about 3.558 milliseconds per round, folding about 1,098 milliseconds per token output.
The vLLM reference configuration throughput recorded by the team was 309.83 tokens/s. Therefore, the decoding throughput of the IQuestRT in this group of tests is 2.94 times that of the IQuestRT.
It should be noted that the controlling parties used different speculative strategies: the vLLM reference configuration used five fixed candidates, while the IQuestRT dynamically selected three to seven candidates.
The technical report also provides a context-based one-step computation test.
In a standalone test of fixed-five candidates using synthetic KV caching, the mean single-step delay increased from 3.23 milliseconds to 3.76 milliseconds, an increase of about 16%, when the context length was increased from 1K to 192K.
This test differs from the previous 400 running trajectories in that it reflects single-step delay changes under specified computational conditions.
In addition, 910.70 tokens/s measure the performance of the Decode phase, which includes operations such as candidate verification, sampling, regression prediction and cache processing, excluding expenses such as Prefill, model loading and computational graph capture.
Therefore, it is still necessary to understand the difference between single-request decoding throughput and full-task, multi-request and concurrent service throughput.
However, for smart applications that require continuous generation of code, tool instructions, and content reasoning, shortening the waiting time of the single request generation phase still has direct technical value.
AI Agent participates in underlying research and developmentTwo weeks more than the man-made streamlineIQuestRT also has a notable feature: AI Agent is involved in its R&D process.
Optimization of high-performance GPU computing typically requires a great deal of fine engineering work.
Engineers need to continuously analyze the calculation bottlenecks, adjust the kernel implementation, data layout, synchronization, and task scheduling strategies, and test for effectiveness.
An optimization may lose its advantage even after gaining benefits from independent testing and entering a complete reasoning process due to additional scheduling or data processing costs.
To this end, the research team has developed a set of optimization processes involving engineers and AI Agents.
Engineers define workloads, numerical constraints, and acceptance standards, and decide which modifications will eventually fit the code. AI Agents participate in reading the code, proposing optimization assumptions, implementing programs, running experiments, debugging, and code review.
Each experiment left a corresponding version of the code, the test environment, and the original results, making it easier to contrast with subsequent iterations.
In the process, the actual running trajectory also helped the team discover problems that were easily concealed by synthetic testing.
For example, in a performance analysis, even if the length of the KV cache reading is the same, the actual single-loop decoding request is still about 0.36 milliseconds slower than the synthetic test.
The reason is mainly from expert visits to the MoE model.
The actual request averages 37.7 different experts per layer, while the synthetic input is only 11.0.
This discovery prompted the team to adjust the experimental approach: for optimizations that could affect expert weight access, it was necessary to go back to verify performance gains in the actual running track.
Automated experiments can also produce failed programs.
For example, the team attempted to further compress the weight of experts to FP4, hoping to reduce the cost of parameter reading. However, the test found that additional data unpackaging and computational scheduling costs offset the benefits of reduced weight storage, so it was ultimately not adopted.
Throughout the IQuestRT research and development process, 32 optimization checkpoints were recorded.
From the initial implementation written by the engineer, to subsequent improvements such as computational integration, communication optimization, dynamic speculative decoding and GPU autonomous cycling, the single-step delay in recording decreased from 6,449 milliseconds to 3,526 milliseconds, a cumulative decrease of about 45.3%.
This set of data shows the optimization process of the system as a whole, and contrasts with the previous vLLM reference configuration belong to different tests.
32 optimization checkpoints and one-step delay changes in the IQuestRT research and development process
In terms of R&D methods, the team, after more than a month of exploration, gradually established an automated optimization and verification process involving Agent.
Following the release of IQuest-Q1, multiple AI Agents simultaneously implemented single-request reasoning optimization on multiple GPU servers.
According to the technical report, the results of the optimization in just two weeks outperformed the performance of the team's previously artificially fine-tuned Tile Pipeline.
The results also show a viable way for AI Agents to participate in the development of complex underlying systems.
Engineers remain responsible for setting targets, correctness constraints, and final audits, while Agents can simultaneously conduct more code exploration and experimental verification.
When these efforts are combined into replicable tools and processes, the engineering costs of developing high-performance logic engines for specific models are also expected to be further reduced.
Specific reasoning enginesMoving to open sourceCurrently, IQuestRT is open source on GitHub, licensed under the Apache 2.0 license, and provides instructions for installation, operation, and performance testing.
But it still has a clear boundary of application.
According to the project's public documentation, IQuestRT is targeted at IQuest-Q1 and specific GPU computing architectures and interconnected environment designs, requiring 8 path parallel volumes and individual activity requests, while temporarily not supporting other models, GPU architectures and parallel configurations.
This means that 910.70 tokens/s and 2.94 times performance results should still be understood in the corresponding test environment.
From IQuestRT, it can be seen that as applications such as Coding Agent begin to take on increasingly complex, continuous tasks, logical optimization is getting deeper into specific model structures and actual workloads.
GPUs execute scheduling, speculate depth selection, MoE expert weight reading, and cross-card communication, all of which are underlying optimizations that affect the generation efficiency of a single request.
At the same time, AI Agent is involved in the practice of underlying performance optimization and provides new directions for research and development of high-performance computing software.
When performance optimization of dedicated logic engines is combined with automated R&D processes, efficient logic systems for different models may be built in the future, with more engineering pathways worth trying.
Technical report:
