In the process of training Tool-Calling Agents, we have repeatedly encountered a seemingly simple but crucial question: Where does the training data come from?
Tool-Calling Agents need to call an external API based on user intent, integrate intermediate results, and ultimately provide answers. To train an Agent like this through Reinforcement Learning, we need a large number of tasks, each with the correct intermediate tool calls and verifiable final answers. Even more data-efficient RL programs often require thousands of such tasks; whereas the current mainstream agent benchmark typically provides tens to hundreds of tasks each.
It's a natural idea to make an LLM to synthesize tasks. But soon we're faced with the following question: Is a task imagined by an LLM, its ground truth true? Furthermore, if these tasks rely on real-world APIs, and the results of the APIs can change at any time, how can we repeat the same environment in training?
We gradually realized that the key to the problem might not be how to generate a better task string, but rather the order of the task itself.
In other words, we think: instead of asking questions, then looking for answers, instead of exploring the real API and then reversing the questions based on the results that have been observed.
In our NeurIPS 2026 work Firefly: Illuminating Large-Scale Verified Tool-Call Data Generation from Real APIs, we followed this idea by building a pipeline of verifiable tool-call data from real MCP servers and an offline replicable Retrieval-Augmented Tool-Call Simulator. Based on this pipeline, we obtained 5,144 verifiable tasks covering 240 server 993 tools.
Address of the paper:https://arxiv.org/abs/2605.17558
Code and data:https://github.com/leoleoasd/firefly-neurips-2026
Meeting time: Thu, Dec 10, 2026-10:00 AM EST in Hall C1−Atlanta Poster Session 3
Scale, authenticity and verifiability: the threefold dilemma of existing training data
Currently, the method of constructing Tool-Calling training data can be roughly divided into three categories: artificial labeling, synthesis in a purely simulated environment, and forward-to-back synthesis based on real tools.
Figure 1: It is scalable, real, verifiable, and there are three types of methods that are lacking.
1.1 Artificial marking: It's reliable, but it's not scalable
Artificial tagging ensures the nature of user intentions and the correctness of responses, and works such as the Tau-Bench series, API-Bank, and others have adopted this approach. However, the cost of artificial tagging is high, the domain is highly dependent, and it is difficult to cover the long tail of the real API. In the Tau2-Bench example, there are only 50 to 115 tasks per domain; MCPMark provides 127 tasks, and MCP-Atlas publishes 500 tasks. These numbers are suitable for evaluation, but not enough to support RL training.
The cost of constructing verifiable AI tasks is also low. According to industry interviews with Epoch AI, an AI lab typically costs $200 to $2,000 to purchase an RL mission, and exclusive licensing costs 4 to 5 times as much. Such prices and exclusive terms make such data mainly concentrated in a small number of large manufacturers.
1.2 Pure simulation environments: scalable enough, but away from the real API.
In order to break through the scale constraints, a class of work allows the LLM to simultaneously play the role of a mouse user and a mouse tool: the return of the tool is also generated by the LLM. Such environments are easy to expand, but the output of the tool created by the LLM often differs from the behavior of real APIs. In the real world, field formats, boundary situations, feedback and even the data itself are difficult to accurately simulate. Agent trained in such an environment may learn how mouse interacts with the simulator rather than how mouse interacts with the real world.
1.3 Moving towards synthesis: real enough but lacking verifiable ground truth
The other type of work uses real tools, using a process we call the Forward Generation: first, the description of all tools is given to the LLM, by which it writes a user task that needs to call these tools; then a tool-call trajectory is rolled out by another solver model for that task, and the solver's final answer is considered the ground truth of the task. Consider this simple example:
Which domain was registered first, amazon.com or netflix.com, and how many years apart were they registered?
In the Forward Generation, the problem is first written out, with the LLM not knowing what the tool will actually return. This leads to two failures. First, the task itself may not be completed: the final state it requires cannot be reached at all on the real tool. Second, the solver may make a mistake, such as checking the wrong domain name or calculating the wrong year, and its answer will be treated as ground truth. In other words, the ground truth of such a task has no mechanism to guarantee that it is true.
Figure 2: The process of the Forward Generation and its two failures.
1.4 Environmental Drift: It's hard to train even with verifiable tasks.
Furthermore, even if we already have real-world tools and verifiable tasks, it is still very difficult to do RL training directly on the live API. Many MCP tools rely on external data sources such as search indexes, weather, financial data, DNS records, and their output changes over time; servers may go offline; multiple repeat calls in RL rollouts may trigger a rate limit. As a result, the same task may be returned by different tools today and next week, and training and evaluation are difficult to replicate.
Figure 3: Three questions to do RL directly on a live server.
Therefore, building scalable, real, verifiable training tasks and supporting an environment that maintains reproducibility in the RL process is still an open question.
02 Reverse synthesis process: making the correctness of ground truth a constructive characteristic
Our core observation is that it is difficult for the Forward Generation to guarantee ground truth accuracy, largely because the puzzle is solved before the puzzle occurs. If the order is reversed, the problem may become much simpler.
Back-chaining: explore first, ask questions again
Going back to the domain name example above. In Firefly, the task was not written out first. On the contrary, a robust LLM first explored the real MCP Server, which invoked the whois query tool in networkcalc-mcp, queryed amazon.com and netflix.com respectively, and actually observed their registration years: 1994 and 1997.
We call this process Back-Chaining. Since every ground truth comes from a state that the real tool actually returned to, its correctness is no longer dependent on subsequent screening or verification, but is guaranteed by the way the data is generated itself.
In other words, the correctness of the ground truth has evolved from a filtered result to a structural characteristic of the hydrogen data generation process.
Figure 4: Start by solving a problem and then go back to exploring it and then solving it.
2.2 From the tool space to the real MCP ecosystem
It's worth pointing out that this is not the first time we've come up with the idea of pulling back-to-track missions. Trajectory2Task applied a similar idea to smaller or fixed tool spaces, where possible toolchains can be enumerated or manually constrained.
This isn't a simple engineering problem. When the number of tools increases to nearly a thousand, distributed over hundreds of heterogeneous servers, there are challenges that don't exist in at least two tool spaces:
First, the scalability of exploration. The number of tool pairs grows rapidly with the number of tools, and aimless exploration will waste most of the call on a semantically unrelated tool pairing.
Second, the reproducibility of the training environment. Back-Chaining ensures that the ground truth is correct when generated, but if the return of the API has changed during training, this ground truth may no longer be consistent with the environment.
Back-Chaining, of course, does not mean that AI labeling is no longer valuable. AI labeling benchmarks are still irreplaceable in terms of the nature of the task and the reliability of the assessment, and in fact, we are also testing the migration effects of Firefly on these benchmarks.
Firefly: From real MCP Server to offline RL training
When designing Firefly, we wanted to answer three questions at the same time:
How to select tools to build verifiable data from an open MCP ecosystem?
How to effectively explore and get semantically coherent multi-tool trajectories across nearly a thousand tools?
How can training and evaluation get rid of the reliance on the live API while maintaining the authenticity of the tool returns?
Around these three issues, Firefly consists of three phases: MCP Server Collection, Task Synthesis, and Reinforcement Learning.
Figure 5: The overall framework of Firefly.
3.1 Server Collection: From 3,000 servers to 240.
We collected MCP Server from the Smithery registry, combining extensive list queries, front-end listings, and domain keyword searches. After going back, we got about 3,000 servers, of which about 900 exposed at least one tool.
But not all servers are suitable for constructing verifiable data. We use LLM to screen for four criteria: Statelessness (no cross-call session status) No User Authentication (no need for OAuth or user's own credentials) Schema Clarity (parameter type and description clarity) and Non-Triviality (not echo or pure diagnostic tools). Only servers that meet four criteria will be retained at the same time.
3.2 Tool Graph: Provides a weak semantic precedent for exploration
In the face of nearly a thousand tools, an intuitive scheme is to judge which tools can be synchronized based on the type matching of the schema. But the type matching is too rough: the two string fields are type-compatible and may be semantically irrelevant.
So we built a directional Tool Graph on all the tools. Each node is a tool, and each side represents the output of the source tool that can be converted into or used directly as the input of the target tool. We let the LLM judge this compatibility on a case-by-case basis based on the tool description and schema, marking the confidentiality of high, medium or low for each side.
We believe that the role of the tool graph is not to dictate what task graphs should be generated, but rather to provide a weak semantic precedent: it keeps the final task open while directing exploration toward neighboring regions that are more likely to produce meaningful, combinable trajectories.
3.3 DAG Exploration: Graphing growth tools on real servers
Each exploration begins with a tool with at least two high-confidence sequences. A strong LLM then interactively builds a tool-call DAG on a live server: each round, it obtains a random subset from the finished node sequence in the chart, and can choose to simultaneously call multiple tools (fan-out), sequence a sequential, or merge the output of multiple previous nodes into a single call (fan-in).
Each node in the DAG records the tool name, input parameters, and actual output. Since they all come from the actual execution of the live server, all states are guaranteed to be accessible. These DAGs are both the raw material for the subsequent task synthesis and the source of the next Simulator's index.
Figure 6: A search on Tool Graph, fan-out and fan-in were performed on real servers.
3.4 Back-Chaining Task Synthesis and Task Validation
For each DAG, LLM selects a part of the node and builds a natural language task around it. The task description contains the information required to fill in the tool-call parameters, while ground truth is extracted directly from the output of the selected node; the rest of the node naturally becomes a disruptive object or unrelated search path. By selecting different node subset and changing the expression, the same DAG can produce multiple tasks of different complexity.
Each task is also equipped with a structured Answer Schema (a JSON template with a named placeholder) and a corresponding natural language Answer Template. Thus, evaluation can be done automatically by field-level comparisons without having to rely on free text matching.
To ensure the quality of the task, we used LLM Judge to validate each task-trajectory pair from five dimensions: Verifiability (the answer can be objectively checked) Well-Specifiedness (there is only one correct answer) Interpretability (the output is human readable rather than opaque ID) Realism (the authenticity score is not less than 5/10) and Difficulty Calibration (the marking difficulty is consistent with the trajectory complexity).
3.5 Retrieval-Augmented Tool-Call Simulator: Get RL out of the live API
To address Environment Drift, we cached every tool call observed during the exploration phase, including tool names, input parameters, and return results, and built a Retrieval-Augmented Tool-Call Simulator based on these caches to completely replace the live server in training and evaluation.
Exact Match: Hash searches for input parameters, and the hit returns the actual output of the cache directly.
Fuzzy Match with LLM Generation: retrieve the most similar top-k historical call inputs, prioritize the ground-truth trajectory of the current task, and then use these real outputs to condition, select, or generate a response by LLM.
No Data: If the tool does not have any historical calls, it returns an error.
Figure 7: The three-tier strategy of the simulator. Fuzzy Match mainly deals with parameters that are essentially the same, with only different calls for details such as format.
The design of the third layer is intentionally conservative. Since each task is synthesized from a known path, calls away from the cached path are unlikely to be required for correct solutions and therefore should not be rewarded.
Figure 8: The core assumption is that the path to ground truth has been cached.
Unlike a purely simulated environment, each response in this Simulator is rooted in a real API output. It brings with it two important elements: Reproducibility, which returns a completely consistent tool whenever the same task is run again; and Verifiability, which has indexed the responses needed to derive each ground truth and is always available through Exact Match.
In RL training, Agents always interact with the Simulator. The rewards are two-fold: first, try to match the field level accurately with the Answer Schema, and then, if fail, by LLM Judge to judge if the semantics are equivalent. Since the same Simulator also serves as an evaluation environment, training and evaluation uses exactly the same tool behaviour and can be rolled out on a large scale without any API cost and availability concerns.
Firefly data sets: verifiable, multi-step, based on real API
For a data generation framework, simply describing the methodology is not enough. We applied the entire pipeline to the real MCP ecosystem, built the Firefly dataset, and opened it with the Simulator as a resource that can be used directly for offline RL training and evaluation.
From the Smithery registry, about 3,000 servers were released, and 240 servers and 993 tools were retained after screening. Toolchain exploration generated 9,749 DAG tracks, from which Back-Chaining synthesized 9,275 candidate missions and 5,144 missions passed full quality standards. We retained 200 missions as a fixed test set and the remaining 4,944 for training. Each mission was equipped with ground-truth tracks, an Answer Schema, and all the tool-call responses required for offline playback.
What's that?
The task covers three difficulty levels: easy ((19.5%) ∆ medium ((77.0%) and hard ((3.5%) ∆ Ground-truth trajectories require an average of 3.0 tool calls ((range 1 to 10), and the Answer Schema contains an average of 4.6 fields, which means that most tasks need to integrate multi-faceted information rather than single-value queries.
It is worth noting that about 38.5% of tasks require multiple server combination tools. Most already have benchmarks to limit tasks within a single API provider, and cross-server combination is precisely a common occurrence in the real MCP ecosystem. The test set covers 98 different servers and 221 different tools alone. LLM Judge's average accuracy rating is 7.8/10.
4.2 Simulator coverage
A natural concern is that during the RL process, the Agent may issue a large number of calls that have never been seen before in the exploration phase, thus falling frequently into the No Data layer. During the entire training process, 42.2% of Agent tool calls were resolved through Exact Match, and 57.8% through Fuzzy Match, falling into the No Data layer at 0%.[3] This suggests that the exploration phase provides sufficient coverage for offline training.
4.3 Cost of Production
The entire data set was produced using Claude Sonnet 4.5 through AWS Bedrock batch inference, consuming about 23.5B tokens (input 21.4B, output 2.0B) at a total cost of about $47K. This was a one-time input: the generated data set and the Simulator could be used repeatedly, and support for offline RL training would no longer require an API fee.
It's the same as the data set.
In our comparison data sets, Firefly also provides multi-step trajectory, verifiable ground truth, and is based on real rather than simulated API servers. Of course, we also note that the design goals for different datasets vary, such as Toucan, which is far larger in scale than Firefly.
The results of the experiment: small models, large upgrades and cross-distributed migration.
To verify the effectiveness of Firefly, we started with Qwen3-4B-Thinking-2507 and used GRPO and Dynamic Filtering for RL training on 4,944 training tasks. Each prompt sampled 8 rollouts; when the model matched 8 rollouts on a given prompt, the prompt was permanently moved out of the training set, while the entire error prompt was retained. This similar curriculum mechanism makes it difficult to train the gradually focused model.
5.1 Firefly Test Set 4B is the equivalent of Claude Sonnet 4.6.
On 200 held-out missions, Firefly RL increased the pass@1 of Qwen3-4B from 28.1% to 41.5% ((+13.4 points), essentially equivalent to Claude Sonnet 4.6 ((42.2%) under the same evaluation protocol.
Figure 9: Pass@1 on the Firefly test kit, the same assessment protocol.
Figure 10: The pass@k curve on the Firefly test set during training, the null-line is a three-dimensional Claude model.
At the same time, we also note that the gap between pass@16 ((57.0%) and pass@1 ((41.5%) is not large, which suggests that some of the more difficult tasks may be beyond the scope of the 4B model's capabilities, even given multiple attempts.
5.2 Open Benchmark: It's not just remembering the caching path.
It's one thing to improve on your own test set, and it's another to migrate to different public benchmarks. Firefly is a single-cycle, multi-step dataset: each task has only one user message, but requires Agent to plan and execute multiple tool calls. Public benchmarks that we evaluate differ significantly from Firefly in both tool source and environmental design, with Tau2-Bench requiring multiple users to interact with Agent.
The experimental results showed that the Firefly RL improved on all public benchmarks assessed:
Enhancements vary by task, such as in Tau2-Bench Telecom. Enhancements are relatively limited. However, consistent upstream changes suggest that the model doesn't just remember caching trajectories in Firefly. We believe that training on verifiable single-step multi-step tasks may enhance some portable tool-use behaviors: selecting relevant tools, filling in parameters based on observations, stringing out intermediate outputs, and deriving final answers from structured tool responses.
The most interesting of these is the improvements to Tau2-Bench. Although Firefly only includes single-wheel tasks and Tau2-Bench requires interaction across multiple user rounds, many of the latter's core challenges, such as tool selection, status tracking, API grounding and multi-step reasoning, remain highly relevant to Firefly's ability to train. This suggests that single-wheel multi-step monitoring signals may also benefit multiple-step scenarios to some extent.
From verifiable data to verifiable agents.
Here, we've introduced the core ideas, pipelines, and experimental results of Firefly. But for us, the more interesting part of this work may not be limited to current numbers. What we really care about is: Where else can Agent's training extend when the correctness of ground truth can be guaranteed by data generation?
6.1 From single-cycle to multi-cycle: Let Back-Chaining cover real conversations
Currently, all tasks in Firefly are in a single-cycle format: the user makes a request, and the agent completes it through a multi-step tool call. However, in real-life scenarios, users often add explanations, modify requirements, or continue to ask questions based on previous results.
One question worth exploring is: Can back-chaining naturally extend to multi-round conversations? For example, can we split a DAG into multiple phases to allow each phase to respond to a round of user requests while ensuring that the answers to each round are still verifiable? and how to introduce clarification, correction, etc. in the synthesis process to a pattern of interaction closer to real user behavior? We think this is the most natural next step in Firefly.
6.2 From 4B to the bigger model: Where is the limit for verifiable supervision?
Due to computational resource constraints, we are currently only training and evaluating the 4B model. It is mentioned earlier that the gap between pass@1 and pass@16 is not large, suggesting that the 4B model may have approached its capability limit in more difficult multi-step tasks.
We speculate that larger models (e.g. 30B and above) may benefit more from verifiable supervision by Firefly, while increasing the upper limit of pass@1 and coverage of pass@16. But it should be emphasized that this is still a hypothesis and requires further experimental verification. From a longer perspective, we also want to know: how should the size and scale of verifiable data be aligned with the size of the model?
6.3 Verified by Construction: A more generic data model?
We think that this is probably the direction that Firefly should continue to think about. The core of Firefly is not a specific module, but a more general idea: first produce observable results in real environments, then build tasks around those results, and cache the environment itself. Thus, the accuracy of data and the reproducibility of the environment are no longer two separate issues.
For example, in order to ensure verifiability, we exclude servers that are in state and require user authentication at the screening stage, whereas in the real world there are a large number of important tools that fall into these two categories, how to achieve Back-Chaining and reproducible playback in a state environment?
Therefore, we do not believe that Firefly has solved the verifiability issues of Agent Training data. Rather, we hope that it will serve as a starting point for further research into these issues.
######################################################################################################################################################################################################################################################################################################################################################################################################
As Tool-Calling Agents move from research prototypes to real applications, how to build large-scale, authentic and verifiable training data for them is becoming an increasingly important question. In this work, we try to start from a relatively basic question: in which order should the puzzle and the observation puzzle occur when generating training tasks?
We believe that man-made labeling is reliable but difficult to scale; a purely simulated environment that is scalable but deviates from reality; a synthetic environment that is real but lacks verifiable ground truth; and reversing the sequence, first exploring the real API, then reversing the observed results to synthetic tasks, may be a natural way to strike a balance between the three. Based on this thought, we proposed Firefly: Using Tool Graph to extend Back-Chaining to nearly a thousand real tools, Retrieval-Augmented Tool-Call Simulator to enable RL training and evaluation to be fully offline and reproducible.
As Agent takes on more and more tasks in the real world, the research focus on training data also has the opportunity to move from generating more data sets to generating trusted data sets. We hope that Firefly will not only be a data set, but also be a starting point for the research community to explore verifiable Agent learning.
Introduction by the first author
Prof. Dakuo Wang, Ph.D. student at Northeastern University in the United States. His research focuses on LLM Agent, HCI/UX systems and human behaviour simulation, Tool-Calling Agent training data and RL infrastructure, and he has published several papers at international conferences such as NeurIPS, ACL, ICLR, EMNLP.
