Taobao’s MNN Team Open-Sources a 0.3B On-Device Model Running at 350 Tokens/s on iPhone 18 Pro

MNN team posted
QbitAI is public.

It's been a long time since I've explored how to run a large model on a mobile phone.

From 1B, 4B to 27B scale density models to MoE models with a total parameter of about 30B, activated parameters of about 3B, the possibilities for different sizes and architectures are expanding on the edge of deployment.

But for mobile app developers, successfully loading a model, completing a conversation, is just the beginning.

For everyday high-frequency applications such as Taobao and Shark Shark, large models must be accompanied by self-contained business-sharing computing power, memory and bandwidth. At the same time, continuous calls and multi-tasking must also be accompanied by response times, power consumption and temperature increases.

MNN teamThis design direction is calledApplications to mobile phones for native models- They released it.MNN-Pocket-0.3BThe task needs and resource budget for mobile applications, with limited computing, memory and energy consumption, in line with task capability, response speed and continuous running experience.

Introduction to the original model of mobile phones

Around a comprehensive range of mobile applications such as Taobao, the team implemented a four-dimensional design goal:

• At speed, the decoding throughput exceeds 200 tokens/s at the specified device and context configuration. The goal is to refer to the generation throughput of high-speed cloud model services, which seeks to provide a streamlined generation experience locally.
• In memory, under specified test conditions, peak memory is less than 300MB. Even if the phone has 816GB of memory, it needs to be shared by the system with multiple applications. For example, with a model module budget of about 500MB, the peak control is less than 300MB.
• In terms of power consumption, with a typical logic load (input 512-1024 tokens, output 128 tokens), a single logical power consumption target does not exceed one-tenth of the reference cell phone battery capacity, i.e. 100 logical power consumption does not exceed 1% of the total battery energy, controlling the duration of high-frequency calls.
• The ability to achieve a Level 8B model in a specific business scenario, to accommodate Chinese-English understanding, world knowledge and mathematics, and code reasoning, to focus on strengthening instruction, and to reserve space for extensive contextual and multi-step tasks.

These goals also need to be jointly tested in continuous calls: controlling power consumption and temperature rise while meeting task requirements, keeping the whole machine running smoothly, to integrate into daily use of applications.

MNN-Pocket-0.3B is the specific practice of this design direction.The MNN team designed and trained this approximately 300 million parameter, with a focus on supporting lightweight language models in Chinese and English, integrating model structures, vocabulary, training data, pre- and post-training methods, and collaborative optimization with quantification and reasoning engines to provide a task-understanding, responsive, adaptable and integrated model base for Taobao businesses and open source community developers.

0.3B parameters, how to achieve lightness and speed?

The model runs completely offline via the MNN logic engine, decoding at an average rate of about 350 tokens/s. The recording screen synchronizes input processing with peak memory occupancy, response efficiency and resource expenditure.

Deployment volume and running memory:The MNN-Pocket-0.3B's INT4 MNN deployment file size is only limited to173MBIn the iPad Pro M5, Metal rear end, 512 token test configuration, the model loads and the peak during the reasoning period. Metal residence memory increases relative to the pre-load baseline.228MBIn the same test, the deployment file size of Qwen3-0.6B was 340 MB and the peak memory was 427 MB. In comparison, the deployment size of MNN-Pocket-0.3B was reduced by about 0.3 MB.49%Peak memory growth decreases by about47%For mobile phones, smaller deployment volumes can reduce the cost of downloading and updating models, while lower operating memory allows more space for business modules such as pages, images, and video, reducing the resource burden of application integration.

Cross-platform reasoning speed:On Apple and Android devices, the team used Metal, OpenCL and CPU back-end to evaluate logical throughput. Testing using MNN 3.6.2 int4 quantized and unified operating settings, preheating once per set, measuring five times, taking throughput averages, and measuring Prefill and Decode respectively.8,721token/sThe decode throughput reaches.466token/sFor example, Qwen-3-0.6B is equivalent to 286 tokens/s.1.63 timesOn the back end of the CPU of the Snapdragon 8 Gen 5, the Decode throughput reaches251token/s。

Prefill affects input processing time and Decode affects subsequent generating speed. The above results are based on the report's quantification, context and measurement conditions, the actual performance changes with the equipment, backend and input output length, and the full configuration can be referenced in the technical report section 5.

These results come from three levels of collaboration:Parameters control the overall cost of scale and structure, vocabulary design reduces storage and output calculations, reasoning engines improve hardware execution efficiency.

Model structure determined from the resource budget of mobile applications

The team will be working on a native model for mobile applications.Peak memory of less than 300MB and decoding throughput of more than 200 tokens/sAs an architectural option, the goal was to keep the capacity of the model as much as possible within the resource budget. Around this goal, the team constructed a dense "Dense" model configuration covering about 50M to 1.3B parameters, which was tested in the iPad Pro M5 Unified Logic setting. About 300M was the largest candidate size to meet both constraints in the group experiment at the same time.

After determining the scale, the team further adjusted the model depth, hidden layer width, front-end network and attention configuration within a similar parametric budget. From 16 layers increased to 28 layers; continued to increase to 36 layers, with no further improvement in ratings, and decoding throughput decreased from 355 tokens/s to 323 tokens/s.

Ultimately, MNN-Pocket-0.3B uses a 28-layer structure, combined with configurations such as Group Query Attention (GQA), to strike a balance between model capacity, operating memory and reasoning efficiency.

Simplify the wording and reduce storage and output calculation costs

For the 0.3B model, word embedding and output layers account for a considerable portion of the parameters. Larger word tables typically help compress text lengths, and increase storage, visiting, and output projection costs, thus requiring a combination of target language and task design.

For Chinese-language mobile Internet applications, MNN-Pocket-0.3B focuses on supporting Chinese and English as languages, reducing the priority of pronunciation efficiency over other languages. Based on the Qwen3 dictionary, the team changed the vocabulary from roughly151K narrowed to 64K, reducing the number of words by about 57%.Reduced parameters and storage needs for word embedding and output layers, and reduced output projection costs when generating tokens.

In the cutting process, the team uses a content-filtering word order that covers Chinese English, code, mathematics, and instructional formats, while retaining a special Token-, byte-back Token and the necessary BPE integration dependence. By combining word tables size and coding efficiency, the model is able to retain the efficient coding that is commonly expressed, avoiding excessive cuts that cause sequence changes and offset deployment gains.

Models synchronize with engines, accelerating lateral reasoning

The end-side efficiency of the MNN-Pocket-0.3B comes from the integration of model design and reasoning engines. The MNN customizes the low-bit kernel on the CPU side for arm points and matrix instructions, optimizes caching access, calculates blocks and thread configurations on the GPU side according to different architectures, and provides a dedicated backend for the high-end Hexagon. At the same time, the MNN supports spectral decoding and decoding for selecting the appropriate execution paths, combining QKV, FFN algorithms with GPU command replication, reducing the storage and scheduling.8-draft configuration for a dedicated GPU coreOptimizing the parallel computation of multiple tokens.

The team evaluated MNN-Pocket-0.3B's reasoning performance on CPUs, GPUs, and NPUs, respectively, and compared it to llama.cpp. Testing MNN's performance in 4-bit quantification, the Prefill performance was 1.51 to 2.90 times that of llama.cpp, and the Decode performance was 1.43 to 1.82 times that of llama.cpp.

In terms of power consumption, the team also tested the power consumption of a 4-bit model of MNN-Pocket-0.3B using MNN on an extra 13 (Lonegro 8 PSV) version. At CPU 4 wires, input 1024 and output 128 baseline loads, the average increase in power consumption in three rounds of test discounting was approximately $4 billion, by collecting battery electricity and voltage, and subtracting the default baseline power consumption.2.7 J/wheelOn the basis of the phone's battery capacity of 6000 mAh, the cumulative incremental power consumption of the phone is about 0.3% of the total battery power of the phone.。

MNN-LLM has been accumulated through unified model conversion and reasoning interfaces, reducing the cost of cross-chip, cross-operating system integration.16K GitHub StarI'm not sure what I'm saying.200+ large modelsThe model has been downloaded over a million times, providing the engineering foundation for the actual application of MNN-Pocket-0.3B.

The deployment efficiency determines the cost of the model entering the application of the device. The next thing to verify is whether it can accurately understand the input under limited parameters and complete the task as required.

0.3B parameters and mission performance are equally excellent

The ability of mobile phones to apply native models needs to be reflected in the results that business processes can use. Comment extraction requires faithful reproduction of the original, label judgement needs to be limited within the candidate set, and structured output must comply with fields and formats. Content omissions, unrelated additions or class boundaries can affect subsequent processing.

Thus, MNN-Pocket-0.3B focuses on strengthening English understanding, mathematical reasoning and coding skills in construction.Instructions to follow with restricted outputProviding a basis for the adaptation of different business tasks.

The model's ability to demonstrate:In the rating setting used in technical reports, the English IFEval for MNN-Pocket-0.3B scores as follows:73.20The Chinese instructions built by the team follow the IFEval-zh score of74.59In 11 assessments covering English knowledge, reasoning, mathematics, code, and instruction, the model scored an average score of50.00The results of the study were published in the Journal of Chinese Mathematics (JCM) and published in the Journal of Chinese Mathematics (JCM) and in the Journal of Chinese Mathematics (JCM) in the Journal of Chinese Mathematics.51.20The code generates a HumanEval score of53.66These results provide a basic capability reference for application adaptation, where specific businesses still need to be validated through mission data and acceptance standards. Knowledge-intensive tasks and complex reasoning are still directions in which models need to continue to improve.

On a fixed parameter scale, the team passedMulti-stage pre-training builds a foundation, SFT iteration supplements shortcuts, multi-teacher distill integration, and retains deployment effectiveness through quantitative perceptual training。

Multi-stage pre-training: making data more suitable for small model learning

Small models are more sensitive to data quality and ratio.Areas, form of text, density of knowledge, difficulty and qualityThe five dimensions are marked and graded to create more detailed formulas for the different training stages.

Is the data worth studying?On the basis of cleaning and weighting, evaluate quality and knowledge density respectively: quality focuses on accuracy, consistency and completeness of content, knowledge density focuses on effective information content and redundancy.Q1–Q3、D1–D3Distinguishing the content of the filter from clean filter and filter information that is filled with filter, improving the accuracy of the filter and distribution ratio.

Is the data appropriate for school?Divide the difficulty intoH1–H3The third level, combined with the field and the text form, controls the distribution of samples of different complexities. At the same time, the structured organization of questions, tables and lists, the addition of concepts, reasoning steps or explanations in some mathematical and code samples, enhances learning.

When should data be learned?To combine the above dimensions into a data bucket, using candidate formula trials with actual mixed training, to co-adjust the sample proportions of each stage. For example, when improving the proportion of mathematical data, it also takes into account the quality, knowledge density and difficulty of the composition, so that the field assignment matches the current learning phase of the model.

On the basis of this classification system, MNN-Pocket-0.3B completes the4.1 Ttoken, four stages of pre-trainingBuilding a language and knowledge base, strengthening Chinese, mathematics and code, consolidating firefighting capabilities through high-quality data, and finally extending context support to32K。

SFT Loop Engine: Autonomous switching training around capability shortcuts

On the basis of pre-training, the team constructed approximately3,160,000 samplesThe SFT data format, which covers knowledge queries, mathematics, programming, Chinese tasks and open generation, translates basic capabilities into the ability to complete tasks on command.

In order to improve iterative efficiency, the team has developedSFT Loop EngineThis is a closed loop of training, evaluation and data improvement through Agent sorting:

Training→ Evaluation→ Failure attribution→ Allocation of data quotas→ Synthesis and screening→ Re-training→

The process will fail to respond to mapping to a sub-particle task category, combining the distribution of historical training data to determine the category, language, and sample number that needs to be added. The synthetic link does not directly read the original assessment question, and performs decryption and leak checks, then uses the filtered data for the next round of training.

This mechanism allows training to improve response to specific problems, while checking the benefits and other capabilities of newly added data through retrospective evaluation. This helps to maintain the balance of shortcuts and maintenance capabilities for mobile applications that need to cover multiple business tasks.

MOPD: Integrate multidisciplinary capabilities into the same model

Upon completion of the SFT, the team further addressed the problem of coordination of multidisciplinary capabilities under limited capacities. Mathematics, code, knowledge and instruction follow each of the optimization goals, mixed training requires the interaction between the goals to be addressed, and sequential training may also cause the already capable to retreat.

MNN-Pocket-0.3B is usedMany teachers on-policy distillation (MOPD)To build four pathways for teachers to optimize field reinforcement learning, select teachers based on sample fields, and co-supervise a student model.

Among them,The key to On-Policy was written by Mr. StudentTeachers read the actual context of the students and provide Token distribution supervision at these locations, so that the training covers the generating states that students themselves are easily exposed to. Teams adopt distribution matching targets to provide more intensive learning signals for small models and improve training stability.

After MOPD, the average score of the model in 11 assessments was35.79 increased to 50.00Ten of these improvements, one of which is consistent. Chinese mathematics, Chinese instructional follow-up and mathematical reasoning have achieved significant improvements.

Multi-faculty capacity guides during the training phase, retaining only one 0.3B student model when deployed, so capacity improvement does not require an increase in the storage and reasoning costs of end-to-end teacher models.

QAT: Leave the training effects to the deployment version

The low-precision models that are ultimately used in mobile applications will also require a capability test.Self-OPD and Quantitative Perception Training (QAT)W4A16 allows students to align the frozen BF16 model in their own contexts to mitigate the loss of ability to quantify.

In the same group of 11 assessments, the average score of the INT4 version reached49.59This is a decrease of 50.00 from the BF16 version.0.41 pointsThis enables the deployment efficiency shown in the previous chapter and the mission capabilities of this chapter to ultimately be delivered on the same version of INT4.

From basic training to low-precision deployments, the goal of MNN-Pocket-0.3B has always been to develop the ability to be used in applications.Then, through different stages of model opening and task verification, researchers and developers can continue to adapt, deploy and test.

The 0.3B model is open multi-phase weighting, moving from task mitigation to real-world applications.

The value of a mobile application's native model needs to be verified through task adaptation and application integration. For this, MNN-Pocket-0.3B offers multiple versions of deployments from pre-training and post-training to lateral deployments, allowing researchers and developers to choose the right starting point and continue training, evaluating and building applications.

From research to deployment, the MNN team has opened up model inputs at different stages:

For students and researchers,Base and SFT checkpointsThe 0.3B parameter scale helps to reduce the cost of one-off mitigation and evaluation, and supports more efficient multi-cycle experiments.

For application developers, the final instruction model can be used for business adaptation.MNN versionIn addition, the company's mission is to provide ecosystem-friendly deployment assets.Published Dec.Classification and candidate selection tasks, completing the scoring of candidates in a one-time forward calculation by the decision-maker;DFlash 2 draft modelIn order to accelerate the generation, eight candidate tokens were proposed at a time, which was unified by the target model, reducing the serial costs generated by each token.

Around the business adaptation and interaction experience, the team further selected e-commerce commentary point of view exploration and two end-to-end voice dialogue links to test the performance of the model in specific tasks and complete applications.

E-commerce commenting capability of the 0.3B model

The shading functionality is good, and the shading functionality is poor. The shading functionality includes a brief commentary on the overall positive evaluation and a negative evaluation of the specific functionality. The model needs to recognize the interaction between the two and distinguish between explicit expression and implicit information.It's all about attributes, opinions, categories, emotional polarity.It consists of four fields. The attributes and opinions must come from the original, and the categories and emotions must be restricted to a predefined set of fields that are not explicitly expressed. More than one point of view in a comment also needs to be fully extracted.

The team used the Sky Pool mission to provide3,229 commentsIn the experiment, the F1 of the MNN-Pocket-0.3B reached the F1 of the MNN-Pocket-0.3B.80.15In the same setting, Qwen3-0.6B ((77.47) and Qwen3-1.7B ((78.59)) are higher than the parametric scale of less than one-fifth of the latter.1.56 pointsThe results of the missions.

This result validated the modeling ability of the model in the task: through field data and LoRA training, the 0.3B model is able to convert general understanding and instruction-following capabilities into structured outputs that meet business requirements and provide a light path for text analysis tasks in mobile applications.

Full-Side Voice Dialogue: Putting speed and capability into full interaction

Voice interaction further tested the model's responsiveness and ability to collaborate across the entire application chain.Full side stream voice systemAll of the neural network modules run on the device locally via MNN. The system adopts streamlined processing to shorten the wait time: After the speech is converted to text, MNN-Pocket begins to generate responses, text increment segments are sent to TTS, and then play out step by step, without waiting for a complete response. When the user generates again, the system stops the current generation, synthesizes and plays and enters a new round of interaction.

In the reporting heater test conditions, the system tested the first audio delay P50 as:

• iPad Pro M5：600ms；
• Snapdragon 8 Gen 5 devices: 935ms;
• iPhone 13 Pro：1,055ms。

In a unified ASR front-end and assessment setting, the system with MNN-Pocket achieved total scores on nine VoiceBench tasks.45.44For the highest score in the language baseline compared to the report, the model demonstrates comprehensive task capability under voice input conditions.

This link validates the feasibility of MNN-Pocket as the language processing base for end-to-end applications built in conjunction with voice perception, synthesis, and interaction control. The complete system contains additional modules such as ASR, TTS, whose overall storage and memory needs need to be assessed separately in conjunction with specific applications.

Let real applications continue to define the model

The value of mobile phones applying native models will ultimately be reflected in real business, real devices and everyday interactions.

MNN-Pocket-0.3B revolves around the needs of applications, co-designing model structures, data with training methods, quantification and reasoning engines, combining task capability, response speed and deployment costs on a 0.3B parametric scale to provide a trainable, adaptable, and integrated model base for application development.

Next, the team will continue to advance end-to-end model research and development, understand the needs of businesses such as Taobao, search for recommendations and intelligent interaction, improve model reliability, coverage and end-to-end efficiency, and explore applications on devices such as AI glasses.

By opening up models and deployment versions at different stages, the team looks forward to working with researchers and developers to validate and improve models in more specific tasks, allowing intelligence to be naturally integrated into everyday applications.

GitHub:
Hugging Face:
ModelScope:
Technical Report:

Please leave your thoughts in the comments section!

— It's over. —

Please send an email on weekdays to:ai@qbitai.comThe headline points to a postcard and tells us:Who are you?，Where did it come from?，Content of the postAddedProjects/Home page linksandContacts。