分享好友 Global News首页 Global News分类 切换频道

It speaks 24 languages, can "pick" any tone, and can be changed by hand: the Little Red Book FireRed

2026-08-21 07:0670RoboInd Editorial TeamWeChat Official Account

The Red Book FireRed team has released a new generation of voice generation and editing model FireRedTTS3. It uses the same model to unlock the three major capabilities of voice in multilingualism, multilingual zero-sample sound cloning, customizable voice design for "description" in natural language, and precise voice editing for "where not to change where not to change".

Give it a few seconds of your voice, and it can say "you" in 24 languages, 21 Chinese dialects; give it a word of description, and it can "pick" out a voice that never exists; give it a built-in tone, and it moves just where you turn, like a document, and the rest is unchanged.

The secret lies in a speech expression called RedAE: it asks a "semantic teacher" to put the semantic "teaching" into the voice expression at the source. So without stacks of modules, without stages of training, without overlapping complex structures, it steadily suppresses the old headache that generates the most headaches of continuous speech, "the more you say, the more you run away".

The score was also tough enough: the accuracy rate and similarity of sound cloning topped the list (Seed-TTS-Eval 3.04% / 78.8%), the cloning of 24 languages topped the list (3.75% / 84.8%), and sound design and voice editing topped both lists.

First in the ear: Can you tell the difference between the two?

Following FireRedTTS-2's long conversations with podcasts generated to an industrial level, FireRedTTS3 shifted its ability from "speak a word" to "clone, design, edit hand in hand".

Listen to how friends from all over the world talk about the World Cup. During the heatwave of the World Cup, different languages, different chords are playing; but you might not think that these sounds actually come from the same set of models.

Next, we're going to listen to netizens from different regions ask teachers in a kind language. Do you think their accents are not true?

If the spelling "speak all kinds of languages" is width, then the spelling below is "no middle life". This time, we don't provide the model with reference audio, only pass the word "description of needs".

Is it possible to "operate" on an already recorded sound with an aerial sound? The last one is a bit like magic: in a finished word, we just want to change one of the words, or adjust the speed of the voice a little bit, and no word elsewhere is allowed to move.

So the question arises: how does a model do all three things well at the same time?

To clarify this, we have to go back to a more fundamental question of why we need such a model.

Over the past two years, TTS has become quite mature in zero-sample audio cloning. But the demands of users who are rapidly upgrading the sound are no longer satisfied with "repeat a sound", but hopefully:

In a nutshell, demand is moving from "clone" to "control".

If you want the machine to "listen", where's the card?

In order to be "controlled", the model must pass through three stages: understanding natural language instructions → placing the instructions accurately on a certain speech attribute or segment → without breaking the unspecified part. The basis of all this is that the speech indicates that two types of information must be encoded at the same time: semantic (content/style/intention) and acoustic details (tone/pitch/pitch)

There are currently three technical routes, each with a shortcut on "singular + acoustic + controllable":

Continuous regression is currently one of the most promising paths, but "accumulation of errors" is the only way around it. The previous academic community mainly remedied it from two directions: one to continuous representation of semantics (such as Ming-UniAudio additional module abstraction semantics; VibeVoice uses separate semantics VAE; dots.tts add ASR targets to VAE training); the second is to correct feature spaces (such as VoxCPM introducing FSQ bottlenecks) during the regression phase. However, the direction often requires additional semantic modules, or multi-stage tokenizer training directions; the latter will complicate the architecture.

So FireRed TTS3 changed its mind and asked if it was possible to mitigate the accumulation of errors in the "representation" layer while the system was still simple.

The first step: let "express" itself with the semantics of RedAE

The first cornerstone of FireRedTTS3 is the continuous voice tokenizer RedAE. Its core idea can be summarized in one sentence: not to add additional semantic modules, not to do multi-stage training, but to ask a "semantic teacher" to "teach" the semantics in continuous representation during the tokenizer training.

图片

Figure 1 shows the structure of the RedAE

Where did the semantics teacher come from? The team first trained an audio understanding model, FireRedAudio, to learn how to use its Audio Encoder to understand ASR, speaker verification and other voice-comprehension tasks, and to naturally understand "content semantics" and "speaker acoustics cues".

When training RedAE, freeze this Audio Encoder as a semantic teacher, so that the output of RedAE continuously represents the attributes of the teacher that are aligned (minimizing the MSE of both, i.e. loss of semantic distillation).

Why accuracy is not compromised. RedAE deliberately omits the common KL rule to avoid overcompressing acoustic information, thus leaving the "easy to quantify erased" details in the tone intact. This is the key to its lead in sound similarity.

Structure and training details. The input 24 kHz wavelength is cut into 480 non-overlapping sample points, obtained in a 50 Hz sequence. The Qwen3 style Transformer is handled through two levels: the first modeling the context at 50 Hz, and the second sampling to 25 Hz through attention-based polarization. Each pair synthesizes a patch.

Overall, training under the GAN framework, generator losses include resistance, multi-dimensional Mel rebuilding, feature matching, four semantic distilling. The RedAE's Encoder and Decoder are combined in a single phase, with no additional semantic branches, no multi-phase flow lines; training with 500,000 hours of diverse audio (500,000 hours of clean voice 50%/band noise 25%/sound efficiency 10%/music 15%), running 550,000 steps on 32 pages of H800).

Step Two: A light frame, a vacuum, three things

The next step is to create a generating framework that uses it. On top of the RedAE, FireRedTTS3 uses a lightweight LLM-DiT framework that consists of three components:

图片

Image 2 LLM-DiT structure, left to FireRed-Base and right to FireRed-Instruct

In the same set of frameworks, the team separated two versions of "Expert" and "Omnipotent":

FireRedTTS3-Base is aimed at multilingual, multilingual cloning.

FireRedTTS3-Instruct (Unified Cloning / Design / Editing)

It's a monkey, it's a horse, it's a horse, it's a horse, it's a horse, it's a horse, it's a horse, it's a horse.

It's nice to talk about it, but it's also worth talking about the results. We pulled the FireRed TTS3 into four open ratings sets, with some of the strongest models in the industry in front of us.

On the Seed-TTS-Eval of the English zero-sample sound clone in the evaluation, FireRedTTS3-Base achieved the lowest average error rate and the highest average speaker resemblance with Test-EN and Test-ZH. This confirms that "singularised continuous representations" can provide a stable modeling goal for LLM-DiT, resulting in a more robust text-to-speech alignment; and it has clear advantages in tone resemblance, especially as it avoids the quantized loss of audio information.

图片

Figure 3 shows the results of zero-sample cloning of Seed-TTS-Eval.

A model that speaks fluent Chinese is not difficult, with the difficulty of being "common" in dozens of languages. When the range was extended to 24 languages, MiniMax-MLS-Test, FireRedTTS3-Base still got the lowest average error rate of all languages (3.75%) with the highest average similarity (84.8%), and placed first or second in 22/24 language similarities.

Particularly noteworthy is the fact that Portuguese and Ukrainian are not included in the training data of RedAE, and the model still gives competitive results, reflecting RedAE's ability to generalize unseen languages.

图片

Figure 4 of the MiniMax-MLS-Test multilingual test results

In Instruct TTSEval, which examines sound design capabilities, FireRed TTS3-Instruct scored best on all three categories of instructional acoustics parameters: APS, style description, DSD, role-playing, and RP.

This is due to its design "to first analyze instructions into 12 dimensional structured text programs, then synthesize them", effectively reducing the ambiguity of natural language instructions and achieving a solution to the instructions.

Specifically, for APS tasks, it derives explicit properties from complex acoustic parameterization instructions; for DSD and RP tasks, it converts abstract style descriptions or character images into specific acoustic programs, thus improving compliance with user instructions.

In addition to deciphering instructions, FireRedTTS3-Instruct also learns the steady mapping between these well-planned sequences of acoustic properties and RedAE voice representations to effectively control target tone synthesis.

图片

Instructions for controlling the results of sound design by TTSEval

Lastly, on the Ming-Freeform-Audio-Edit list of voice editors, whether it's audio editing (speed, pitch, volume), or semantic editing (insert, remove, replace), FireRedTTS3-Instruct is mostly at the forefront and performs equally well in the open setting of the free instruction.

From the core rating indicators: lower WER (word error rate) with NoEdit (word error rate) with unedited zone WER (word error rate), combined with extremely high ACC (editing accuracy rate), together to ensure the accuracy of content modification; significantly superior SIM (speaker similarity) ensures sound stability; and extremely low RDE/RAE (acoustic error rate) proves that it can accurately modify to the target volume and time.

图片

Figure 6 Mining-Freeform-Audio-Edit Acoustic editing results

图片

Figure 7 Mining-Freeform-Audio-Edit Semantically edited results

In the end, let the voice be as free as the text.

The first of the four lists is actually a plain judgement. The key to FireRedTTS3 is to bring the "suppression of error accumulation" down to the representation layer: using a frozen semantic teacher, injecting the semantic into the tokenizer stage with a continuous representation of RedAE, making stability an inherent attribute of representation.

As a result, sound cloning, sound design, and voice editing were for the first time integrated into the same simple set of LLM-DiT frameworks: tokenizer single-stage training, no additional semantic modules, no complex structures, but leading the way in four public assessment sets simultaneously.

From FireRedTTS, FireRedTTS-1S, FireRedTTS-2 to today's FireRedTTS3, the Little Red Book FireRed team has always been doing the same thing: building a growing voice generation infrastructure.

When both "generation" and "modification" are completed in a single sentence, the barrier to sound creation will be greatly flattened so that ordinary people who don't understand the acoustic parameters can also edit speech like text. In the future, the team will continue to expand the language and dialects, the boundaries of instructional control and editing, so that sound can truly be written, modified, and creatively expressed.

This is just the beginning of a path that "represents" as a starting point.

点赞 0
举报
收藏 0
分享 2