Controllable and Unified Modeling for Speech Generation
Zhizheng Wu
The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen)
Keynote speech at ISCSLP 2026
Acknowledgement to My Co-Authors in This Talk
2
In the order their work appears in the talk: Haorui He, Chaoren Wang, Jiaqi Li, Yuancheng Wang, Xueyao Zhang, Dekun Chen, Huan Liao, Qinke Ni, Yunjia Zhang, Jun’an Zhang, Yingda Shen
Speech Generation Today: Near-Human Naturalness
3
1
Zero-shot voice cloning
MaskGCT · zero-shot TTS
2
Instruction-based style control
FlexiVoice · instruction + voice prompt
3
Humming-to-singing synthesis
Vevo2 · melody + lyrics + voice
Speech prompt
a few seconds of a voice
Same voice, another language
Chinese text
maskgct.github.io
Instruction
“Speak this sentence using happy emotion.”
Voice prompt
spoken sadly
Output
same voice, now happy
flexi-voice.github.io
Melody: humming
Beethoven, Für Elise
Output: singing
that melody, a reference voice
versasinger.github.io
Naturalness is near human. Two questions remain: how to control it, and whether one model can serve every task
Wang et al., “MaskGCT”, ICLR 2025 · Chen et al., “FlexiVoice”, ICLR 2026 · Zhang et al., “Vevo2”, IEEE TASLP (arXiv 2508.16332)
Speech Carries More Than Words
Linguistic
information
What is said: words, syntax, meaning
In the transcript
Paralinguistic
information
How it is said: intention, attitude, speaking style
Not in the transcript
Nonlinguistic
information
Age, gender, physical and emotional state
Not in the transcript
Acoustic scene
beyond Fujisaki’s three
Where it is said: background noise, music
Not in the transcript
Speech signal
A speech generator
has to produce all of it, and we want to steer each layer
Text gives only the linguistic layer; a generator must produce the rest, and control should reach every layer
After H. Fujisaki, “Prosody, Models, and Spontaneous Speech”, in Y. Sagisaka, N. Campbell, N. Higuchi (eds.), Computing Prosody, Springer, 1997. Acoustic scene added.
4
Speech Generation Tasks
5
Text-to-speech
text + voice prompt → speech
Voice conversion
source speech + target voice → speech
Singing synthesis and conversion
lyrics + melody (+ voice) → singing
Enhancement and restoration
degraded speech → clean speech
Speech editing
speech + edited text → speech
Video-to-audio
silent video → speech or sound
One conditional model
p(speech | conditions)
Control = which conditions we can set, and how precisely
Historically separate models, data and communities; today they can share one conditional model
Speech Generation as Token Prediction
6
Text LLM
Speech
carries
more
than
words
The model predicts the next token
Speech generation the same recipe, with speech tokens
Text
what to say
Speech prompt
a few seconds
Speech tokenizer
codec encoder
text tokens
prompt tokens
Language model
predicts the speech tokens
speech tokens
Decoder
to waveform
speech
Next-token prediction
Autoregressive: one token at a time, as in a text LLM
■ predicted in one step
VALL-E · in this talk: Vevo · FlexiVoice · Vevo2 · AnyEdit
Masked-token prediction
Parallel: many tokens at once, refined over a few steps
■ predicted in one step
SoundStorm · in this talk: MaskGCT · Metis · AnyEnhance
Once speech is a token sequence, generation becomes token prediction, and a speech prompt becomes the context
Borsos et al., “AudioLM”, IEEE/ACM TASLP 2023 · Wang et al., “VALL-E”, arXiv 2301.02111 (IEEE TASLP 2025) · Borsos et al., “SoundStorm”, arXiv 2305.09636
From Pipelines to Token Prediction
7
Pipeline TTS Tacotron 2 · FastSpeech 2 | Token prediction VALL-E | |
Formulation | Regression of a mel spectrogram, then a vocoder | Language modeling over codec tokens |
Training data | About 24 h of read speech, one speaker | 60k+ h of speech, thousands of speakers |
Duration and prosody | Explicit predictors (FastSpeech 2): duration, pitch, energy | Implicit: sampled with the tokens |
A new voice | Adaptation: fine-tuning or a speaker embedding | In context: a 3-second prompt, no fine-tuning |
Data what the model learns from
60k h is large, but it is read audiobook speech
Part I · Emilia · Emilia2
Tokens what the model predicts
VALL-E: 75 frames per second × 8 codebooks
Part I · DualCodec · TaDiCodec · FlexiCodec
Context what the model is given
VALL-E: the text and a 3-second speech prompt
Part II · MaskGCT · Vevo · FlexiVoice
The model became generic; the data, the tokens and the context determine what it can learn
Shen et al., “Tacotron 2”, ICASSP 2018 · Ren et al., “FastSpeech 2”, ICLR 2021 · Wang et al., “VALL-E”, arXiv 2301.02111 (IEEE TASLP 2025)
Agenda
8
I
Foundations
Data and tokens
Emilia · Emilia2
DualCodec · TaDiCodec
FlexiCodec
›
II
Controllability
Example → attribute →
words & example → editing
MaskGCT
NaturalSpeech 3 · Vevo
FlexiVoice
NVSpeech · AnyEdit
›
III
Unified modeling
Generation & understanding
Metis
Vevo2
AnyEnhance
CycleSpeech
›
IV
Preference alignment
Judge → reward → align
INTP
SpeechJudge
IPO
Disclaimer: This talk is not a comprehensive introduction to speech generation, but describes its development based on the speaker’s experience.
PART I
Foundations
Data and tokens
Emilia · Emilia2 · DualCodec · TaDiCodec · FlexiCodec
I · Foundations
II · Controllability
III · Unified modeling
IV · Preference alignment
Emilia: In-the-Wild Speech at Scale
10
Emilia-Pipe: six steps from raw audio to training data
101k h
Emilia · 6 languages
216k h
Emilia-Large
More spontaneous
More diverse than audiobook data
Spontaneous speech needs in-the-wild data
He et al., “Emilia”, IEEE SLT 2024; extended version, IEEE TASLP 2025
Emilia2: Long-Form and Dialogue Speech at Scale
11
Emilia2-Pipe: one recording gives short, long-form and dialogue examples, with control labels
229k h, 41 languages
Emilia: 6 languages, 30 s clips
Long-form and dialogue
Half of long-form hours > 5 min
Labels for control
4.77M instructions, 933k NV tags
0.6B models trained on it
Lowest CER/WER on TTSD-Eval
From seconds to minutes: long-form, dialogue and control labels in one dataset
Ni et al., “Emilia2”, arXiv 2611.13614
Speech Tokens: The Design Space
12
Tokenizers by bitrate, WER and quality (from TaDiCodec)
Acoustic or semantic?
Waveform detail, or linguistic content
Frame rate
Fewer frames, shorter sequences
How many codebooks?
Acoustic detail vs. modeling complexity
Reconstruction ≠ generation
The decoder drives naturalness
A good tokenizer balances reconstruction quality and ease of modeling
Li et al., “Investigating Neural Audio Codecs for Speech Language Model-Based Speech Generation”, IEEE SLT 2024
Speech Tokens: Toward Lower Frame Rates
13
Frames per second (Hz) · ours in red
DualCodec
Semantics in the first layer
TaDiCodec
Text-aware diffusion decoder
FlexiCodec
The rate and length are dynamic
Low frame rates are viable when tokens preserve semantic content
Li et al., “DualCodec”, Interspeech 2025 · Wang et al., “TaDiCodec”, NeurIPS 2025 · Li et al., “FlexiCodec”, ICLR 2026 · Hsu et al., “TASLA”, arXiv 2510.14934
FlexiCodec: Dynamic Token Rate
14
Merge similar frames
Adjacent frames with similar semantic features
3 to 12.5 Hz, one model
The rate adapts per utterance
Syllable-scale boundaries
Token boundaries align with syllables
Li et al., “FlexiCodec”, ICLR 2026 · Wang et al., “Interpreting and Evaluating Dynamic-Rate Speech Codec Boundaries”, arXiv 2609.36951
A dynamic rate spends tokens only where the content changes
PART II
Controllability
Example → attribute → words & example → editing
MaskGCT · NaturalSpeech 3 · Vevo · FlexiVoice · NVSpeech · AnyEdit
I · Foundations
II · Controllability
III · Unified modeling
IV · Preference alignment
Three Ways to Specify Control
16
1
By example
a speech prompt
All attributes are transferred together
MaskGCT
›
2
By attribute
separate factors
Controlling each attribute separately
NaturalSpeech 3 · Vevo
›
3
By words & example
description & speech prompt
Controlling with both a description & a speech prompt
FlexiVoice
From example, to attribute, to words & example
MaskGCT: No Alignment, No Phone Duration Prediction
17
Text → semantic tokens
A masked generative transformer
Semantic → acoustic tokens
A second masked generative transformer
Fully non-autoregressive
No alignment, only total length
Masked generative modeling: the robustness of NAR with the naturalness of AR
Wang et al., “MaskGCT”, ICLR 2025
MaskGCT: What a Prompt Controls
18
Clone from a prompt
The prompt sets the voice, across languages
Speech prompt
Same voice, in Chinese
The prompt also transfers emotion, accent and room acoustics
Set the speaking rate
Same words, three total durations
0.6 × duration
1.0 × duration
1.2 × duration
A prompt determines timbre, style and emotion jointly
Wang et al., “MaskGCT”, ICLR 2025 · audio: maskgct.github.io (excerpts)
The Limits of Control by Example
19
A prompt transfers all attributes
Timbre
Accent
Emotion
Speaking rate
Noise
Desired control
“This voice, but happier”
“This accent, another speaker”
“This melody, my voice”
This calls for disentangled representations
NaturalSpeech 3: Factorize, Then Generate
20
FACodec: content · prosody · timbre · acoustic detail
Factorized diffusion
Four subspaces
Content, prosody, timbre, detail
One prompt per attribute
Separate prompts for prosody and timbre
Large-scale data
1B parameters, 200k h
A prompt is decomposed into attribute-level prompts
Ju et al., “NaturalSpeech 3”, ICML 2024
Vevo: Disentanglement and Separate Control of Timbre and Style
21
Stage 1: AR
Content → content-style, from a style reference
Stage 2: flow matching
Content-style → mel, from a timbre reference
One framework
Conversion, accent, emotion, TTS
Each stage takes its own reference
Zhang et al., “Vevo”, ICLR 2025
Vevo: Zero-Shot Accent and Emotion Conversion
22
Accent: Hindi → American
Source
Style reference
Output
Emotion: neutral → angry
Source
Style reference
Output
No style labels
60k h of audiobooks, no conversion data
Self-supervised disentanglement gives zero-shot style control
Zhang et al., “Vevo”, ICLR 2025 · audio: versavoice.github.io
FlexiVoice: Timbre from the Prompt, Style from the Instruction
23
LLM backbone
Text, an instruction, a voice prompt
Progressive post-training
1 DPO
2 Multi-objective GRPO
3 Instruction GRPO
Progressive post-training enables decoupled control
Chen et al., “FlexiVoice”, ICLR 2026
FlexiVoice: One Instruction, Any Voice
24
Instruction: “Speak this sentence using happy emotion.”
Neutral voice prompt
Voice prompt
Output
Sad voice prompt
Voice prompt
Output
still happy
No voice prompt
Output
The instruction sets the style; the prompt sets only the voice
Chen et al., “FlexiVoice”, ICLR 2026 · audio: flexi-voice.github.io
NVSpeech: Modeling Nonverbal Vocalizations
25
Annotated seed
48k utterances, 18 word-level categories
Tags as tokens
An ASR model labels 573 h more
Nonverbal vocalizations are modeled as tokens in the text
Liao et al., “NVSpeech”, ICASSP 2026 (arXiv 2508.04195)
TTS with a [Breathing] tag
“Still need [Breathing]… adjustment”
TTS with a [Laughter] tag
“…I’ll give it a try, [Laughter]”
NV-Bench: Measuring Nonverbal Vocalization Synthesis
26
1,651 utterances
14 categories, with human audio
Instruction alignment
Paralinguistic character error rate
Acoustic fidelity
Distribution gap to real recordings
You cannot model & control what you cannot measure
Ni et al., “NV-Bench”, Interspeech 2026
AnyEdit: Edit the Words, Keep the Acoustic Scene
27
Edit in a clean token space
An AR editor predicts clean tokens from noisy audio
Render in context
Flow matching restores the noise, reverb and background music
Original, noisy recording
“…on Web Browsers and Web Servers”
Edited by AnyEdit
“…on Cloud Storage and Data Centers”
Change the words, keep the acoustic scene: speech and singing, no source separation
Zhang et al., “AnyEdit”, NeurIPS 2026 · audio: any-edit.github.io
Original, noisy recording
“我不唱声嘶力竭的情歌”
Edited by AnyEdit
“我不唱生死离别的情歌”
Control in Time and in the Real World
28
Timestamp control
Temporal control of sound events · PicoAudio, AudioTime
Pitch manipulation
A pitch-independent latent · Neurodyne
Noise-robust conversion
One shot, from a noisy reference · Noro
Streaming conversion
Zero lookahead · Zero-VC
Whisper ↔ normal
Pseudo-parallel data · WhispEar
Control also covers timing, pitch, and real-world conditions
Xie et al., “PicoAudio” and “AudioTime”, ICASSP 2025 · Gu et al., Interspeech 2025 · He et al., APSIPA ASC 2025 · Li et al.; Fang et al., Interspeech 2026
Demo: The Control Spectrum
29
By example
MaskGCT: slower speaking rate
By attribute
Vevo: American accent
By attribute
Vevo: angry
By attribute
Vevo2: Hindi accent
By words & example
FlexiVoice: “happy”
By tag
NVSpeech: [Breathing]
Increasingly fine-grained control: example → attribute → words & example → tag
Wang et al., ICLR 2025 (MaskGCT) · Zhang et al., ICLR 2025 (Vevo) · Zhang et al., IEEE TASLP (Vevo2) · Chen et al., ICLR 2026 (FlexiVoice) · Liao et al., arXiv 2508.04195 (NVSpeech)
Controllability: Part II Takeaway
30
Example
+ Easy to provide
− Entangled
All attributes transfer together
›
Attribute
+ Precise
− Needs disentanglement
Vevo: disentanglement & control
›
Words & example
+ Natural
− Needs post-training to enforce separation
FlexiVoice: post-training enforces the separation
Controllability takes different forms: example, attribute, words & example
PART III
Unified modeling
Generation & understanding
Metis · Vevo2 · AnyEnhance · CycleSpeech
I · Foundations
II · Controllability
III · Unified modeling
IV · Preference alignment
Tasks Differ in Where Each Factor Comes From
32
Task | Content | Prosody / melody | Style | Timbre |
Zero-shot TTS | text | prompt | prompt | prompt |
Instruction TTS | text | instruction | instruction | prompt |
Voice conversion | source | source | source | reference |
Accent / emotion conversion | source | reference | reference | source |
Singing from humming | lyrics | humming | reference | reference |
Enhancement / extraction | input | input | input | input + reference |
A new task is a new assignment of conditions, not a new model
Shaded cells: taken from a prompt, a reference, an instruction or humming
What “Unified” Means
33
1
Many tasks
TTS, conversion, enhancement, extraction
Metis · AnyEnhance
2
Many domains
Speech and singing in one model
Vevo2
3
Generation & understanding
Each task supervises the other
CycleSpeech
Earlier: SpeechT5 · Voicebox · UniAudio
Three levels: tasks, domains, generation & understanding
Ao et al., ACL 2022 (SpeechT5) · Le et al., NeurIPS 2023 (Voicebox) · Yang et al., ICML 2024 (UniAudio)
Metis: Speech Generation in Two Stages
34
Step 1: unconditional pre-training
Task-independent pre-training
Step 2: fine-tuning
Adapted to specific tasks
Only the first stage is pre-trained
The second stage (semantic → acoustic) is shared across tasks
MaskGCT is Metis without pre-training
Wang et al., “Metis”, NeurIPS 2025
Metis: Masked Generative Pre-training
35
300k h of unlabeled speech
No conditions
Speech tokens only, no text or labels
Mask and predict
The MaskGCT model, without text
A prefix stays unmasked
So prompting works after fine-tuning
Pre-training learns an unconditional prior over speech tokens
Wang et al., “Metis”, NeurIPS 2025
Metis: One Fine-Tuning Scheme for Any Condition
36
Task | Condition | Trainable |
TTS | text | LoRA 32M or full |
Voice conversion | source features | LoRA 9M |
Extraction | mixture + reference | LoRA 2–18M or full |
Enhancement | noisy speech | LoRA 2–18M or full |
Lip-to-speech | lip video | full |
A new task is a new condition
Wang et al., “Metis”, NeurIPS 2025
Metis: Five Tasks, One Pre-trained Model
37
Task | Metric | Baseline | Metis |
TTS | WER ↓ | MaskGCT 2.47 | 2.28 |
Enhancement | OVRL ↑ | MaskSR 3.25 | 3.44 |
Lip-to-speech | SIM ↑ | Lip2Speech-Unit 29.34 | 59.73 |
Voice conversion | SIM ↑ | Vevo 0.38 | 0.50 |
Extraction | NISQA ↑ | WeSep 4.04 | 4.36 |
Limitation: WER on conversion and extraction is still above the specialized baselines
Noisy input
Enhanced
Two speakers
One extracted
One pre-trained model, five tasks; TTS fine-tuned on 10k h vs. 100k h of training for MaskGCT
Wang et al., “Metis”, NeurIPS 2025 · test sets: SeedTTS-en, DNS 2020 (reverb), LRS2, VCTK, LibriMix · audio: metis-demo.github.io
Metis: The Effect of Pre-training
38
TTS WER (%), same 10k h of fine-tuning
Metis-Omni
four tasks, one model
5.90
Metis-Omni WER
2.70
Metis-Omni
with a text hint
Pre-training halves WER; joint fine-tuning lets conditions combine
Wang et al., “Metis”, NeurIPS 2025
4.27
GT WER
6.31
Metis-TSE WER
Vevo2: Singing Data Is Scarce
39
101k h
speech
7k h
singing
Singing is hard to annotate
Scores and alignments need experts
Singing = speech + a melody
Same words, same voice
So train them together
One model, both domains
Joint training transfers knowledge between speech and singing
Zhang et al., “Vevo2”, IEEE TASLP (arXiv 2508.16332)
Vevo2: Two Tokenizers
40
Prosody tokenizer
Content-style tokenizer
Prosody tokens
Chromagram · 6.25 Hz, 512 codes
Content-style tokens
Chromagram + Whisper · 12.5 Hz
Timbre removed by VQ
The bottleneck again, as in Vevo
One melody representation for speech, singing and instruments
Zhang et al., “Vevo2”, IEEE TASLP (arXiv 2508.16332)
Vevo2: Architecture and Conditioning
41
Stage 1: AR
Conditioned on text, melody and style
Stage 2: flow matching
Conditioned on the timbre reference
Each task
A different set of inputs
One framework; each task uses a different set of inputs
Zhang et al., “Vevo2”, IEEE TASLP (arXiv 2508.16332)
Vevo2: Mutual Benefit of Joint Training
42
Explicit or implicit
Prosody tokens given, or inferred from the text
Singing WER (%) ↓
Adding speech data: 25.38 → 15.78 (−38% relative)
Expressive-speech WER (%) ↓
Adding singing data: 15.52 → 14.32 (−8% relative)
Joint training helps both directions
Zhang et al., “Vevo2”, IEEE TASLP (arXiv 2508.16332)
Vevo2: Multi-Objective Post-training
43
Instrument-to-singing accuracy
Intelligibility reward
A Bradley-Terry model on INTP data
Prosody reward
Chromagram distance to the target
11.48
speech WER (%)
pre-trained: 14.32
7.66
singing WER (%)
pre-trained: 15.78
Zhang et al., “Vevo2”, IEEE TASLP (arXiv 2508.16332)
Reward models raise both text and melody accuracy
Vevo2: One Model, Many Tasks
44
Task (WER ↓) | Best baseline | Vevo2 |
Singing synthesis (en / zh) | TCSinger 40.83 / 14.26 | 24.77 / 9.83 |
Singing conversion (en / zh) | SeedVC 22.86 / 15.65 | 11.64 / 14.53 |
Accent / emotion conversion | Vevo 30.37 / 13.31 | 7.73 / 7.89 |
Speech / singing editing | F5-TTS 21.35 / 29.78 | 16.83 / 17.98 |
Humming (input)
Singing (output)
American accent
Hindi accent
One model: synthesis, conversion, editing
Zhang et al., “Vevo2”, IEEE TASLP (arXiv 2508.16332) · audio: versasinger.github.io
AnyEnhance: One Model for Speech and Singing Enhancement
45
Five tasks, no fine-tuning
Denoising, dereverberation, declipping, super-resolution, target-speaker extraction
A prompt turns on extraction
The same model for target-speaker extraction
A self-critic refines tokens
The model scores and refines its own tokens
Restoration can be unified as well: one model, with an optional prompt
Zhang et al., “AnyEnhance”, IEEE TASLP 2025
CycleSpeech: Mutual Supervision of Generation and Understanding
46
Generation
Can understanding models help with objective optimization?
Understanding
Can inferred profiles support reconstruction?
Each task supervises the other
Parameter sharing alone is insufficient
Goal: reciprocal feedback
Liao et al., “CycleSpeech”, arXiv 2609.24771
CycleSpeech: One Profile, Both Directions
47
Understanding: speech → profile
Input speech
Field | Predicted |
Speaker | Young woman, American |
Affect | Happy, playful |
Delivery | Medium-fast |
Texture | Bright, crisp |
Generation: instruction → speech
“Speak slowly and softly in a deep baritone, with a vulnerable, tentative tone.”
Output speech
20,046 examples share one 10-field profile
Liao et al., “CycleSpeech”, arXiv 2609.24771 · audio: cyclespeech.github.io
CycleSpeech: Model and Training
48
One backbone, two adapters
GEN: instruction → speech
UND: speech → profile
Stage 1: joint SFT
Paired speech and profiles
Stage 2: CycleGRPO
Five updates each, in turn
Each adapter provides the reward for the other
Liao et al., “CycleSpeech”, arXiv 2609.24771
CycleSpeech: Two Reciprocal Loops
49
Forward: trains GEN
Instruction
GEN: 6 candidates
UND: recover profiles
Match the target?
Reward: 0.70 profile · 0.15 content · 0.15 MOS
Backward: trains UND
Real speech
UND: profiles
GEN: re-synthesize
Match the speech?
Reward: 0.60 profile · 0.20 content · 0.20 style
Each direction serves as the reward model for the other
Liao et al., “CycleSpeech”, arXiv 2609.24771
CycleSpeech: Results in Chinese and English
50
Instruction match (%) ↑
Understanding score ↑
+4.5 / +10.1
instruction match vs. backbone (zh / en)
+1.5 / +1.8
style accuracy from the cycle itself
Cycle feedback improves instruction following and understanding
Liao et al., “CycleSpeech”, arXiv 2609.24771
Beyond Speech
51
Audio editing
Change only what the words ask · AUDIT
Video-to-sound
Sound synchronized with the video · FoleyCrafter
Accompaniment
From a melodic bottleneck · AnyAccomp
The same methods extend to general audio
Wang et al., NeurIPS 2023 (AUDIT) · Zhang et al., IJCV 2026 (FoleyCrafter) · Zhang et al., arXiv 2509.14052 (AnyAccomp)
Unified Modeling: Part III Takeaway
52
Model | TTS | Conversion | Singing | Enhancement | Understanding |
MaskGCT | ● | ● | – | – | – |
Vevo | ● | ● | – | – | – |
Metis | ● | ● | – | ● | – |
Vevo2 | ● | ● | ● | – | – |
AnyEnhance | – | – | ● | ● | – |
CycleSpeech | ● | – | – | – | ● |
Shared tokens
One interface across tasks
Pre-train, then condition
A task is a condition, not a model
Share data across domains
Speech teaches singing
Tasks supervise each other
Generation meets understanding
A unified model helps each task
PART IV
Preference alignment
Judge → reward → align
INTP · SpeechJudge · IPO
I · Foundations
II · Controllability
III · Unified modeling
IV · Preference alignment
Why Evaluation Is the Bottleneck
54
Human text accuracy vs. WER: WER fails on expressive speech
MOS does not scale
Too slow and costly for training-time feedback
WER misses hard cases
It fails on expressive speech
Alignment needs a reward
RLHF and DPO require a preference signal
Saeki et al., Interspeech 2022 (UTMOS) · Ouyang et al., NeurIPS 2022 (RLHF) · Rafailov et al., NeurIPS 2023 (DPO) · Tee et al., “SP-MCQA”, arXiv 2510.26190
INTP: Aligning TTS for Intelligibility
55
Human reading accuracy before and after alignment
Target the hard cases
Tongue twisters, code-switching
DPO for any architecture
Autoregressive, flow, masked
Preference data aimed at failure modes
Zhang et al., “Advancing Zero-shot TTS Intelligibility … via Preference Alignment”, ACL 2025
SpeechJudge: Alignment Needs Human Judgment
56
No preference data for speech
Available for text LLMs, not for speech
Focus: naturalness
The most basic subjective metric
Pairwise, not pointwise
Which of the two sounds better?
A dataset, a benchmark and a reward model
Zhang et al., “SpeechJudge”, ICLR 2026
SpeechJudge-Data: Human Agreement on Naturalness Is Only 69%
57
99k
speech pairs
69
annotators
6
TTS systems
69%
human agreement
Pairs by TTS system
Agreement: full · weak · weak disagreement · full disagreement
Chinese 74%, English 62%: close to InstructGPT’s 73%
Zhang et al., “SpeechJudge”, ICLR 2026 · InstructGPT: Ouyang et al., NeurIPS 2022
SpeechJudge-Eval: Can Models Judge Naturalness?
58
Agreement with humans on 1,000 clear pairs (%)
Objective metrics are near chance (50%)
SIM 44.5, UTMOS 53.7
AudioLLMs < 70%
Best is Gemini 2.5 Flash, at 69.1
Existing judges are far from human agreement
Zhang et al., “SpeechJudge”, ICLR 2026
SpeechJudge-GRM: A Generative Reward Model with Rationales
59
Accuracy on SpeechJudge-Eval (%)
SFT on rationales
25k pairs, teacher agrees with humans
RL on hard cases
17k pairs, teacher disagrees · DAPO
Reasoning plus RL beats a scalar reward model
Zhang et al., “SpeechJudge”, ICLR 2026
SpeechJudge in Use: Judge → Reward → Align
60
As a selector: best-of-100 vs. a random sample
As a reward: DPO post-training of a 0.5B TTS model
84 → 90
text accuracy (%)
+0.25
naturalness CMOS
The judge serves as a reward model for post-training
Zhang et al., “SpeechJudge”, ICLR 2026
IPO: Inference Policy Optimization for Speech Enhancement
61
Left: how often each of 8 decoder configurations is the best (300 utterances). Right: the IPO loop
Multi-policy candidates
One input decoded under 8 settings of CFG scale, temperature and step count
Online, on-policy updates
Pairs ranked by a multi-objective reward, refreshed from the current model
RL post-training of one AuraSE base model · 300-utterance subset
Method | WER (%) ↓ | DNSMOS OVRL ↑ | SIM ↑ |
AuraSE base | 8.81 | 3.20 | 0.758 |
+ DPO | 9.42 | 3.25 | 0.742 |
+ GRPO | 10.91 | 3.32 | 0.751 |
+ IPO | 7.59 | 3.37 | 0.775 |
IPO derives on-policy preference pairs from inference configurations; quality and WER improve together
Shen et al., “AuraSE”, arXiv 2610.06632 · baselines reproduced there: Zhang et al., AAAI 2026 (multi-metric DPO) · Wang et al., ICASSP 2026 (FlowSE-GRPO)
Three Things to Take Away
62
1
Controllability takes different forms
Vevo · Vevo2 · NaturalSpeech 3 · FlexiVoice
2
A unified model helps each task
MaskGCT → Metis · Vevo2 · CycleSpeech
3
Reward models matter for aligning with human preference
INTP · SpeechJudge · IPO
Back to the start
Humming
Singing
The opening example again: humming → singing
Control and unification both come down to the conditions one model accepts
Open Problems
63
1
Evaluation metrics
Existing objective metrics do not reflect perceived naturalness
2
Paralinguistic control in conversation & in the real world
Fine-grained, and inside a dialogue, in real-world environments
3
One model that understands, generates and converses
Full duplex, in real time, with multiple people
4
Judges beyond naturalness
Conversations, singing, safety
Next: speech models that understand, generate and converse in the real world
Thank You
Questions and comments are welcome
Thanks to the students and collaborators behind these works
Zhizheng Wu · The Chinese University of Hong Kong, Shenzhen
maskgct.github.io · versasinger.github.io · metis-demo.github.io · cyclespeech.github.io
Personal page
We are hiring postdoctoral researchers · research assistant professors
Backup · SpeechJudge: All Judges and Limitations
65
Judge | Accuracy (%) |
SIM | 44.5 |
FAD | 48.6 |
UTMOS | 53.7 |
WER | 57.9 |
DNSMOS | 57.9 |
Gemma-3n | 48.2 |
GPT-4o mini Audio | 50.5 |
MiMo-Audio | 54.1 |
Voxtral | 56.0 |
Phi-4-MM | 57.0 |
Qwen2.5-Omni-7B | 60.6 |
MiDashengLM | 61.6 |
Gemini-2.5-Pro | 66.5 |
Kimi-Audio | 67.0 |
GPT-4o Audio | 67.4 |
Gemini-2.5-Flash | 69.1 |
SpeechJudge-GRM (+ voting@10) | 77.2 (79.4) |
Chain of thought does not always help
helps Gemini, hurts Qwen2.5-Omni and Kimi-Audio
Limitations
zh/en only · utterance level · teacher bias
Zhang et al., “SpeechJudge”, ICLR 2026
| → space | next slide | click or tap | next (left quarter: back) |
| ← | previous slide | O esc | overview |
| Home End | first / last | F | full screen |
| 1…9 | play the n-th audio clip | swipe | next / previous |
| G then number | go to slide | ? | close this |