Controllable and Unified Modeling for Speech Generation

Zhizheng Wu

The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen)

Keynote speech at ISCSLP 2026

1

Acknowledgement to My Co-Authors in This Talk

2

In the order their work appears in the talk: Haorui He, Chaoren Wang, Jiaqi Li, Yuancheng Wang, Xueyao Zhang, Dekun Chen, Huan Liao, Qinke Ni, Yunjia Zhang, Jun’an Zhang, Yingda Shen

2

Speech Generation Today: Near-Human Naturalness

3

1

Zero-shot voice cloning

MaskGCT · zero-shot TTS

2

Instruction-based style control

FlexiVoice · instruction + voice prompt

3

Humming-to-singing synthesis

Vevo2 · melody + lyrics + voice

Speech prompt

a few seconds of a voice

Same voice, another language

Chinese text

maskgct.github.io

Instruction

“Speak this sentence using happy emotion.”

Voice prompt

spoken sadly

Output

same voice, now happy

flexi-voice.github.io

Melody: humming

Beethoven, Für Elise

Output: singing

that melody, a reference voice

versasinger.github.io

Naturalness is near human. Two questions remain: how to control it, and whether one model can serve every task

Wang et al., “MaskGCT”, ICLR 2025 · Chen et al., “FlexiVoice”, ICLR 2026 · Zhang et al., “Vevo2”, IEEE TASLP (arXiv 2508.16332)

3

Speech Carries More Than Words

Linguistic

information

What is said: words, syntax, meaning

In the transcript

Paralinguistic

information

How it is said: intention, attitude, speaking style

Not in the transcript

Nonlinguistic

information

Age, gender, physical and emotional state

Not in the transcript

Acoustic scene

beyond Fujisaki’s three

Where it is said: background noise, music

Not in the transcript

Speech signal

A speech generator

has to produce all of it, and we want to steer each layer

Text gives only the linguistic layer; a generator must produce the rest, and control should reach every layer

After H. Fujisaki, “Prosody, Models, and Spontaneous Speech”, in Y. Sagisaka, N. Campbell, N. Higuchi (eds.), Computing Prosody, Springer, 1997. Acoustic scene added.

4

4

Speech Generation Tasks

5

Text-to-speech

text + voice prompt → speech

Voice conversion

source speech + target voice → speech

Singing synthesis and conversion

lyrics + melody (+ voice) → singing

Enhancement and restoration

degraded speech → clean speech

Speech editing

speech + edited text → speech

Video-to-audio

silent video → speech or sound

One conditional model

p(speech | conditions)

Control = which conditions we can set, and how precisely

Historically separate models, data and communities; today they can share one conditional model

5

Speech Generation as Token Prediction

6

Text LLM

Speech

carries

more

than

words

The model predicts the next token

Speech generation  the same recipe, with speech tokens

Text

what to say

Speech prompt

a few seconds

Speech tokenizer

codec encoder

text tokens

prompt tokens

Language model

predicts the speech tokens

speech tokens

Decoder

to waveform

speech

Next-token prediction

Autoregressive: one token at a time, as in a text LLM

■ predicted in one step

VALL-E · in this talk:  Vevo · FlexiVoice · Vevo2 · AnyEdit

Masked-token prediction

Parallel: many tokens at once, refined over a few steps

■ predicted in one step

SoundStorm · in this talk:  MaskGCT · Metis · AnyEnhance

Once speech is a token sequence, generation becomes token prediction, and a speech prompt becomes the context

Borsos et al., “AudioLM”, IEEE/ACM TASLP 2023 · Wang et al., “VALL-E”, arXiv 2301.02111 (IEEE TASLP 2025) · Borsos et al., “SoundStorm”, arXiv 2305.09636

6

From Pipelines to Token Prediction

7

Pipeline TTS  Tacotron 2 · FastSpeech 2
Token prediction  VALL-E
Formulation
Regression of a mel spectrogram, then a vocoder
Language modeling over codec tokens
Training data
About 24 h of read speech, one speaker
60k+ h of speech, thousands of speakers
Duration and prosody
Explicit predictors (FastSpeech 2): duration, pitch, energy
Implicit: sampled with the tokens
A new voice
Adaptation: fine-tuning or a speaker embedding
In context: a 3-second prompt, no fine-tuning

Data  what the model learns from

60k h is large, but it is read audiobook speech

Part I · Emilia · Emilia2

Tokens  what the model predicts

VALL-E: 75 frames per second × 8 codebooks

Part I · DualCodec · TaDiCodec · FlexiCodec

Context  what the model is given

VALL-E: the text and a 3-second speech prompt

Part II · MaskGCT · Vevo · FlexiVoice

The model became generic; the data, the tokens and the context determine what it can learn

Shen et al., “Tacotron 2”, ICASSP 2018 · Ren et al., “FastSpeech 2”, ICLR 2021 · Wang et al., “VALL-E”, arXiv 2301.02111 (IEEE TASLP 2025)

7

Agenda

8

I

Foundations

Data and tokens

Emilia · Emilia2

DualCodec · TaDiCodec

FlexiCodec

›

II

Controllability

Example → attribute →
words & example → editing

MaskGCT

NaturalSpeech 3 · Vevo

FlexiVoice

NVSpeech · AnyEdit

›

III

Unified modeling

Generation & understanding

Metis

Vevo2

AnyEnhance

CycleSpeech

›

IV

Preference alignment

Judge → reward → align

INTP

SpeechJudge

IPO

Disclaimer: This talk is not a comprehensive introduction to speech generation, but describes its development based on the speaker’s experience.

8

PART I

Foundations

Data and tokens

Emilia  ·  Emilia2  ·  DualCodec  ·  TaDiCodec  ·  FlexiCodec

I · Foundations

II · Controllability

III · Unified modeling

IV · Preference alignment

9

Emilia: In-the-Wild Speech at Scale

10

Emilia-Pipe: six steps from raw audio to training data

101k h

Emilia · 6 languages

216k h

Emilia-Large

More spontaneous

More diverse than audiobook data

Spontaneous speech needs in-the-wild data

He et al., “Emilia”, IEEE SLT 2024; extended version, IEEE TASLP 2025

10

Emilia2: Long-Form and Dialogue Speech at Scale

11

Emilia2-Pipe: one recording gives short, long-form and dialogue examples, with control labels

229k h, 41 languages

Emilia: 6 languages, 30 s clips

Long-form and dialogue

Half of long-form hours > 5 min

Labels for control

4.77M instructions, 933k NV tags

0.6B models trained on it

Lowest CER/WER on TTSD-Eval

From seconds to minutes: long-form, dialogue and control labels in one dataset

Ni et al., “Emilia2”, arXiv 2611.13614

11

Speech Tokens: The Design Space

12

Tokenizers by bitrate, WER and quality (from TaDiCodec)

Acoustic or semantic?

Waveform detail, or linguistic content

Frame rate

Fewer frames, shorter sequences

How many codebooks?

Acoustic detail vs. modeling complexity

Reconstruction ≠ generation

The decoder drives naturalness

A good tokenizer balances reconstruction quality and ease of modeling

Li et al., “Investigating Neural Audio Codecs for Speech Language Model-Based Speech Generation”, IEEE SLT 2024

12

Speech Tokens: Toward Lower Frame Rates

13

Frames per second (Hz) · ours in red

Chart. Value: EnCodec 75, SpeechTokenizer 50, WavTokenizer 40, Mimi 12.5, DualCodec 12.5, FlexiCodec 12.5, TaDiCodec 6.25, TASLA 2.6

DualCodec

Semantics in the first layer

TaDiCodec

Text-aware diffusion decoder

FlexiCodec

The rate and length are dynamic

Low frame rates are viable when tokens preserve semantic content

Li et al., “DualCodec”, Interspeech 2025 · Wang et al., “TaDiCodec”, NeurIPS 2025 · Li et al., “FlexiCodec”, ICLR 2026 · Hsu et al., “TASLA”, arXiv 2510.14934

13

FlexiCodec: Dynamic Token Rate

14

Merge similar frames

Adjacent frames with similar semantic features

3 to 12.5 Hz, one model

The rate adapts per utterance

Syllable-scale boundaries

Token boundaries align with syllables

Li et al., “FlexiCodec”, ICLR 2026 · Wang et al., “Interpreting and Evaluating Dynamic-Rate Speech Codec Boundaries”, arXiv 2609.36951

A dynamic rate spends tokens only where the content changes

14

PART II

Controllability

Example → attribute → words & example → editing

MaskGCT  ·  NaturalSpeech 3  ·  Vevo  ·  FlexiVoice  ·  NVSpeech  ·  AnyEdit

I · Foundations

II · Controllability

III · Unified modeling

IV · Preference alignment

15

Three Ways to Specify Control

16

1

By example

a speech prompt

All attributes are transferred together

MaskGCT

›

2

By attribute

separate factors

Controlling each attribute separately

NaturalSpeech 3 · Vevo

›

3

By words & example

description & speech prompt

Controlling with both a description & a speech prompt

FlexiVoice

From example, to attribute, to words & example

16

MaskGCT: No Alignment, No Phone Duration Prediction

17

Text → semantic tokens

A masked generative transformer

Semantic → acoustic tokens

A second masked generative transformer

Fully non-autoregressive

No alignment, only total length

Masked generative modeling: the robustness of NAR with the naturalness of AR

Wang et al., “MaskGCT”, ICLR 2025

17

MaskGCT: What a Prompt Controls

18

Clone from a prompt

The prompt sets the voice, across languages

Speech prompt

Same voice, in Chinese

The prompt also transfers emotion, accent and room acoustics

Set the speaking rate

Same words, three total durations

0.6 × duration

1.0 × duration

1.2 × duration

A prompt determines timbre, style and emotion jointly

Wang et al., “MaskGCT”, ICLR 2025 · audio: maskgct.github.io (excerpts)

18

The Limits of Control by Example

19

A prompt transfers all attributes

Timbre

Accent

Emotion

Speaking rate

Noise

Desired control

“This voice, but happier”

“This accent, another speaker”

“This melody, my voice”

This calls for disentangled representations

19

NaturalSpeech 3: Factorize, Then Generate

20

FACodec: content · prosody · timbre · acoustic detail

Factorized diffusion

Four subspaces

Content, prosody, timbre, detail

One prompt per attribute

Separate prompts for prosody and timbre

Large-scale data

1B parameters, 200k h

A prompt is decomposed into attribute-level prompts

Ju et al., “NaturalSpeech 3”, ICML 2024

20

Vevo: Disentanglement and Separate Control of Timbre and Style

21

Stage 1: AR

Content → content-style, from a style reference

Stage 2: flow matching

Content-style → mel, from a timbre reference

One framework

Conversion, accent, emotion, TTS

Each stage takes its own reference

Zhang et al., “Vevo”, ICLR 2025

21

Vevo: Zero-Shot Accent and Emotion Conversion

22

Accent: Hindi → American

Source

Style reference

Output

Emotion: neutral → angry

Source

Style reference

Output

No style labels

60k h of audiobooks, no conversion data

Self-supervised disentanglement gives zero-shot style control

Zhang et al., “Vevo”, ICLR 2025 · audio: versavoice.github.io

22

FlexiVoice: Timbre from the Prompt, Style from the Instruction

23

LLM backbone

Text, an instruction, a voice prompt

Progressive post-training

1  DPO

2  Multi-objective GRPO

3  Instruction GRPO

Progressive post-training enables decoupled control

Chen et al., “FlexiVoice”, ICLR 2026

23

FlexiVoice: One Instruction, Any Voice

24

Instruction: “Speak this sentence using happy emotion.”

Neutral voice prompt

Voice prompt

Output

Sad voice prompt

Voice prompt

Output

still happy

No voice prompt

Output

The instruction sets the style; the prompt sets only the voice

Chen et al., “FlexiVoice”, ICLR 2026 · audio: flexi-voice.github.io

24

NVSpeech: Modeling Nonverbal Vocalizations

25

Annotated seed

48k utterances, 18 word-level categories

Tags as tokens

An ASR model labels 573 h more

Nonverbal vocalizations are modeled as tokens in the text

Liao et al., “NVSpeech”, ICASSP 2026 (arXiv 2508.04195)

TTS with a [Breathing] tag

“Still need [Breathing]… adjustment”

TTS with a [Laughter] tag

“…I’ll give it a try, [Laughter]”

25

NV-Bench: Measuring Nonverbal Vocalization Synthesis

26

1,651 utterances

14 categories, with human audio

Instruction alignment

Paralinguistic character error rate

Acoustic fidelity

Distribution gap to real recordings

You cannot model & control what you cannot measure

Ni et al., “NV-Bench”, Interspeech 2026

26

AnyEdit: Edit the Words, Keep the Acoustic Scene

27

Edit in a clean token space

An AR editor predicts clean tokens from noisy audio

Render in context

Flow matching restores the noise, reverb and background music

Original, noisy recording

“…on Web Browsers and Web Servers”

Edited by AnyEdit

“…on Cloud Storage and Data Centers”

Change the words, keep the acoustic scene: speech and singing, no source separation

Zhang et al., “AnyEdit”, NeurIPS 2026 · audio: any-edit.github.io

Original, noisy recording

“我不唱声嘶力竭的情歌”

Edited by AnyEdit

“我不唱生死离别的情歌”

27

Control in Time and in the Real World

28

Timestamp control

Temporal control of sound events · PicoAudio, AudioTime

Pitch manipulation

A pitch-independent latent · Neurodyne

Noise-robust conversion

One shot, from a noisy reference · Noro

Streaming conversion

Zero lookahead · Zero-VC

Whisper ↔ normal

Pseudo-parallel data · WhispEar

Control also covers timing, pitch, and real-world conditions

Xie et al., “PicoAudio” and “AudioTime”, ICASSP 2025 · Gu et al., Interspeech 2025 · He et al., APSIPA ASC 2025 · Li et al.; Fang et al., Interspeech 2026

28

Demo: The Control Spectrum

29

By example

MaskGCT: slower speaking rate

By attribute

Vevo: American accent

By attribute

Vevo: angry

By attribute

Vevo2: Hindi accent

By words & example

FlexiVoice: “happy”

By tag

NVSpeech: [Breathing]

Increasingly fine-grained control: example → attribute → words & example → tag

Wang et al., ICLR 2025 (MaskGCT) · Zhang et al., ICLR 2025 (Vevo) · Zhang et al., IEEE TASLP (Vevo2) · Chen et al., ICLR 2026 (FlexiVoice) · Liao et al., arXiv 2508.04195 (NVSpeech)

29

Controllability: Part II Takeaway

30

Example

+  Easy to provide

−  Entangled

All attributes transfer together

›

Attribute

+  Precise

−  Needs disentanglement

Vevo: disentanglement & control

›

Words & example

+  Natural

−  Needs post-training to enforce separation

FlexiVoice: post-training enforces the separation

Controllability takes different forms: example, attribute, words & example

30

PART III

Unified modeling

Generation & understanding

Metis  ·  Vevo2  ·  AnyEnhance  ·  CycleSpeech

I · Foundations

II · Controllability

III · Unified modeling

IV · Preference alignment

31

Tasks Differ in Where Each Factor Comes From

32

Task
Content
Prosody / melody
Style
Timbre
Zero-shot TTS
text
prompt
prompt
prompt
Instruction TTS
text
instruction
instruction
prompt
Voice conversion
source
source
source
reference
Accent / emotion conversion
source
reference
reference
source
Singing from humming
lyrics
humming
reference
reference
Enhancement / extraction
input
input
input
input + reference

A new task is a new assignment of conditions, not a new model

Shaded cells: taken from a prompt, a reference, an instruction or humming

32

What “Unified” Means

33

1

Many tasks

TTS, conversion, enhancement, extraction

Metis · AnyEnhance

2

Many domains

Speech and singing in one model

Vevo2

3

Generation & understanding

Each task supervises the other

CycleSpeech

Earlier: SpeechT5 · Voicebox · UniAudio

Three levels: tasks, domains, generation & understanding

Ao et al., ACL 2022 (SpeechT5) · Le et al., NeurIPS 2023 (Voicebox) · Yang et al., ICML 2024 (UniAudio)

33

Metis: Speech Generation in Two Stages

34

Step 1: unconditional pre-training

Task-independent pre-training

Step 2: fine-tuning

Adapted to specific tasks

Only the first stage is pre-trained

The second stage (semantic → acoustic) is shared across tasks

MaskGCT is Metis without pre-training

Wang et al., “Metis”, NeurIPS 2025

34

Metis: Masked Generative Pre-training

35

300k h of unlabeled speech

Chart. Value: English 185, Chinese 87, French 8, Korean 7.5, German 7, Japanese 2.5

No conditions

Speech tokens only, no text or labels

Mask and predict

The MaskGCT model, without text

A prefix stays unmasked

So prompting works after fine-tuning

Pre-training learns an unconditional prior over speech tokens

Wang et al., “Metis”, NeurIPS 2025

35

Metis: One Fine-Tuning Scheme for Any Condition

36

Task
Condition
Trainable
TTS
text
LoRA 32M or full
Voice conversion
source features
LoRA 9M
Extraction
mixture + reference
LoRA 2–18M or full
Enhancement
noisy speech
LoRA 2–18M or full
Lip-to-speech
lip video
full

A new task is a new condition

Wang et al., “Metis”, NeurIPS 2025

36

Metis: Five Tasks, One Pre-trained Model

37

Task
Metric
Baseline
Metis
TTS
WER ↓
MaskGCT 2.47
2.28
Enhancement
OVRL ↑
MaskSR 3.25
3.44
Lip-to-speech
SIM ↑
Lip2Speech-Unit 29.34
59.73
Voice conversion
SIM ↑
Vevo 0.38
0.50
Extraction
NISQA ↑
WeSep 4.04
4.36

Limitation: WER on conversion and extraction is still above the specialized baselines

Noisy input

Enhanced

Two speakers

One extracted

One pre-trained model, five tasks; TTS fine-tuned on 10k h vs. 100k h of training for MaskGCT

Wang et al., “Metis”, NeurIPS 2025 · test sets: SeedTTS-en, DNS 2020 (reverb), LRS2, VCTK, LibriMix · audio: metis-demo.github.io

37

Metis: The Effect of Pre-training

38

TTS WER (%), same 10k h of fine-tuning

Chart. From scratch: English 4.91, Chinese 4.98; Pre-trained: English 2.28, Chinese 2.3

Metis-Omni

four tasks, one model

5.90

Metis-Omni WER

2.70

Metis-Omni

with a text hint

Pre-training halves WER; joint fine-tuning lets conditions combine

Wang et al., “Metis”, NeurIPS 2025

4.27

GT WER

6.31

Metis-TSE WER

38

Vevo2: Singing Data Is Scarce

39

101k h

speech

7k h

singing

Singing is hard to annotate

Scores and alignments need experts

Singing = speech + a melody

Same words, same voice

So train them together

One model, both domains

Joint training transfers knowledge between speech and singing

Zhang et al., “Vevo2”, IEEE TASLP (arXiv 2508.16332)

39

Vevo2: Two Tokenizers

40

Prosody tokenizer

Content-style tokenizer

Prosody tokens

Chromagram · 6.25 Hz, 512 codes

Content-style tokens

Chromagram + Whisper · 12.5 Hz

Timbre removed by VQ

The bottleneck again, as in Vevo

One melody representation for speech, singing and instruments

Zhang et al., “Vevo2”, IEEE TASLP (arXiv 2508.16332)

40

Vevo2: Architecture and Conditioning

41

Stage 1: AR

Conditioned on text, melody and style

Stage 2: flow matching

Conditioned on the timbre reference

Each task

A different set of inputs

One framework; each task uses a different set of inputs

Zhang et al., “Vevo2”, IEEE TASLP (arXiv 2508.16332)

41

Vevo2: Mutual Benefit of Joint Training

42

Explicit or implicit

Prosody tokens given, or inferred from the text

Singing WER (%) ↓

Adding speech data: 25.38 → 15.78 (−38% relative)

Chart. Value: Singing only 25.38, Speech only 19.39, Speech + singing 15.78

Expressive-speech WER (%) ↓

Adding singing data: 15.52 → 14.32 (−8% relative)

Chart. Value: Speech only 15.52, Speech + singing 14.32

Joint training helps both directions

Zhang et al., “Vevo2”, IEEE TASLP (arXiv 2508.16332)

42

Vevo2: Multi-Objective Post-training

43

Instrument-to-singing accuracy

Intelligibility reward

A Bradley-Terry model on INTP data

Prosody reward

Chromagram distance to the target

11.48

speech WER (%)
pre-trained: 14.32

7.66

singing WER (%)
pre-trained: 15.78

Zhang et al., “Vevo2”, IEEE TASLP (arXiv 2508.16332)

Reward models raise both text and melody accuracy

43

Vevo2: One Model, Many Tasks

44

Task (WER ↓)
Best baseline
Vevo2
Singing synthesis (en / zh)
TCSinger 40.83 / 14.26
24.77 / 9.83
Singing conversion (en / zh)
SeedVC 22.86 / 15.65
11.64 / 14.53
Accent / emotion conversion
Vevo 30.37 / 13.31
7.73 / 7.89
Speech / singing editing
F5-TTS 21.35 / 29.78
16.83 / 17.98

Humming (input)

Singing (output)

American accent

Hindi accent

One model: synthesis, conversion, editing

Zhang et al., “Vevo2”, IEEE TASLP (arXiv 2508.16332) · audio: versasinger.github.io

44

AnyEnhance: One Model for Speech and Singing Enhancement

45

Five tasks, no fine-tuning

Denoising, dereverberation, declipping, super-resolution, target-speaker extraction

A prompt turns on extraction

The same model for target-speaker extraction

A self-critic refines tokens

The model scores and refines its own tokens

Restoration can be unified as well: one model, with an optional prompt

Zhang et al., “AnyEnhance”, IEEE TASLP 2025

45

CycleSpeech: Mutual Supervision of Generation and Understanding

46

Generation

Can understanding models help with objective optimization?

Understanding

Can inferred profiles support reconstruction?

Each task supervises the other

Parameter sharing alone is insufficient

Goal: reciprocal feedback

Liao et al., “CycleSpeech”, arXiv 2609.24771

46

CycleSpeech: One Profile, Both Directions

47

Understanding: speech → profile

Input speech

Field
Predicted
Speaker
Young woman, American
Affect
Happy, playful
Delivery
Medium-fast
Texture
Bright, crisp

Generation: instruction → speech

“Speak slowly and softly in a deep baritone, with a vulnerable, tentative tone.”

Output speech

20,046 examples share one 10-field profile

Liao et al., “CycleSpeech”, arXiv 2609.24771 · audio: cyclespeech.github.io

47

CycleSpeech: Model and Training

48

One backbone, two adapters

GEN: instruction → speech
UND: speech → profile

Stage 1: joint SFT

Paired speech and profiles

Stage 2: CycleGRPO

Five updates each, in turn

Each adapter provides the reward for the other

Liao et al., “CycleSpeech”, arXiv 2609.24771

48

CycleSpeech: Two Reciprocal Loops

49

Forward: trains GEN

Instruction

GEN: 6 candidates

UND: recover profiles

Match the target?

Reward:  0.70 profile · 0.15 content · 0.15 MOS

Backward: trains UND

Real speech

UND: profiles

GEN: re-synthesize

Match the speech?

Reward:  0.60 profile · 0.20 content · 0.20 style

Each direction serves as the reward model for the other

Liao et al., “CycleSpeech”, arXiv 2609.24771

49

CycleSpeech: Results in Chinese and English

50

Instruction match (%) ↑

Chart. Step-Audio-2-mini: Chinese 74.5, English 75.47; Joint SFT: Chinese 77.5, English 81.97; CycleGRPO: Chinese 79, English 85.53

Understanding score ↑

Chart. Step-Audio-2-mini: Chinese 57, English 59.45; Joint SFT: Chinese 77.98, English 76.54; CycleGRPO: Chinese 78.36, English 76.84

+4.5 / +10.1

instruction match vs. backbone (zh / en)

+1.5 / +1.8

style accuracy from the cycle itself

Cycle feedback improves instruction following and understanding

Liao et al., “CycleSpeech”, arXiv 2609.24771

50

Beyond Speech

51

Audio editing

Change only what the words ask · AUDIT

Video-to-sound

Sound synchronized with the video · FoleyCrafter

Accompaniment

From a melodic bottleneck · AnyAccomp

The same methods extend to general audio

Wang et al., NeurIPS 2023 (AUDIT) · Zhang et al., IJCV 2026 (FoleyCrafter) · Zhang et al., arXiv 2509.14052 (AnyAccomp)

51

Unified Modeling: Part III Takeaway

52

Model
TTS
Conversion
Singing
Enhancement
Understanding
MaskGCT
●
●
–
–
–
Vevo
●
●
–
–
–
Metis
●
●
–
●
–
Vevo2
●
●
●
–
–
AnyEnhance
–
–
●
●
–
CycleSpeech
●
–
–
–
●

Shared tokens

One interface across tasks

Pre-train, then condition

A task is a condition, not a model

Share data across domains

Speech teaches singing

Tasks supervise each other

Generation meets understanding

A unified model helps each task

52

PART IV

Preference alignment

Judge → reward → align

INTP  ·  SpeechJudge  ·  IPO

I · Foundations

II · Controllability

III · Unified modeling

IV · Preference alignment

53

Why Evaluation Is the Bottleneck

54

Human text accuracy vs. WER: WER fails on expressive speech

MOS does not scale

Too slow and costly for training-time feedback

WER misses hard cases

It fails on expressive speech

Alignment needs a reward

RLHF and DPO require a preference signal

Saeki et al., Interspeech 2022 (UTMOS) · Ouyang et al., NeurIPS 2022 (RLHF) · Rafailov et al., NeurIPS 2023 (DPO) · Tee et al., “SP-MCQA”, arXiv 2510.26190

54

INTP: Aligning TTS for Intelligibility

55

Human reading accuracy before and after alignment

Target the hard cases

Tongue twisters, code-switching

DPO for any architecture

Autoregressive, flow, masked

Preference data aimed at failure modes

Zhang et al., “Advancing Zero-shot TTS Intelligibility … via Preference Alignment”, ACL 2025

55

SpeechJudge: Alignment Needs Human Judgment

56

No preference data for speech

Available for text LLMs, not for speech

Focus: naturalness

The most basic subjective metric

Pairwise, not pointwise

Which of the two sounds better?

A dataset, a benchmark and a reward model

Zhang et al., “SpeechJudge”, ICLR 2026

56

SpeechJudge-Data: Human Agreement on Naturalness Is Only 69%

57

99k

speech pairs

69

annotators

6

TTS systems

69%

human agreement

Pairs by TTS system

Agreement: full · weak · weak disagreement · full disagreement

Chinese 74%, English 62%: close to InstructGPT’s 73%

Zhang et al., “SpeechJudge”, ICLR 2026 · InstructGPT: Ouyang et al., NeurIPS 2022

57

SpeechJudge-Eval: Can Models Judge Naturalness?

58

Agreement with humans on 1,000 clear pairs (%)

Chart. Value: SIM 44.5, UTMOS 53.7, WER 57.9, DNSMOS 57.9, Qwen2.5-Omni-7B 60.6, Gemini-2.5-Pro 66.5, Kimi-Audio 67, GPT-4o Audio 67.4, Gemini-2.5-Flash 69.1, SpeechJudge-GRM (ours) 79.4

Objective metrics are near chance (50%)

SIM 44.5, UTMOS 53.7

AudioLLMs < 70%

Best is Gemini 2.5 Flash, at 69.1

Existing judges are far from human agreement

Zhang et al., “SpeechJudge”, ICLR 2026

58

SpeechJudge-GRM: A Generative Reward Model with Rationales

59

Accuracy on SpeechJudge-Eval (%)

Chart. Value: Bradley-Terry RM 72.7, GRM (SFT) 75.3, GRM (SFT + RL) 77.2, GRM + voting@10 79.4

SFT on rationales

25k pairs, teacher agrees with humans

RL on hard cases

17k pairs, teacher disagrees · DAPO

Reasoning plus RL beats a scalar reward model

Zhang et al., “SpeechJudge”, ICLR 2026

59

SpeechJudge in Use: Judge → Reward → Align

60

As a selector: best-of-100 vs. a random sample

As a reward: DPO post-training of a 0.5B TTS model

84 → 90

text accuracy (%)

+0.25

naturalness CMOS

The judge serves as a reward model for post-training

Zhang et al., “SpeechJudge”, ICLR 2026

60

IPO: Inference Policy Optimization for Speech Enhancement

61

Left: how often each of 8 decoder configurations is the best (300 utterances). Right: the IPO loop

Multi-policy candidates

One input decoded under 8 settings of CFG scale, temperature and step count

Online, on-policy updates

Pairs ranked by a multi-objective reward, refreshed from the current model

RL post-training of one AuraSE base model · 300-utterance subset

Method
WER (%) ↓
DNSMOS OVRL ↑
SIM ↑
AuraSE base
8.81
3.20
0.758
+ DPO
9.42
3.25
0.742
+ GRPO
10.91
3.32
0.751
+ IPO
7.59
3.37
0.775

IPO derives on-policy preference pairs from inference configurations; quality and WER improve together

Shen et al., “AuraSE”, arXiv 2610.06632 · baselines reproduced there: Zhang et al., AAAI 2026 (multi-metric DPO) · Wang et al., ICASSP 2026 (FlowSE-GRPO)

61

Three Things to Take Away

62

1

Controllability takes different forms

Vevo · Vevo2 · NaturalSpeech 3 · FlexiVoice

2

A unified model helps each task

MaskGCT → Metis · Vevo2 · CycleSpeech

3

Reward models matter for aligning with human preference

INTP · SpeechJudge · IPO

Back to the start

Humming

Singing

The opening example again: humming → singing

Control and unification both come down to the conditions one model accepts

62

Open Problems

63

1

Evaluation metrics

Existing objective metrics do not reflect perceived naturalness

2

Paralinguistic control in conversation & in the real world

Fine-grained, and inside a dialogue, in real-world environments

3

One model that understands, generates and converses

Full duplex, in real time, with multiple people

4

Judges beyond naturalness

Conversations, singing, safety

Next: speech models that understand, generate and converse in the real world

63

Thank You

Questions and comments are welcome

Thanks to the students and collaborators behind these works

Zhizheng Wu  ·  The Chinese University of Hong Kong, Shenzhen

maskgct.github.io  ·  versasinger.github.io  ·  metis-demo.github.io  ·  cyclespeech.github.io

Personal page

WeChat

We are hiring    postdoctoral researchers  ·  research assistant professors

64

Backup · SpeechJudge: All Judges and Limitations

65

Judge
Accuracy (%)
SIM
44.5
FAD
48.6
UTMOS
53.7
WER
57.9
DNSMOS
57.9
Gemma-3n
48.2
GPT-4o mini Audio
50.5
MiMo-Audio
54.1
Voxtral
56.0
Phi-4-MM
57.0
Qwen2.5-Omni-7B
60.6
MiDashengLM
61.6
Gemini-2.5-Pro
66.5
Kimi-Audio
67.0
GPT-4o Audio
67.4
Gemini-2.5-Flash
69.1
SpeechJudge-GRM (+ voting@10)
77.2 (79.4)

Chain of thought does not always help

helps Gemini, hurts Qwen2.5-Omni and Kimi-Audio

Limitations

zh/en only · utterance level · teacher bias

Zhang et al., “SpeechJudge”, ICLR 2026

65
1 / 1drwuz.com? keys  ·  O overview

Keyboard

→ spacenext slideclick or tapnext (left quarter: back)
←previous slideO escoverview
Home Endfirst / lastFfull screen
1…9play the n-th audio clipswipenext / previous
G then numbergo to slide?close this