How TTS Models Are Benchmarked

A text-to-speech model is not benchmarked with one magic score. Researchers usually test at least five separate qualities: whether it says the right words, whether it sounds natural, whether a cloned voice resembles its reference, how quickly it generates audio, and how reliably it works across languages and difficult text.

This distinction matters because the model with the lowest word error rate may not have the most expressive voice. The most natural model may need a powerful GPU. A system that performs well in English may be weak in a lower-resource language. The correct question is not simply, “Which TTS model is best?” It is, “Which model is best for this language, voice, device, and workflow?”

A valid TTS ranking compares the same model versions, test sentences, reference clips, recognizer, hardware, and inference settings. If any of those change, the scores may no longer be directly comparable.

What does a TTS benchmark measure?

1. Word error rate and character error rate

Word error rate, or WER, measures how accurately a model speaks the requested text. The generated audio is sent through an automatic speech recognizer. Its transcript is aligned with the target text, and substitutions, deletions, and insertions are counted:

WER = (substitutions + deletions + insertions) / reference words

A WER of 2 percent means the recognizer found roughly two word-level errors per 100 reference words. Character error rate, or CER, applies the same idea to characters. CER is often more useful for Chinese, Japanese, Korean, and other writing systems where word segmentation is not as straightforward.

WER is valuable, but it is not a direct measure of what a human hears. The chosen speech recognizer has its own language, accent, and noise biases. A natural pronunciation can be penalized if the recognizer misunderstands it. This is why a paper must identify the ASR model and text normalization rules. The official Seed-TTS evaluation toolkit, for example, uses Whisper-large-v3 for English, Paraformer-zh for Mandarin, and publishes the evaluation scripts.

2. Naturalness and audio quality

Mean opinion score

Mean opinion score, or MOS, asks human listeners to rate speech, commonly on a one-to-five scale. A typical test tells raters to judge naturalness or overall quality, then averages their responses. MOS is intuitive, but the result depends on the listeners, headphones, language proficiency, instructions, sample selection, and rating scale. A 4.2 from one study should not automatically beat a 4.0 from a different study.

Comparative MOS, or CMOS, asks whether one sample is better or worse than another. MUSHRA-style tests present a reference and several candidates. AB and ABX tests force direct choices between samples. Pairwise listening is often more sensitive than asking people to assign isolated numbers.

Predicted MOS

UTMOS and similar neural evaluators predict human quality ratings automatically. They make large experiments faster and repeatable, but they remain proxies trained on earlier listening data. A model can score well with an automatic quality estimator while still having distracting rhythm, emotion, or pronunciation problems for a specific listener.

3. Speaker similarity and voice cloning

Voice-cloning tests compare a generated sample with a permitted reference recording. An objective SIM score usually embeds both clips with a speaker-verification network, then calculates cosine similarity. Higher is better. Seed-TTS uses a WavLM-large speaker model for this purpose.

Objective similarity can be fooled by recording conditions, accent, or the embedding model. Human similarity MOS, often called SMOS, asks listeners whether the generated voice sounds like the reference. A strong study reports both objective SIM and a controlled human test.

Similarity is not consent. Only clone voices you own or have explicit permission to use. Benchmark quality does not grant rights to a voice, recording, performance, or model checkpoint. Read our guide to voice cloning ethics and privacy before testing a personal voice.

4. Speed, latency, and memory

Real-time factor, or RTF, divides synthesis time by generated audio duration. An RTF of 0.20 means a model takes one second to create five seconds of audio. An RTF below 1.0 is faster than real time.

• RTF: total generation time divided by output duration
• Time to first audio: delay before a streaming model returns its first playable chunk
• Throughput: audio seconds, tokens, or characters produced per second
• Peak memory: highest RAM or VRAM use during model loading and generation
• Cold start: time to load the model before the first request

Speed numbers require hardware context. CPU model, core count, GPU, precision, quantization, batch size, prompt length, output length, and warm-up runs can all change the result. A GPU RTF should never be placed beside a CPU RTF without a clear label. For a local app, first-audio latency and peak memory can matter more than maximum batch throughput.

5. Robustness and real-world text

Clean benchmark sentences are only a starting point. A production test set should include names, acronyms, URLs, currencies, percentages, dates, decimals, quotations, very short prompts, long paragraphs, code switching, and unusual punctuation. For audiobooks, test entire chapters and count repetitions, skipped phrases, hallucinated continuations, unstable voices, and failed generations.

Expressive systems also need tests for pacing, emotion, dialogue, and instruction following. These qualities are commonly evaluated with human preference studies or dedicated instruction-following suites. They cannot be inferred from WER alone.

Popular TTS benchmark sets

Modern multilingual papers often combine several of these. The OmniVoice paper uses LibriSpeech-PC, Seed-TTS, MiniMax-Multilingual-24, and a 102-language FLEURS evaluation. This is much more informative than reporting one English score.

OpenVox-supported models: what can be compared fairly?

The rankings below use the exact upstream model families available in OpenVox as of July 19, 2026. They rely on official papers, repositories, and model cards. A dash means the release does not publish a comparable number, not that the model performed badly.

Ranking 1: published English error rates across all supported models

Every OpenVox-supported model now has a published English WER or CER result. The table below includes all seven models. It is useful as a directional ranking, but it is not a controlled head-to-head test because the source corpora and evaluation pipelines differ. Chatterbox Multilingual V3 reports CER, while the other releases report WER. Both are transcription error rates, but CER and WER are not interchangeable.

Read this table as a map of published evidence, not proof that the first model will sound best in every English workflow. The fairest comparisons are still the smaller groups below, where models share the same test and scoring method.

Controlled comparison: PocketTTS, Kokoro, and Chatterbox Turbo

The PocketTTS paper evaluates three OpenVox-supported models with the same enhanced LibriSpeech test-clean samples and Whisper-large-v3 scoring. This makes it the cleanest direct comparison available for those models.

Source: Continuous Audio Language Models, Appendix F. This table ranks intelligibility on one English corpus. It is not an overall voice-quality ranking. Kokoro does not clone voices, while the paper also reports human quality and similarity evaluations for the cloning models.

Controlled comparison: OmniVoice and Qwen3-TTS on Seed-TTS

OmniVoice and the exact Qwen3-TTS 12Hz 0.6B Base checkpoint both publish Seed-TTS results. The values come from separate official reports, so small differences should be treated cautiously, but the benchmark and metrics are aligned well enough to show their relative strengths.

Sources: Qwen3-TTS official evaluation and OmniVoice Table 1. The OmniVoice report also gives SIM-o scores of 0.741 for English and 0.777 for Mandarin, while the Qwen repository's multilingual table reports 0.829 for English and 0.811 for Chinese. Because these similarity results come from different tables and evaluation pipelines, they should not be ranked against each other as if they were one experiment.

Ranking 2: multilingual coverage

1. OmniVoice: 646 languages
2. Supertonic 3: 31 languages
3. Chatterbox Multilingual: 23+ languages
4. Qwen3-TTS 0.6B Base: 10 languages
5. Kokoro-82M: 8 languages
6. PocketTTS: 6 languages
7. Chatterbox Turbo: English

Coverage is OmniVoice's clearest lead. Its official 24-language test reports an average 2.85 percent WER or CER and 0.830 SIM-o. Its 102-language FLEURS test reports 4.00 percent average CER and 0.788 average SIM-o. No other supported model publishes evidence across a similarly broad set.

Language count is not a quality score. Supertonic 3 publishes measured results for 21 of its 31 languages, including 0.86 percent WER for German, 1.13 percent for Spanish, 2.06 percent for English, 4.61 percent CER for Japanese, and 5.34 percent CER for Hindi. Test your own language in the OpenVox voice and language library before choosing a model.

Ranking 3: smallest model footprint

1. Kokoro-82M: 82 million parameters
2. Supertonic 3: about 99 million parameters
3. PocketTTS: 100 million parameters including its decoder
4. Chatterbox Turbo: 350 million parameters in the PocketTTS comparison
5. Chatterbox Multilingual: 500 million parameters
6. Qwen3-TTS and OmniVoice: 600 million parameter checkpoints, plus supporting components

Parameter count is only a rough proxy for download size and memory. Tokenizers, vocoders, precision, ONNX assets, and runtime frameworks also matter. Still, the ranking explains why Kokoro, Supertonic, and PocketTTS are attractive for local and CPU-oriented workflows. See our practical guide to running local TTS without a GPU.

Ranking 4: speed on normal local hardware

Lab speed results use different hardware, so they are not perfectly interchangeable. For normal laptops and desktops running OpenVox, the practical lightweight-model order is:

1. Supertonic 3: fastest on ordinary OpenVox hardware. Its compact ONNX runtime starts and generates more quickly than PocketTTS in typical local use. The official benchmark reports a mean CPU RTF of 0.172 with a roughly 2.0 GiB peak footprint in its test.
2. PocketTTS: second for normal local hardware and the stronger choice when streaming voice cloning is required. Its model card reports about 200 ms to first audio and about 6 times real-time generation on two CPU cores of a MacBook Air M4, but that vendor result is not a direct OpenVox comparison with Supertonic.
3. Kokoro: compact and CPU-friendly, but its official release does not provide a directly comparable current hardware table.

OmniVoice's model card advertises RTF as low as 0.025, or 40 times real time. That headline should not be placed above the CPU results without the same hardware and settings. For Select and Read, agents, and other interactive uses, first-audio latency is usually more important than offline batch throughput.

Ranking 5: voice cloning in OpenVox

This ranking combines speaker similarity evidence with practical OpenVox output quality, language reach, reference-audio requirements, and local hardware demands. It ranks the complete cloning workflow, not only one similarity number from incompatible test sets.

Only clone a voice you own or have explicit permission to use. Rankings describe technical capability and do not grant permission to reproduce a person's voice.

Ranking 6: strongest published benchmark evidence

1. OmniVoice: the broadest evidence, spanning English, Mandarin, 24-language, and 102-language benchmarks with WER or CER, similarity, naturalness, and human tests.
2. Qwen3-TTS 0.6B Base: detailed exact-checkpoint results for Seed-TTS and ten languages, including content consistency and speaker similarity.
3. PocketTTS: a reproducible English comparison with WER, human quality, similarity, model size, and CPU capability.
4. Supertonic 3: useful multilingual reading-accuracy and deployment charts, though less detail is published about the exact test corpus and scoring pipeline.
5. Chatterbox Multilingual V3: per-language CER is now published across the multilingual release, while objective speaker-similarity and MOS results are still pending.
6. Kokoro and Chatterbox Turbo: useful controlled English results appear in the PocketTTS study, but their release documentation provides less complete standardized coverage than the models above.

This ranks transparency and breadth of evidence, not sound quality. A model can be excellent and still be under-documented. It should then be marked “not enough comparable public data,” not assigned a guessed score.

Licensing is part of model selection

Benchmark leadership does not decide whether a checkpoint fits a commercial workflow. As of this article's publication date, Kokoro and Qwen3-TTS publish Apache 2.0 weights, Chatterbox publishes MIT weights, PocketTTS publishes CC BY 4.0 weights with gated acceptable-use conditions, and Supertonic 3 uses OpenRAIL-M for the model.

OmniVoice requires special attention. Its code is Apache 2.0, but its official model card says the pretrained checkpoint is CC-BY-NC because of training-data constraints. That means a code license must not be mistaken for a checkpoint license. Always read the current upstream license for the exact model version and verify that it permits your intended use. This article is a technical comparison, not legal advice.

How to run a useful TTS benchmark yourself

1. Freeze the versions. Record the exact checkpoint hash, app version, runtime, and settings.
2. Build a representative test set. Include each language, domain, and text pattern you use.
3. Use identical inputs. Keep text, reference clips, sample rate, and voice conditions fixed.
4. Measure intelligibility. Run one ASR system and normalization pipeline across every output.
5. Run blind listening. Randomize model names and ask separate questions about quality and similarity.
6. Measure deployment. Log cold start, first audio, RTF, peak memory, and failures on the target computer.
7. Test long-form stability. Count skipped, repeated, or invented content over chapters, not only sentences.
8. Publish uncertainty. Report sample count, confidence intervals, ties, and missing results.

A creator can use a smaller version of this method. Prepare 20 difficult sentences and one five-minute passage, generate them with the same models, hide the filenames, and score pronunciation, naturalness, voice fit, and fatigue. Then repeat the speed test on the computer that will actually do the work.

Which OpenVox model should you choose?

• Choose OmniVoice when language breadth and cross-language cloning research are the priority, subject to its checkpoint license.
• Choose Qwen3-TTS when you want strong zero-shot cloning evidence across its ten supported languages and have suitable hardware.
• Choose Supertonic 3 for compact, multilingual, CPU-friendly reading and a broad preset voice library.
• Choose PocketTTS for low-latency streaming and CPU voice cloning in English, French, German, Spanish, Portuguese, or Italian.
• Choose Kokoro for a very small, permissively licensed preset-voice model with strong English intelligibility.
• Choose Chatterbox Turbo for fast English cloning and expressive controls where its higher WER on the cited test is acceptable.
• Choose Chatterbox Multilingual for expressive cloning across its supported languages, then validate the exact language yourself because V3 lacks a comparable public numeric table.

No leaderboard can tell you whether a particular narrator suits your audiobook or whether names from your field are pronounced correctly. Use published benchmarks to narrow the field, then listen to multiple voices and test your real workflow. Our broader guide to the best local and cloud TTS models in 2026 explains the practical tradeoffs beyond model scores.

Primary sources

• Seed-TTS official evaluation toolkit
• OmniVoice paper and benchmark tables
• OmniVoice model card and checkpoint license
• Qwen3-TTS official repository and evaluation tables
• PocketTTS evaluation in the CALM paper
• Kyutai announcement of six-language PocketTTS support
• PocketTTS model card and runtime details
• Supertonic 3 model card and benchmark charts
• Kokoro-82M model card
• Chatterbox model family card
• Chatterbox Multilingual V3 CER benchmark
• Chatterbox Turbo cloning and latency study
• SOMOS human MOS dataset paper