Everpop

youtube shorts · captions · translation · quality

AI Clipping in Non-English Video: What Holds Up

What published benchmarks say about transcription accuracy by language, what breaks in non-Latin and right-to-left captions, and how to check.

· Everpop

Yes, for well-resourced languages. On clean read speech, the best-scoring systems put the major European languages in a narrow error band, while widely used open models run wider. Accuracy falls for lower-resourced languages and some non-Latin scripts. A leading public benchmark scores English and Spanish on different corpora, so there is no clean league table — and audio quality is the variable worth fixing first.

How accurate is AI transcription in Spanish, French or German?

The most useful public reference is the ASR Leaderboard, which compares "86 open-source and proprietary systems across 12 datasets". Its multilingual track "currently evaluates German, French, Italian, Spanish, and Portuguese transcription", scored on three read-speech corpora: CoVoST-2, FLEURS and MLS.

In that track's results table (version 4 of the paper, dated 30 March 2026), the top-scoring system, ElevenLabs Scribe v2, records word error rates of 2.27 for German, 3.28 for French, 2.58 for Italian, 2.33 for Spanish and 2.83 for Portuguese. A widely used open model in the same table, Whisper Large v3, sits higher and wider: 4.26 German, 6.36 French, 4.69 Italian, 3.65 Spanish, 4.96 Portuguese.

Two things follow. Figures like these describe read speech, not a podcast recorded over a laptop microphone. And even across five well-resourced European languages, one system runs from 3.65 to 6.36 — the language on the tape is not a rounding error.

Why can't you compare Spanish accuracy against English accuracy?

Because nobody is scoring them on the same audio. In the same paper's dataset table, the English track runs on meetings, earnings calls, TED talks and audiobooks; the multilingual track runs on CoVoST-2, FLEURS and MLS. No dataset appears in both, so subtracting one average from the other compares corpora, not languages.

That effect is large. Speechmatics' benchmarking guide notes that "Librispeech and CallHome, two commonly cited ASR test sets, frequently exhibit up to a 10% Word Error Rate difference for the same engine" — one engine, one language, different recordings.

Scoring rules bite too. The leaderboard "applies an English text normalization pipeline closely following that of Whisper" before computing error rates, while Speechmatics warns that "Normalization is inherently language specific". The metric itself also strains: a NAACL 2025 Findings paper argues that "as ASR systems expand to multilingual contexts, WER fails in various ways, particularly with morphologically complex languages or those without clear word boundaries".

Which languages actually fall off?

Meta's announcement of Omnilingual ASR, dated 10 November 2025, reports that its "7B-LLM-ASR system achieves state-of-the-art performance across 1,600+ languages, with character error rates (CER) below 10 for 78% of those languages". That is real coverage growth, and also a statement about the languages above that line. An earlier Meta paper reported a multilingual model that "more than halves the word error rate of Whisper on 54 languages of the FLEURS benchmark" — headroom that large shows how unsettled the long tail remains.

The writing system matters independently of how the language sounds. A study that fine-tuned Wav2Vec2-XLSR-53 on 25 languages with 15 writing systems found that "orthographic complexities significantly correlate with low ASR accuracy, while phonological complexity shows no significant correlation".

Getting the words right is a different problem from getting names right, and different again from whether to publish a translated version at all.

Does audio quality matter more than the language?

The published evidence points that way, though no single study measures both effects side by side.

Speechmatics lists "background noise", "codec (8-bit audio is lower quality)" and "cross-talk between speakers" among factors on which "ASR performance can differ significantly", and says of benchmark results: "This will depend a lot on the audio and speaker."

A 2025 study on transcription noise ran the experiment directly, in English. It reverberated clean audio and mixed in background office sound "with one of five signal-to-noise ratios", transcribed with whisper-small.en, and reported that across the resulting sets "The WER scores distribute within 0 and 0.9" — from a perfect transcript to one where most words are wrong. That span dwarfs the gaps between the European languages above, but it was measured on different material by different researchers, so treat it as a direction, not a ratio.

What breaks in non-Latin and right-to-left captions?

One failure mode word error rate cannot see is fluent output in the wrong writing system. A 2026 benchmark of ten models on FLEURS test sets, covering ten languages and six writing systems, calls this script collapse: "the model's decoder abandons the target script entirely, producing text that is phonetically plausible in some writing system but orthographically unusable". Across 100 model-language pairs, 21 collapsed, and in those cases "WER ranges from 96 % to 396 %" — poor accuracy on the face of it, with no hint of which alphabet came out. Collapse was model-dependent: two of the ten systems tested produced target-script output at 99 % or above on every language.

Word boundaries are a second problem. Unicode's text-segmentation annex states that "reliable detection of word boundaries in languages such as Thai, Lao, Chinese, or Japanese requires the use of dictionary lookup or other mechanisms, analogous to English hyphenation". Word-by-word caption timing has to decide what a word is before it can time one.

Right-to-left scripts add a display step. The Unicode Bidirectional Algorithm exists because "The Unicode Standard prescribes a memory representation order known as logical order", which is not the order Arabic or Hebrew appears in on screen. Lines mixing digits or Latin words into right-to-left text are where that ordering breaks.

Do the clip picks still hold up on an imperfect transcript?

Partly, and the evidence is indirect. The same transcription-noise study measured three downstream language tasks against rising error and found that on meeting summarization, "models tend to tolerate a noise level of about 0.2 WER (NTP between 0.07 and 0.3)". Picking a clip is not summarizing a meeting, and the authors note that other configurations "could involve non-English languages" — so read it as a plausible shape, not a measurement.

A pre-flight check before you clip non-English footage

Check Why it matters
The tool's own language list Coverage is a product fact, not a research fact — read it first
Your worst audio, not your best Benchmark figures come from read speech, and results depend heavily on the audio and the speaker
The script, not just the spelling Wrong-script output can register a finite error rate while being unreadable
Line breaks in spaceless scripts Unicode: reliably detecting word boundaries in Thai, Lao, Chinese or Japanese needs dictionary lookup or similar
Right-to-left lines with digits or Latin words Mixed-direction lines are where display ordering breaks
A read-through by a speaker of the language Wrong script, bad line breaks and reversed ordering are obvious to a speaker of the language; word error rate alone will not identify them

Where review fits in the workflow

Whatever tool you use, the last line of defence is a review step before anything burns in. That is the shape Everpop is built around — captions are word-timed and burned into the finished clip you review, nothing publishes until you approve it with one tap unless you turn Autopilot on, and the editor handoff exports FCPXML, EDL and SRT, so the transcript file leaves with you rather than staying locked in a tool.

Whether a particular tool covers your language at all is a separate question from any benchmark above, and the honest place to answer it is that tool's own documentation.

Frequently asked questions

Is AI transcription less accurate in languages other than English?
There is no clean comparison available, because the leading public benchmark scores its English track and its multilingual track on different corpora. What that benchmark does show is a narrow band across German, French, Italian, Spanish and Portuguese on read speech. Separately, published research shows accuracy falling for lower-resourced languages and for complex writing systems.
Which language has the lowest word error rate?
Not a question published benchmarks can settle. Test sets differ in recording quality, and the text normalization applied before scoring is itself language specific, so systems can swap places purely on which corpus you pick. Treat any cross-language league table with suspicion unless every language was scored on the same audio.
Do burned captions work in Arabic, Hebrew, Thai or Japanese?
They can, but two extra steps decide it. Right-to-left text needs bidirectional ordering to display correctly, and the risk concentrates in lines that mix in digits or Latin words. Scripts written without spaces need word segmentation before word-by-word timing means anything. Test both on real footage rather than assuming.
Does bad audio hurt more than the language does?
The published evidence points that way. One English-language study degraded clean audio with reverberation and office background sound at five signal-to-noise ratios, and the resulting transcripts spanned almost the full error scale. No study measures language and audio quality against each other directly, so treat this as a direction rather than a ratio.
How do I check whether a clipping tool handles my language?
Read that tool's own language documentation first, then run a sample of your worst audio rather than your best. Check that the output is in the right script, check line breaks and word timing, and have someone who speaks the language read the captions before anything is published.

Turn your long videos into Shorts — with receipts.

Your first video is on us: up to 3 clips, no card.

Start free →