SV2TTS · Nepali · 10 speakers
A three-stage neural pipeline that learns a speaker’s timbre from one short reference clip, then reads arbitrary Nepali text back in that voice — without ever being trained on the speaker.
Built on the SV2TTS architecture and adapted to a low-resource language. The model server runs on two CPU cores, so everything you hear is synthesised live.
How it works
The trick behind zero-shot cloning: the speaker’s identity is separated from the words entirely. Stage one answers who, stage two answers what, stage three makes it audible. Watch them light up when you synthesise below.
GE2E · 3-layer LSTM
Hears ~5s of a voice and distils it to a single fixed-length vector that captures timbre, independent of what was said.
reference wav256-d embedding
Tacotron · attention
Conditions on that vector and decodes the text into a mel spectrogram, one frame at a time, attending to characters as it goes.
text + embedding80 × T mel
WaveRNN · autoregressive
Turns the spectrogram back into pressure over time, sampling each of 16,000 values per second in sequence.
80 × T mel16 kHz audio
The studio
Output
Original on the left, cloned on the right, stage by stage. Every plot is drawn in your browser from raw arrays — the server used to render these as PNGs and ship 592 KB per request, 86% of it pictures.
01
The 256-d vector that encodes the voice, as a 16×16 grid. If cloning worked, the two should look alike — that similarity is the whole objective.
Cosine similarity
The angle between the two 256-d vectors. Both norms come out at 1.000 because the GE2E encoder L2-normalises its output — every voice lands on the unit sphere — so the cosine here reduces to the plain dot product. 1.000 would mean the encoder places the cloned audio at exactly the same point as the original, i.e. it considers them the same speaker.
02
80 mel bands over time. The original is what the reference audio looks like; the cloned one is what Tacotron generated and handed to the vocoder.
Which character the decoder attended to at each step. A clean diagonal means the model tracked the text; smearing or stalling means it lost its place. There is no “original” here — alignment only exists for generated speech.
03
The waveform, back in the time domain. Click either plot to seek.
Server timings
Under the hood
The synthesizer’s vocabulary is 68 ASCII symbols — the stock Tacotron
set. Devanagari never reaches the network: the text front-end runs
unidecode first, so नमस्ते becomes
namaste before embedding lookup. The Preeti and romanised keyboards
above are typing aids for the human, not the model.
Digits aren’t in those 68 symbols and nothing expands them, so
२०२४ transliterates to 2024 and then
vanishes character by character. The studio warns you rather than letting it
fail silently. Proper Nepali numeral expansion is genuinely hard — the
words above twenty are irregular — so it is listed as future work rather
than half-done.
The encoder was trained for speaker verification, not synthesis. It learned to map any voice to a point in 256-d space where the same person lands in the same place. That generalises to voices it has never heard, which is what makes five seconds enough.
The model server is a free Hugging Face Space. The vocoder is autoregressive — it samples 16,000 values per second of audio, in sequence — so longer text costs proportionally more. A daily cron keeps the Space awake, since a cold start means loading roughly 490 MB of weights.
Output is audibly synthetic, and attention sometimes loses the diagonal on long or unusual input — you can watch that happen in the alignment plot. A listening study put it at 3.45/5 for naturalness and 3.28/5 for speaker similarity across 57 raters, with every clip and caveat published.
The decoder’s prenet applies dropout at inference time, not just during training. Synthesising the same sentence twice gives two slightly different readings — inherited from the original architecture, and audible if you run it back to back.