VibeVoice-1.5B-hf

VibeVoice is a novel framework designed for generating expressive, long-form, multi-speaker conversational audio, such as podcasts, from text. It addresses significant challenges in traditional Text-to-Speech (TTS) systems, particularly in scalability, speaker consistency, and natural turn-taking. A core innovation of VibeVoice is its use of continuous speech tokenizers (Acoustic and Semantic) operating at an ultra-low frame rate of 7.5 Hz. These tokenizers efficiently preserve audio fidelity while significantly boosting computational efficiency for processing long sequences. VibeVoice employs a next-token diffusion framework, leveraging a Large Language Model (LLM) to understand textual context and dialogue flow, and a diffusion head to generate high-fidelity acoustic details. The model can synthesize speech up to **90 minutes** long with up to **4 distinct speakers**, surpassing the typical 1-2 speaker limits of many prior models. ➡️ **Technical Report:** [VibeVoice Technical Report](https://arxiv.org/abs/2508.19205) ➡️ **Project Page:** [microsoft/VibeVoice](https://microsoft.github.io/VibeVoice) This model was contributed by [Eric Bezzam](https://huggingface.co/bezzam). Until VibeVoice is part of a Transformers release, you can install it from source: pip install git+https://github.com/huggingface/transformers.git A noise scheduler is needed as audio generation relies on a diffusion process. By default, the model will create a noise scheduler with `diffusers` internally. pip install diffusers pip install soundfile # for saving audio from transformers import AutoProcessor, AutoModelForTextToWaveform model_id = "vibevoice/VibeVoice-1.5B-hf" processor = AutoProcessor.from_pretrained(model_id) model = AutoModelForTextToWaveform.from_pretrained(model_id) import os from transformers import AutoProcessor, AutoModelForTextToWaveform model_id = "vibevoice/VibeVoice-1.5B-hf" text = "Hello, nice to meet you. How are you?" processor = AutoProcessor.from_pretrained(model_id) model = AutoModelForTextToWaveform.from_pretrained(model_id, device_map="auto") conversation = [{"role": "0", "content": [{"type": "text", "text": text}]}] inputs = processor.apply_chat_template( conversation, return_dict=True, tokenize=True, add_generation_prompt=True, ).to(model.device, model.dtype) audio = model.generate(**inputs) file_name = f"{os.path.basename(model_id)}_tts.wav" processor.save_audio(audio, file_name) print(f"Saved output to {file_name}") A voice can be cloned by providing a reference audio alongside the text within the chat template dictionary. import os from transformers import AutoProcessor, AutoModelForTextToWaveform, set_seed model_id = "vibevoice/VibeVoice-1.5B-hf" text = "Hello, nice to meet you. How are you?" set_seed(42) # for deterministic results processor = AutoProcessor.from_pretrained(model_id) model = AutoModelForTextToWaveform.from_pretrained(model_id, device_map="auto") sampling_rate = processor.feature_extractor.sampling_rate conversation = [ { "role": "0", "content": [ {"type": "text", "text": text}, { "type": "audio", "url": "https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/voices/en-Alice_woman.wav", }, ], } ] inputs = processor.apply_chat_template( conversation, return_dict=True, tokenize=True, add_generation_prompt=True, ).to(model.device, model.dtype) audio = model.generate(**inputs) fn = f"{os.path.basename(model_id)}_tts_clone.wav" processor.save_audio(audio, fn) print(f"Saved output to {fn}") Below is an example to generate a conversation between two speakers, whose voices are cloned by providing a reference audio for each unique role ID in the chat template. The example below also uses the `monitor_progress` option to track the generation progress. import os import time from transformers import AutoProcessor, AutoModelForTextToWaveform model_id = "vibevoice/VibeVoice-1.5B-hf" max_new_tokens = 400 # `None` to ensure full generation conversation = [ { "role": "0", "content": [ { "type": "text", "text": "Hello everyone, and welcome to the VibeVoice podcast. I'm your host, Linda, and today we're getting into one of the biggest debates in all of sports: who's the greatest basketball player of all time? I'm so excited to have Thomas here to talk about it with me.", }, { "type": "audio", "url": "https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/voices/en-Alice_woman.wav", }, ], }, { "role": "1", "content": [ { "type": "text", "text": "Thanks so much for having me, Linda. You're absolutely right—this question always brings out some seriously strong feelings.", }, { "type": "audio", "url": "https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/voices/en-Frank_man.wav", }, ], }, { "role": "0", "content": [ { "type": "text", "text": "Okay, so let's get right into it. For me, it has to be Michael Jordan. Six trips to the Finals, six championships. That kind of perfection is just incredible.", }, ], }, { "role": "1", "content": [ { "type": "text", "text": "Oh man, the first thing that always pops into my head is that shot against the Cleveland Cavaliers back in '89. Jordan just rises, hangs in the air forever, and just sinks it", }, ], }, ] processor = AutoProcessor.from_pretrained(model_id) model = AutoModelForTextToWaveform.from_pretrained(model_id, device_map="auto") inputs = processor.apply_chat_template( conversation, return_dict=True, tokenize=True, add_generation_prompt=True, ).to(model.device, model.dtype) model.generation_config.max_new_tokens = max_new_tokens start_time = time.time() audio = model.generate(**inputs, monitor_progress=True) generation_time = time.time() - start_time print(f"Generation time: {generation_time:.2f} seconds") fn = f"{os.path.basename(model_id)}_script.wav" processor.save_audio(audio, fn) print(f"Saved output to {fn}") For batch processing, a list of conversations can be passed to `processor.apply_chat_template`: import os import time from transformers import AutoProcessor, AutoModelForTextToWaveform model_id = "vibevoice/VibeVoice-1.5B-hf" max_new_tokens = 400 # `None` to ensure full generation conversation = [ [ { "role": "0", "content": [ { "type": "text", "text": "Hello everyone, and welcome to the VibeVoice podcast. I'm your host, Linda, and today we're getting into one of the biggest debates in all of sports: who's the greatest basketball player of all time? I'm so excited to have Thomas here to talk about it with me.", }, { "type": "audio", "url": "https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/voices/en-Alice_woman.wav", }, ], }, { "role": "1", "content": [ { "type": "text", "text": "Thanks so much for having me, Linda.", }, { "type": "audio", "url": "https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/voices/en-Frank_man.wav", }, ], }, ], [ { "role": "0", "content": [ { "type": "text", "text": "Hello and welcome to Planet in Peril. I'm your host, Alice. We're here today to discuss a really sobering new report that looks back at the last ten years of climate change. I'm joined by our expert panel. Welcome Carter, Frank, and Maya.", }, { "type": "audio", "url": "https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/voices/en-Alice_woman.wav", }, ], }, { "role": "1", "content": [ {"type": "text", "text": "Hi Alice, it's great to be here. I'm Carter."}, { "type": "audio", "url": "https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/voices/en-Carter_man.wav", }, ], }, { "role": "2", "content": [ {"type": "text", "text": "Hello, uh, I'm Frank. Good to be on."}, { "type": "audio", "url": "https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/voices/en-Frank_man.wav", }, ], }, { "role": "3", "content": [ {"type": "text", "text": "And I'm Maya. Thanks for having me."}, { "type": "audio", "url": "https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/voices/en-Maya_woman.wav", }, ], }, ], ] processor = AutoProcessor.from_pretrained(model_id) model = AutoModelForTextToWaveform.from_pretrained(model_id, device_map="auto") inputs = processor.apply_chat_template( conversation, return_dict=True, tokenize=True, add_generation_prompt=True, ).to(model.device, model.dtype) model.generation_config.max_new_tokens = max_new_tokens start_time = time.time() audio = model.generate(**inputs, monitor_progress=True) generation_time = time.time() - start_time print(f"Generation time: {generation_time:.2f} seconds") output_dir = f"{os.path.basename(model_id)}_batch" processor.save_audio(audio, output_dir) print(f"Saved output to {output_dir}") VibeVoice can also be loaded as a pipeline. We also show below how the diffusion parameters can be adjusted. import os import soundfile as sf from transformers import pipeline model_id = "vibevoice/VibeVoice-1.5B-hf" text = "Hello, nice to meet you. How are you?" pipe = pipeline("text-to-speech", model=model_id) conversation = [ { "role": "0", "content": [ {"type": "text", "text": text}, { "type": "audio", "url": "https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/voices/en-Alice_woman.wav", }, ], } ] generate_kwargs = {"guidance_scale": 1.3, "num_diffusion_steps": 10} output = pipe(conversation, generate_kwargs=generate_kwargs) fn = f"{os.path.basename(model_id)}_pipeline.wav" sf.write(fn, output["audio"], output["sampling_rate"]) print(f"Saved output to {fn}") VibeVoice can be trained with the loss outputted by the model. from transformers import AutoProcessor, AutoModelForTextToWaveform model_id = "vibevoice/VibeVoice-1.5B-hf" processor = AutoProcessor.from_pretrained(model_id) model = AutoModelForTextToWaveform.from_pretrained( model_id, diffusion_loss_weight=0.75, # by default, equal weighting (0.5) of language modeling loss (CE) and diffusion loss is applied device_map="auto" ) model.train() conversation = [ [ { "role": "0", "content": [ { "type": "text", "text": "VibeVoice is this novel framework designed for generating expressive, long-form, multi-speaker, conversational audio.", }, { "type": "audio", "url": "https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/realtime_model/vibevoice_tts_german.wav", }, ], } ], [ { "role": "0", "content": [ { "type": "text", "text": "Hello everyone and welcome to the VibeVoice podcast. I'm your host, Alex, and today we're getting into one of the biggest debates in all of sports: who's the greatest basketball player of all time? I'm so excited to have Sam here to talk about it with me. Thanks so much for having me, Alex. And you're absolutely right. This question always brings out some seriously strong feelings. Okay, so let's get right into it. For me, it has to be Michael Jordan. Six trips to the finals, six championships. That kind of perfection is just incredible. Oh man, the first thing that always pops into my head is that shot against the Cleveland Cavaliers back in '89. Jordan just rises, hangs in the air forever, and just sinks it.", }, { "type": "audio", "url": "https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/example_output/VibeVoice-1.5B_output.wav", }, ], } ], ] inputs = processor.apply_chat_template( conversation, tokenize=True, return_dict=True, processor_kwargs={"output_labels": True}, ).to(model.device, model.dtype) outputs = model(**inputs, ddpm_batch_multiplier=2, num_diffusion_steps=2) print(f"Total loss: {outputs.loss.item():.4f}") outputs.loss.backward() The model can be compiled with `torch.compile` for faster inference. A few warmup runs are needed before the compiled model reaches full speed. On an A100 with batch size 4, we observed a speed-up between compiled vs. non-compiled inference, see [this script](https://gist.github.com/ebezzam/c45b9fdee65f3029e17d566e30c59399). import os import time import torch from transformers import AutoModelForTextToWaveform, AutoProcessor, CompileConfig model_id = "vibevoice/VibeVoice-1.5B-hf" num_warmup = 5 max_new_tokens = 128 torch.set_float32_matmul_precision("high") processor = AutoProcessor.from_pretrained(model_id) model = AutoModelForTextToWaveform.from_pretrained(model_id, dtype=torch.bfloat16, device_map="auto").eval() conversation = [ [ { "role": "0", "content": [ {"type": "text", "text": "VibeVoice is a novel framework for generating expressive audio."}, { "type": "audio", "url": "https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/realtime_model/vibevoice_tts_german.wav", }, ], } ], ] * 4 # batch size 4 inputs = processor.apply_chat_template( conversation, tokenize=True, return_dict=True, add_generation_prompt=True, ).to(model.device, model.dtype) compile_config = CompileConfig(mode="default", dynamic=False) generate_kwargs = dict( **inputs, max_new_tokens=max_new_tokens, cache_implementation="static", compile_config=compile_config, ) print("Warming up...") warmup_start = time.time() with torch.inference_mode(): for _ in range(num_warmup): torch.compiler.cudagraph_mark_step_begin() _ = model.generate(**generate_kwargs) torch.cuda.synchronize() print(f"Warmup complete in {time.time() - warmup_start:.2f}s. Ready!") with torch.inference_mode(): torch.compiler.cudagraph_mark_step_begin() audio = model.generate(**generate_kwargs) output_folder = f"{os.path.basename(model_id)}_compiled_output" processor.save_audio(audio, output_folder) print(f"Saved output to {output_folder}") Transformer-based Large Language Model (LLM) integrated with specialized acoustic and semantic tokenizers and a diffusion-based decoding head. - LLM: [Qwen2.5-1.5B](https://huggingface.co/Qwen/Qwen2.5-1.5B) for this release. - Tokenizers: - Acoustic Tokenizer: Based on a σ-VAE variant (proposed in [LatentLM](https://arxiv.org/pdf/2412.08635)), with a mirror-symmetric encoder-decoder structure featuring 7 stages of modified Transformer blocks. Achieves 3200x downsampling from 24kHz input. Encoder/decoder components are ~340M parameters each. - Semantic Tokenizer: Encoder mirrors the Acoustic Tokenizer's architecture (without VAE components). Trained with an ASR proxy task. - Diffusion Head: Lightweight module (4 layers, ~123M parameters) conditioned on LLM hidden states. Predicts acoustic VAE features using a Denoising Diffusion Probabilistic Models (DDPM) process. Uses Classifier-Free Guidance (CFG) and DPM-Solver (and variants) during inference. - Context Length: Trained with a curriculum increasing up to 65,536 tokens. - Training Stages: - Tokenizer Pre-training: Acoustic and Semantic tokenizers are pre-trained separately. - VibeVoice Training: Pre-trained tokenizers are frozen; only the LLM and diffusion head parameters are trained. A curriculum learning strategy is used for input sequence length (4k -> 16K -> 32K -> 64K). Text tokenizer not explicitly specified, but the LLM (Qwen2.5) typically uses its own. Audio is "tokenized" via the acoustic and semantic tokenizers. The VibeVoice model is limited to research purpose use exploring highly realistic audio dialogue generation detailed in the [tech report](https://arxiv.org/pdf/2508.19205). Use in any manner that violates applicable laws or regulations (including trade compliance laws). Use in any other way that is prohibited by MIT License. Use to generate any text transcript. Furthermore, this release is not intended or licensed for any of the following scenarios: - Voice impersonation without explicit, recorded consent – cloning a real individual's voice for satire, advertising, ransom, social‑engineering, or authentication bypass. - Disinformation or impersonation – creating audio presented as genuine recordings of real people or events. - Real‑time or low‑latency voice conversion – telephone or video‑conference "live deep‑fake" applications. - Unsupported language – the model is trained only on English and Chinese data; outputs in other languages are unsupported and may be unintelligible or offensive. - Generation of background ambience, Foley, or music – VibeVoice is speech‑only and will not produce coherent non‑speech audio. While efforts have been made to optimize it through various techniques, it may still produce outputs that are unexpected, biased, or inaccurate. VibeVoice inherits any biases, errors, or omissions produced by its base model (specifically, Qwen2.5 1.5b in this release). Potential