Silent AI Video Is Dead: This Open-Source Model Dreams Up Movies and Soundtracks Together

The last great open problem in AI video generation was never sharper pixels — it was sound. For years, open-source video models produced beautiful silent clips. If you wanted dialogue, footsteps, or a crackling fire to match the picture, you stitched together three separate tools: a video generator, a voice or sound generator, and hours of manual sync in an editor. A new release from Team Kandinsky collapses that whole pipeline into a single model — and gives it away under the MIT license.

Meet Kandinsky 6.0 Video (arXiv:2610.05608, published October 4, 2026): a family of diffusion models that generate five-second video clips with synchronized 44 kHz audio — including lip-synced speech — from a text prompt or a starting image. Code, checkpoints, and a diffusers integration are all public. Let's break down what the researchers actually built, why it matters, and how you can run it yourself today.

What the researchers did

Kandinsky 6.0 Video comes in two sizes: Lite (3 billion parameters) for fast iteration, and Pro (29 billion parameters) for maximum quality. Both work in two modes:

• Text-to-audio-video (T2AV) — describe the scene and its sounds in words, get a complete clip back.
• Image-to-audio-video (TI2AV) — hand it a starting frame (a product shot, a family photo, a storyboard frame) and it animates it with a matching soundtrack.

A built-in super-resolution stage raises the final output to Full HD (1920×1080), and the audio runs at 44 kHz — the quality standard of most streaming services.

The technical heart of the paper is a dual-stream CrossDiT architecture. Instead of generating video first and bolting on audio afterward, the model runs two streams in parallel inside one diffusion transformer: a pretrained video stream and a freshly trained audio stream, connected by bidirectional cross-attention. The video stream sees what the audio stream is planning, and vice versa, at every step. Think of it like a film editor and a sound editor sharing the same timeline from the first day of production — rather than the sound team arriving after the picture is locked.

The training recipe matters too. The team first trained the audio stream from scratch on large-scale audio corpora, then trained both streams jointly on a massive filtered collection of paired audio-video segments (checked for audio quality and audio-visual match), while preserving each stream's individual fidelity. That was followed by supervised fine-tuning, reinforcement-learning-based post-training, and distillation. In side-by-side human evaluation, the Pro model clearly outperformed its predecessor (Kandinsky 5.0 Video Pro) and stayed competitive with leading audio-video generation models — particularly on speech quality, with coverage of the release reporting a steep drop in speech word error after the RL stage.

And then the part that makes this a story instead of just a paper: everything is open. Code and checkpoints under the MIT license, a diffusers integration, plus official support in ComfyUI, vLLM-Omni, SGLang, and FastVideo. A free Hugging Face demo of the distilled Pro model lets you try it without installing anything.

Why it matters

1. Sound was a closed-model feature. Now it isn't. Synchronized audio-video generation lived behind the APIs of proprietary labs. An MIT-licensed 29B option puts it in reach of any developer or studio with a few GPUs — including commercial use, which the MIT license permits.

2. One model replaces a three-tool pipeline. Until now, a synced short meant: video model for silent footage → voice/sound generator → manual alignment to avoid lip-sync jitter. That was roughly two hours of work per clip for a freelance creator. Kandinsky 6.0 does it in a single pass — one prompt in, finished clip out.

3. The honest limits are clear. Five seconds per clip (longer is planned), and the big model is heavy: on an RTX 4090, a Pro Full-HD clip takes around 20 minutes per the team's own benchmarks, while Lite is far quicker and the practical choice for iteration. Treat this as a stinger/B-roll/intro machine today, not a feature-film generator — and it's genuinely excellent at exactly that.

4. It's a research unlock, not just a tool. The open training recipe — staged audio pretraining, joint audio-video training, then SFT + RL + distillation — is now public. Expect a wave of follow-up work building on it within weeks.

How YOU can use it today

Option A — Try it free in your browser (zero install)

Head to the official Hugging Face demo space for the distilled Pro model (linked from the kandinsky-6 GitHub repo). Type a prompt, get a clip. This is the fastest way to learn what the model is good at before you invest in a local install.

Option B — No-code route: ComfyUI

In ComfyUI Manager, install the kandinsky6 and kandinsky6-sr extensions, restart, and drop the model node into your workflow. The repo ships example workflows to get you generating in minutes.

Option C — The developer route (local CLI)

You need an NVIDIA GPU and Python 3.13 or 3.14, plus the uv and just tools. Straight from the official README:

git clone https://github.com/kandinskylab/kandinsky-6.git
cd k6_video
just setup
just download pro-distill
just generate "a cat on a mat"

For serving behind an API, the team also documents a vLLM-Omni route with the kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers checkpoint, including image-conditioned generation:

python examples/offline_inference/image_to_video/image_to_video.py \
  --model kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers \
  --image first_frame.png \
  --prompt "A golden retriever runs along a sunny beach, waves crashing, cinematic footage" \
  --seed 42 \
  --enable-cpu-offload \
  --output kandinsky6_i2va.mp4

Prompting tips for synced audio

• Name the sounds explicitly. Don't just describe the scene — describe what you hear: "A barista slides a latte across the counter, steam hissing, and says 'Your coffee is ready!' — bright morning light, cozy café."
• Use quoted dialogue for lip-sync. Put spoken lines in quotes so the model knows what the mouth should be forming.
• Animate stills for promos. Upload one strong product or portrait frame as the image anchor, then prompt the motion + voiceover. A 5-second synced promo is often all a social post needs.
• Stitch clips into longer stories. Generate consistent 5-second beats and join them — each with its own synced sound — to build trailers and shorts scene by scene.

Key takeaways

• Kandinsky 6.0 Video generates 5-second clips with synchronized 44 kHz audio and lip-sync in one model — text-to-audio-video and image-to-audio-video.
• Its dual-stream CrossDiT architecture lets the video and audio streams coordinate throughout generation instead of syncing after the fact.
• The training recipe (audio-first pretraining → joint training → SFT → RL post-training → distillation) delivers standout speech quality in human evaluations.
• Code, checkpoints, and integrations (diffusers, ComfyUI, vLLM-Omni, SGLang, FastVideo) are MIT-licensed — free for commercial use.
• Start free on the Hugging Face demo, prototype in ComfyUI, and run the 3B Lite model locally for fast iteration.

Read the original paper: Kandinsky 6.0 Video: Foundation Models for Synchronized Video and Audio Generation (Team Kandinsky, arXiv:2610.05608). Code & checkpoints: github.com/kandinskylab/kandinsky-6 · Hugging Face.