Published by Video Runtime · Source check 2026-09-21 · Evidence method · Suggest a correction
EDITORIALLY VERIFIED PRODUCT INFORMATION
About JoyAI-Video-Edit
Real-time instruction-guided editing of open-ended video streams with an autoregressive diffusion transformer. An MLLM condition encoder, a causal video VAE and a 16B multimodal diffusion transformer edit frames causally as they arrive, without waiting for the full video, a predefined length or future frames. Aligned autoregressive distribution-matching distillation, long-horizon optimisation and bounded KV-state inference sustain high-throughput streaming while limiting temporal drift.
Key features
Publishes deployment code, checkpoints, a live webcam demo on Hugging Face and a consumer-GPU path for a single RTX 5090. Instructions cover subject, local, background, style, motion and reference-image-guided edits.
Getting started
Read the deployment guide, download the released DiT weights from Hugging Face, reproduce the documented benchmark profile on your GPU, then try the browser demo with a webcam before wiring your own stream.
Real-time interaction
An MLLM condition encoder, a causal video VAE and a 16B multimodal diffusion transformer edit frames causally as they arrive, without waiting for the full video, a predefined length or future frames. Aligned autoregressive distribution-matching distillation, long-horizon optimisation and bounded KV-state inference sustain high-throughput streaming while limiting temporal drift.
Pricing
Apache-2.0 code; self-hosted compute costs extra Repository code is published under Apache-2.0. Confirm separate model, dataset and dependency terms before reuse. No hosted price is inferred. Budget for the documented GPU, storage and setup requirements.
Limitations
The documented real-time profiles need high-end GPUs; the consumer profile is 840 × 480 at 24 FPS on an RTX 5090. Quality figures come from the maintainers' own deployment benchmark and the model targets editing, not text-to-video world generation.
EDITORIAL VIEW
Editor's Verdict
JoyAI-Video-Edit is included as a source-available implementation relevant to real-time generative video or interactive world systems.
Run the supplied example on representative hardware and review every linked licence before production use.
How It Works
- An MLLM condition encoder, a causal video VAE and a 16B multimodal diffusion transformer edit frames causally as they arrive, without waiting for the full video, a predefined length or future frames.
- Aligned autoregressive distribution-matching distillation, long-horizon optimisation and bounded KV-state inference sustain high-throughput streaming while limiting temporal drift.
What We Like
- Publishes deployment code, checkpoints, a live webcam demo on Hugging Face and a consumer-GPU path for a single RTX 5090.
- Instructions cover subject, local, background, style, motion and reference-image-guided edits.
Current Limitations
- The documented real-time profiles need high-end GPUs; the consumer profile is 840 × 480 at 24 FPS on an RTX 5090.
- Quality figures come from the maintainers' own deployment benchmark and the model targets editing, not text-to-video world generation.
Pricing & Access
- Repository code is published under Apache-2.0. Confirm separate model, dataset and dependency terms before reuse.
- No hosted price is inferred. Budget for the documented GPU, storage and setup requirements.
Verification Summary
- Official repository and licence reviewed on 2026-09-21. Video Runtime has not installed or benchmarked this project.
- Performance and compatibility statements below are attributed to the maintainers and are not site measurements.
Technical notes & official performance
Official Performance Data
- Maintainers report 30 FPS end-to-end at 720 × 1248, 840 × 480 at 24 FPS on one RTX 5090 (32 GB) and a live RTX PRO 6000 demo. These are author claims, not Video Runtime measurements.
Technical Notes
- The README documents end-to-end 30 FPS at 720 × 1248 in the authors' deployment benchmark and an upgraded reference-guided checkpoint released 2026-08-14.
Sources
- Official GitHub repository ↗github · accessed 2026-09-21
- Code licence ↗github · accessed 2026-09-21
Access, stage & mechanism
These are separate source-checked facts, not a live uptime monitor. Unverified fields are left open; transport and persistent sessions do not establish how frames are generated.
- Release stage
- Not verified
- Access mode
- Not verified
- Source availability
- Not verified
- Transport
- Not verified
- Interaction
- Not verified
- Session mode
- Not verified
- Generation mechanism
- Not verified
- Continuity method
- Not verified
- Licence status
- Not verified
- Tasks
- Not classified
Read how official-source and hands-on records stay separate in our Methodology.