Blogs›Replicate›Video Style Transfer with LoRAs
Replicate has introduced support for fine-tuning open-source video models, specifically adapting the Musubi Tuner for HunyuanVideo. This allows users to train custom video generation models on their own visual content, enabling personalized video stylization with in-motion style transfer. The process involves creating training data using a dedicated model (`zsxkib/create-video-dataset`), training the LoRA model on Replicate, and then generating videos via the browser, Playground, or API. This post highlights further advancements in fine-tuning experiments, including custom LoRAs for effects like "cakeify", "squish", and "dissolve" on Wan2.1, enabling flexible transformations and creative control over video generation. It also introduces specific LoRA trainers for Wan2.1, such as Ostris's and zsxkib's, facilitating fine-tuning with images and applying styles to existing videos. The post also notes upgrades to text-to-video models like Luma, which now generates 720p video faster, and Kling v1.6 Pro, which adds end frame support for enhanced control over video generation.
7/31/2025
This post details the release and optimization of Wan 2.2, an open-source video generation model. It highlights significant improvements in generation speed (~30s for 480p), cost reduction ($0.05 per video), and quality (sharp physics, more control). It introduces optimized image-to-video and text-to-video endpoints at 480p and 720p resolutions. The post also includes API examples for using the Wan 2.2 models via Replicate's platform.
5/15/2025
This post details the integration of Hugging Face LoRAs into Replicate's inference pipeline. It explains how Hugging Face's inference client can now route requests to Replicate, specifically to the `black-forest-labs/flux-dev-lora` model. The core technical mechanism involves passing the LoRA weights as a dynamic input parameter (`lora_weights`) to this single backend model, enabling support for over 30,000 LoRAs without individual hosting. This expands the usability of LoRAs by allowing them to be run directly from the Hugging Face interface with Replicate as the inference provider.
4/1/2025
This post details the implementation of video style transfer using LoRAs with the Wan 2.1 model on Replicate. It explains how to use premade styles by providing a `lora_weights` URL to models like `wavespeedai/wan-2.1-t2v-480p` and `wavespeedai/wan-2.1-t2v-720p`. It also outlines the process for users to train their own custom styles using the `ostris/wan-lora-trainer` model, which involves preparing training images, setting a trigger word, and configuring captioning options. The post highlights the technical aspects of applying LoRAs for fast inference and the user-facing workflow for both using and creating custom video styles.
3/28/2025
This post details advancements in fine-tuning experiments for video generation models, specifically highlighting custom LoRAs for effects like "cakeify", "squish", and "dissolve" on Wan2.1. It explains how adjusting LoRA weights and prompts allows for creative transformations, such as changing objects within a scene (e.g., a knife to a lightsaber) and altering the content (e.g., to cake). The post also introduces two specific LoRA trainers for Wan2.1: Ostris's trainer for fine-tuning with images and zsxkib's trainer for applying styles to existing videos. Additionally, it mentions upgrades to text-to-video models, including Luma's faster 720p video generation and Kling v1.6 Pro's new end frame support for greater control.
1/24/2025
This post details the integration of the Musubi Tuner for fine-tuning open-source video models like HunyuanVideo on Replicate. It introduces a new model (`zsxkib/create-video-dataset`) for automating the creation of training data from video URLs or files, including clip segmentation and caption generation. The process of training a LoRA model on Replicate using this data is outlined, along with instructions for running the fine-tuned models via the UI and API. Programmatic training using the Replicate Python client is also demonstrated, showcasing the end-to-end workflow from data preparation to model deployment and inference.
1/17/2025
This post details a workflow for generating AI videos using the Replicate playground. It outlines a step-by-step process: starting with an image (potentially generated by another Replicate model or fine-tuned), generating and refining the video using models like minimax/video-01-live with prompt optimization, and adding sound with models like zsxkib/mmaudio. The post highlights the use of the playground for quick iteration and experimentation with video models, emphasizing a structured and repeatable approach to AI video creation.
2/7/2023
This post introduces LoRA (Low-Rank Adaptation) as a faster and more efficient method for fine-tuning Stable Diffusion models. It details how LoRA reduces the number of trainable parameters, leading to significantly faster training times (minutes vs. twenty minutes for DreamBooth) and much smaller model outputs (around 5MB vs. several gigabytes). The post explains the process of gathering training images, uploading them, training a LoRA concept using Replicate's `replicate/lora-training` or `replicate/lora-advanced-training` models, and then using the trained LoRA model (`replicate/lora`) to generate new images with custom styles or objects. It highlights the benefits of instant predictions without cold boots and the ability to combine multiple LoRA concepts. The post also contrasts LoRA with DreamBooth, noting its strengths in styles and weaknesses in faces, and provides code examples for training and inference.