Video Style Transfer with LoRAs

7 posts · Feb 2023 – Jul 2025

What this trail covers

Replicate continues to expand its support for fine-tuning open-source video models, specifically adapting LoRA-based style transfer for models like HunyuanVideo. This allows users to train custom video generation models on their own visual content, enabling personalized video stylization with in-motion style transfer. The process involves using dedicated training models and then generating videos via the browser, Playground, or API, with optimizations for faster inference. This post highlights new capabilities for fine-tuning HunyuanVideo using the Musubi Tuner, enabling in-motion style transfer and character animation.

All Replicate trails

Jul 2025Open source video is backThis post details the release and optimization of Wan 2.2, an open-source video generation model. It highlights significant improvements in generation speed (~30s for 480p), cost reduction ($0.05 per video), and quality (sharp physics, more control). It introduces optimized image-to-video and text-to-video endpoints at 480p and 720p resolutions. The post also includes API examples for using the Wan 2.2 models via Replicate's platform.
May 2025Run 30,000+ LoRAs on Hugging Face with ReplicateThis post details the integration of LoRAs for image generation models with Hugging Face. It explains how Hugging Face's inference client can route requests to Replicate's backend, specifically using the `black-forest-labs/flux-dev-lora` model. The LoRAs are passed as dynamic input via the `lora_weights` parameter, allowing a single backend model to support numerous LoRAs without individual hosting. This enables fast, low-cost inference directly from the Hugging Face interface for over 30,000 LoRAs.
Apr 2025Stylized video with Wan2.1This post details the application of premade LoRA weights to the Wan 2.1 video generation model, enabling users to achieve stylized video output with minimal setup by providing a URL to arbitrary safetensors weights. It also outlines a new workflow for training custom LoRA weights using the `ostris/wan-lora-trainer` model, which involves preparing training images, setting a trigger word, and utilizing autocaptioning for style guidance. The post provides Python client examples for applying LoRAs and lists several premade styles.
Mar 2025Wan2.1 parameter sweepThis post details an experiment exploring the impact of `guidance scale` and `shift` parameters on the Wan 2.1 text-to-video model. It provides specific ranges and observations for each parameter, recommending optimal values for different output goals (creativity vs. realism) and analyzing the effects on visual artifacts, subject appearance, and motion flow. The post also shares the GitHub code used for the parameter sweep.
Jan 2025You can now fine-tune open-source video modelsThis post details the process of fine-tuning the HunyuanVideo model on Replicate using the Musubi Tuner. It outlines steps for data preparation using a new `zsxkib/create-video-dataset` model, training the fine-tuned model via the Replicate UI, and generating videos using the API. It also provides experimentation tips for training settings and an example of programmatic model training and generation using the Replicate Python client.
Advertisement
Jan 2025Generate short videos with the Replicate playgroundThis post details a workflow for generating AI videos using the Replicate playground. It outlines a step-by-step process: starting with an image (potentially generated by another Replicate model or fine-tuned), generating and refining the video using models like minimax/video-01-live with prompt optimization, and adding sound with models like zsxkib/mmaudio. The post highlights the use of the playground for quick iteration and experimentation with video models, emphasizing a structured and repeatable approach to AI video creation.
Feb 2023Introducing LoRA: A faster way to fine-tune Stable DiffusionThis post introduces LoRA (Low-Rank Adaptation) as a faster and more efficient method for fine-tuning Stable Diffusion models. It details how LoRA reduces the number of trainable parameters, leading to significantly faster training times (minutes vs. twenty minutes for DreamBooth) and much smaller model outputs (around 5MB vs. several gigabytes). The post explains the process of gathering training images, uploading them, training a LoRA concept using Replicate's `replicate/lora-training` or `replicate/lora-advanced-training` models, and then using the trained LoRA model (`replicate/lora`) to generate new images with custom styles or objects. It highlights the benefits of instant predictions without cold boots and the ability to combine multiple LoRA concepts. The post also contrasts LoRA with DreamBooth, noting its strengths in styles and weaknesses in faces, and provides code examples for training and inference.