Automate VFX Sky Replacement with an Animated Atmosphere Using a Single ComfyUI Workflow
A VFX artist built a repeatable workflow in ComfyUI that replaces static skies with generated, animated atmospheres, demonstrating a practical AI application for reducing video production time and cost.
Practical Summary
This workflow takes video footage, decodes frames, and uses an image model to rewrite the sky (e.g., changing daytime to sunset or a storm). It then automatically generates a video prompt using Qwen-VL and uses Wan 2.2 with ControlNet to add realistic motion like clouds and lighting shifts. The final step reconstructs the video with original metadata. An 'App Mode' is also included for a simplified upload-and-generate process.
Why It Matters
For video production teams and freelancers, this workflow automates a traditionally time-consuming and skilled VFX task. By using AI to generate and animate sky replacements, it can significantly reduce the hours (and therefore cost) required for such shots, while maintaining quality through a locked-to-shot process. This is a clear example of an AI cost-saving workflow in a creative professional context.
How the ComfyUI Sky Replacement Workflow Works
This end-to-end workflow is designed to take raw video footage and output a new version with an AI-generated, animated sky. It automates multiple steps that would normally require manual VFX work, from prompt generation to motion synthesis.
Step 1: Footage Ingestion and Frame Decoding
The process begins by taking your input video file. The workflow automatically decodes this footage into individual frames, preparing them for the AI image model to process.
Step 2: AI-Powered Sky Rewriting
An image model analyzes each frame and rewrites the sky region. You can direct this transformation—changing a daytime sky to a sunset, adding a storm, or applying a stylized look. This generates a new static frame with the desired sky.
Step 3: Automatic Video Prompt Generation
To ensure the generated sky looks consistent and high-quality when animated, the workflow uses Qwen-VL. This AI model automatically writes a descriptive video prompt based on the newly generated frame, capturing the scene's details for the next step.
Step 4: Adding Realistic Motion
This is the core animation step. Using the generated prompt and frame, the Wan 2.2 model, guided by ControlNet, synthesizes motion. This adds dynamic elements like drifting clouds, shifting lighting, and atmospheric changes that are physically consistent with the shot.