Documentation menu

Text to Video

Generate short MP4, WebM or GIF clips from a text prompt in PHP with local diffusers video models like Wan2.1 and CogVideoX.

Generate short video clips from a text prompt with a local diffusers video model. The output format follows the file extension: MP4, WebM, MKV or GIF.

Setup#

Terminal 
vendor/bin/loves-ai setup text-to-video
vendor/bin/loves-ai pull Wan-AI/Wan2.1-T2V-1.3B-Diffusers

Supported models#

Diffusers video pipelines, e.g. Wan-AI/Wan2.1-T2V-1.3B-Diffusers, zai-org/CogVideoX-2b or an AnimateDiff pipeline.

Video models are the heaviest task. They are several gigabytes to pull and need a lot of memory, and a few seconds of video take minutes on a GPU and up to hours on a CPU. Start with the smallest model, few frames and a small frame size, and always generate in a queue job.

From the command line#

Terminal 
vendor/bin/loves-ai text-to-video Wan-AI/Wan2.1-T2V-1.3B-Diffusers "a cat walking through tall grass" --frames=33 --output=cat.mp4
Output 
🎬 Rolling the camera with Wan-AI/Wan2.1-T2V-1.3B-Diffusers… Films take their time — perfect for a pot of tea and some cookies 🍪
If you wish to see all logs, re-run the command with the "--debug" option.
🎉 Video saved to /var/www/my-app/cat.mp4

Options#

Option Meaning
--output=PATH Video file to write; its extension picks the format: .mp4, .webm, .mkv, .gif (default: a timestamped .mp4 in output_dir)
--negative-prompt=TEXT What the video should not contain
--frames=N Number of frames to generate (default: the pipeline's own)
--fps=N Frames per second of the written file (default: 8)
--steps=N Inference steps (default: the pipeline's own)
--guidance=SCALE Guidance scale (default: the pipeline's own)
--width=PX, --height=PX Frame size (default: the pipeline's own)
--seed=N Random seed, for reproducible videos
--device=DEVICE cpu, cuda, mps… (default: the best available)
--log-file=PATH Append the runner's output to this file
--debug Show the runner's output while filming

Defaults come from config/text-to-video.php (output_dir, log_file).

From PHP#

PHP 
use PhpLovesAi\Runner\TextToVideo;

// Finds the runner and the pulled model in the project's .local directory by itself.
$clip = (new TextToVideo())->generate(
    model: 'Wan-AI/Wan2.1-T2V-1.3B-Diffusers',
    prompt: 'a cat walking through tall grass',
    outputPath: storage_path('app/cat.mp4'),
    frames: 33,
    fps: 16,
);

Errors#

  • BinaryNotInstalledException when setup text-to-video has not been run
  • ModelNotFoundException when the model was not pulled yet
  • UnsupportedModelException when the model is not a diffusers video pipeline (a still-image pipeline says so and points to text to image)
  • RunFailedException, with the runner's error output, when generation fails, e.g. for an output format it cannot write or when memory runs out