TrailBlazer

This repository contains the implementation of the following paper: > TrailBlazer: Trajectory Control for Diffusion-Based Video Generation
> Wan-Duo Kurt Ma¹, J.P. Lewis², W. Bastiaan Kleijn¹,
Victoria University of Wellington¹, NVIDIA Research²

🔥 Overview

TrailBlazer focuses on enhancing controllability in video synthesis by employing straightforward bounding boxes to guide the subject in various ways, all without the need for neural network training, finetuning, optimization at inference time, or the use of pre-existing videos. Our algorithm is constructed upon a pre-trained (T2V) model, and easy to implement. The subject is directed by a bounding box through the proposed spatial and temporal attention map editing. Moreover, we introduce the concept of keyframing, allowing the subject trajectory and overall appearance to be guided by both a moving bounding box and corresponding prompts, without the need to provide a detailed mask. The method is efficient, with negligible additional computation relative to the underlying pre-trained model. Despite the simplicity of the bounding box guidance, the resulting motion is surprisingly natural, with emergent effects including perspective and movement toward the virtual camera as the box size increases.

🔥 Requirements

The codebase is tested under NVIDIA GeForce RTX 3090 with the python library pytorch-2.1.2+cu121 and diffusers-0.21.4. We strongly recommend using a specific version of Diffusers as it is continuously evolving. For PyTorch, you could probably use other version under 2.x.x. With RTX 3090, I follow the post to avoid the compatibility of sm_86 issue.

🔥 Timeline

• [2024/04/08]: Our new v2 preprint is now appeared on ArXiv (See link)

• [2024/03/23]: A new ArXiv update will be made.

• [2024/03/22]: We release the multiple object synthesis (See link), and the Peekaboo integration (See link)

• [2024/02/07]: The Gradio app is updated with better keyframe interface (See (link))

• [2024/02/06]: We now have Gradio web app at Huggingface Space!

• [2024/02/01]: The official codebase released

• [2024/01/03]: Paper released

• [2023/12/31]: Paper submitted on ArXiv

🔥 Usage

[Prepare]

First of all, download the pre-trained zeroscope model (link). You need to register huggingface and make access token (link)

git clone https://huggingface.co/cerspense/zeroscope_v2_576w ${MODEL_ROOT}/cerspense/zeroscope_v2_576w

where MODEL_ROOT is your preference that stores the model. Then, clone this Repo and cd into it:

git clone https://github.com/hohonu-vicml/Trailblazer && cd Trailbalzer

[Run it]

Our executable script is located in the "bin" folder, and the core module is implemented in the "TrailBlazer" folder under the project root. Therefore, no additional dependencies need to be added to PYTHONPATH; you can simply run the command below 😏 :

python bin/CmdTrailBlazer.py -mr ${MODEL_ROOT} --config config/XXXX.yaml  ## single experiment
python bin/CmdTrailBlazer.py -mr ${MODEL_ROOT} --config config/  ## run all yamls in a folder

💘UPDATE💘: TrailBlazer has just released Gradio app for the alternative interface. Please checkout our documentation (Gradio.md) for more information. To run the app, simply run:

python bin/CmdGradio.py ${MODEL_ROOT} # no -mr here

When the shell environment variable ZEROSCOPE_MODEL_ROOT is specified, then you can ignore the -mr (--model-root) argument above.

export ZEROSCOPE_MODEL_ROOT=/path/to/your/diffusion/root
# then you can ignore -mr term to simplify the command
python bin/CmdTrailBlazer.py --config config/XXXX.yaml

Please see here for more information about the command set used in TrailBlazer.

[Config]

A list of config example files is stored in the config folder. Feel free to run each of them and the result will be written in the /tmp folder. For more information how to design the config file, and the visual result of each config. Please visit here and there for more details about config structure and the visual result, respectively.

🔥 Contribution

This project is still working in progress, and there are numerous directions in which it can be improved. Please don't hesitate to contact us if you are interested, or feel free to make a pull request to strengthen the ideas.

🔥 TODO

We regret to inform you that this repository is currently not fully accessible to the public. Nevertheless, the majority of the core modules have been made available (e.g., Single, Multiple objects synthesis, and Peekaboo comparison). Our next release will include useful tools for measuring metrics.

🔥 Fun

Poor cat: Someone, Stop me!

Am I a cat, or a dog...

Please inform us if you have generated any interesting videos!

🔥 Citation

TrailBlazer is built on top of its mother project DirectedDiffusion, which recently published at AAAI2024. If you find our work useful for your research, please consider citing our paper.

@article{ma2023trailblazer,
    title={TrailBlazer: Trajectory Control for Diffusion-Based Video Generation},
    author={Wan-Duo Kurt Ma and J. P. Lewis and W. Bastiaan Kleijn},
    year={2023},
    eprint={2401.00896},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}

@article{ma2023directed,
    title={Directed Diffusion: Direct Control of Object Placement through Attention Guidance},
    author={Wan-Duo Kurt Ma and J. P. Lewis and Avisek Lahiri and Thomas Leung and W. Bastiaan Kleijn},
    year={2023},
    eprint={2302.13153},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}