Accepted as [NeurIPS 2024] Spotlight Presentation Paper
APACHE-2.0 License
[Paper] [Project Page] [Jittor Version] [ Comic Generation Demo ]
Official implementation of StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generation.
https://github.com/HVision-NKU/StoryDiffusion/assets/49511209/d5b80f8f-09b0-48cd-8b10-daff46d422af
You can visit here to visit update history.
StoryDiffusion can create a magic story by generating consistent images and videos. Our work mainly has two parts:
Leveraging the images produced through our Consistent Self-Attention mechanism, we can extend the process to create videos by seamlessly transitioning between these images. This can be considered as a two-stage long video generation approach.
Note: results are highly compressed for speed, you can visit our website for the high-quality version.
Combining the two parts, we can generate very long and high-quality AIGC videos.
Video1 | Video2 | Video3 |
---|---|---|
Our Image-to-Video model can generate a video by providing a sequence of user-input condition images.
Video1 | Video2 | Video3 |
---|---|---|
Video4 | Video5 | Video6 |
---|---|---|
Video1 | Video2 | Video3 |
---|---|---|
Video4 | Video5 | Video6 |
---|---|---|
conda create --name storydiffusion python=3.10
conda activate storydiffusion
pip install -U pip
# Install requirements
pip install -r requirements.txt
Currently, we provide two ways for you to generate comics.
You can open the Comic_Generation.ipynb
and run the code.
Run the following command:
(Recommend) We provide a low GPU Memory cost version, it was tested on a machine with 24GB GPU-memory(Tesla A10) and 30GB RAM, and expected to work well with >20 G GPU-memory.
python gradio_app_sdxl_specific_id_low_vram.py
If you have any questions, you are very welcome to email [email protected] and [email protected]
This project strives to impact the domain of AI-driven image and video generation positively. Users are granted the freedom to create images and videos using this tool, but they are expected to comply with local laws and utilize it responsibly. The developers do not assume any responsibility for potential misuse by users.
Following are some third-party implementations of StoryDiffusion.
If you find StoryDiffusion useful for your research and applications, please cite using this BibTeX:
@article{zhou2024storydiffusion,
title={StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generation},
author={Zhou, Yupeng and Zhou, Daquan and Cheng, Ming-Ming and Feng, Jiashi and Hou, Qibin},
journal={NeurIPS 2024},
year={2024}
}