Educational repository for applying the main video data curation techniques presented in the Stable Video Diffusion paper.
APACHE-2.0 License
Educational repository for applying the main video data curation techniques presented in the Stable Video Diffusion paper. The goal is to provide an interactive resource for the community dig deeper into the curation techniques presented in the paper. As such, it doesn't guarantee precise correctness.
The curation techniques used in this repository are discussed in Appendix C of the paper, in detail. These include:
The notebooks present these techniques on a single video file sourced from the UCF-101 dataset.
Video | CoCa Caption | V-BLIP Caption | Zephyr-7B Caption |
---|---|---|---|
a group of men playing basketball on a court . | the basketball player dribbles to the basket | A basketball player in a group of men dribbles towards the basket on a court. |
Readers are advised to first go through Appendix C of the Stable Video Diffusion paper before referring to the notebooks.
Below are the primary dependencies:
transformers
opencv-python
numpy
ffmpeg
Other dependencies will be detailed below.
Refer to the video_preprocessing_clip_extraction.ipynb
notebook for this. You'd need to install the scenedetect
library from here: https://github.com/Breakthrough/PySceneDetect. This shows both cascaded cut detection and keyframe-aware clipping. At the end of the notebook, you should expect to see different clips extracted from the provided video.
video_preprocessing_captioning.ipynb
presents synthetic captioning from a single video clip.
This uses three models:
open_clip
)transformers
)We had to apply some corrections to eilev
to make it work. The correction patch can be found here.
video_preprocessing_optical_flow_score.ipynb
notebook shows the optical flow score computation only using the Farneback algorithm. It doesn't, however, show RAFT.
This is straightforward and is implemented in the video_preprocessing_similarity_aesthetics.ipynb
notebook.
Refer to the video_preprocessing_text_detection.ipynb
notebook for this. We use a wrapper library called craft_text_detector
(repository) for this as it provides a handy package around the CRAFT text detection model. However, to make it work, we had to do some changes. The patch can be found here.
Thanks to ChatGPT for all the help.
Thanks to Dhruv Nair for his reviews.