CAIN, Channel Attention Is All You Need for Video Frame Interpolation implemented with ncnn library
MIT License
ncnn implementation of CAIN, Channel Attention Is All You Need for Video Frame Interpolation.
cain-ncnn-vulkan uses ncnn project as the universal neural network inference framework.
Download Windows/Linux/MacOS Executable for Intel/AMD/Nvidia GPU
https://github.com/nihui/cain-ncnn-vulkan/releases
This package includes all the binaries and models required. It is portable, so no CUDA or PyTorch runtime environment is needed :)
CAIN (Channel Attention Is All You Need for Video Frame Interpolation) (AAAI 2020)
https://github.com/myungsub/CAIN
Myungsub Choi, Heewon Kim, Bohyung Han, Ning Xu, Kyoung Mu Lee
2nd place in [AIM 2019 ICCV Workshop] - Video Temporal Super-Resolution Challenge
Project | Paper-AAAI (Download the paper [here] in case the AAAI link is broken) | Poster
Input two frame images, output one interpolated frame image.
./cain-ncnn-vulkan -0 0.jpg -1 1.jpg -o 01.jpg
./cain-ncnn-vulkan -i input_frames/ -o output_frames/
mkdir input_frames
mkdir output_frames
# find the source fps and format with ffprobe, for example 24fps, AAC
ffprobe input.mp4
# extract audio
ffmpeg -i input.mp4 -vn -acodec copy audio.m4a
# decode all frames
ffmpeg -i input.mp4 input_frames/frame_%06d.png
# interpolate 2x frame count
./cain-ncnn-vulkan -i input_frames -o output_frames
# encode interpolated frames in 48fps with audio
ffmpeg -framerate 48 -i output_frames/%06d.png -i audio.m4a -c:a copy -crf 20 -c:v libx264 -pix_fmt yuv420p output.mp4
Usage: cain-ncnn-vulkan -0 infile -1 infile1 -o outfile [options]...
cain-ncnn-vulkan -i indir -o outdir [options]...
-h show this help
-v verbose output
-0 input0-path input image0 path (jpg/png/webp)
-1 input1-path input image1 path (jpg/png/webp)
-i input-path input image directory (jpg/png/webp)
-o output-path output image path (jpg/png/webp) or directory
-m model-path cain model path (default=cain)
-g gpu-id gpu device to use (default=auto) can be 0,1,2 for multi-gpu
-j load:proc:save thread count for load/proc/save (default=1:2:2) can be 1:2,2,2:2 for multi-gpu
-f pattern-format output image filename pattern format (%08d.jpg/png/webp, default=ext/%08d.png)
input0-path
, input1-path
and output-path
accept file pathinput-path
and output-path
accept file directoryload:proc:save
= thread count for the three stages (image decoding + cain interpolation + image encoding), using larger values may increase GPU usage and consume more GPU memory. You can tune this configuration with "4:4:4" for many small-size images, and "2:2:2" for large-size images. The default setting usually works fine for most situations. If you find that your GPU is hungry, try increasing thread count to achieve faster processing.pattern-format
= the filename pattern and format of the image to be output, png is better supported, however webp generally yields smaller file sizes, both are losslessly encodedIf you encounter a crash or error, try upgrading your GPU driver:
dnf install vulkan-headers vulkan-loader-devel
apt-get install libvulkan-dev
pacman -S vulkan-headers vulkan-icd-loader
git clone https://github.com/nihui/cain-ncnn-vulkan.git
cd cain-ncnn-vulkan
git submodule update --init --recursive
mkdir build
cd build
cmake ../src
cmake --build . -j 4
cain-ncnn-vulkan.exe -0 0.png -1 1.png -o out.png