[ICCV2023 oral] Zolly: Zoom Focal Length Correctly for Perspective-Distorted Human Mesh Reconstruction
The first work aims to solve 3D Human Mesh Reconstruction task in perspective-distorted images.
๐ 2024.Jul.18, pretrained models are released: https://huggingface.co/WenjiaWang/Zolly_ckpts, most are better than the results in paper.
๐ 2023.Nov.23, the training code of Zolly is released, pretrained Zolly weight will come soon.
๐ 2023.Aug.12, Zolly is selected as ICCV2023 oral, project page.
๐ 2023.Aug.7, the dataset link is released. The training code is coming soon.
๐ 2023.Jul.14, Zolly is accepted to ICCV2023, codes and data will come soon.
๐ 2023.Mar.27, arxiv link is released.
MMHuman3D
firstly.You should install the needed relies as ffmpeg
, torch
, mmcv
, pytorch3d
following its tutorials.
MMHuman3D
:wget https://github.com/open-mmlab/mmhuman3d/archive/refs/tags/v0.9.0.tar.gz;
tar -xvf v0.9.0.tar.gz;
cd mmhuman3d-0.9.0;
pip install -e .
You can install pytorch3d
from file if you find any difficulty.
E.g. python3.8 + pytorch-1.13.1 + cuda-11.7 + pytorch3d-0.7.4
wget https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch3d/linux-64/pytorch3d-0.7.4-py38_cu117_pyt1131.tar.bz2;
pip install fvcore;
pip install iopath;
conda install --use-local pytorch3d-0.7.4-py38_cu117_pyt1131.tar.bz2;
cd Zolly;
pip install -e .
You can download the files from onedrive; or from huggingface by command huggingface-cli download WenjiaWang/Zolly_release --local-dir Zolly_release --repo-type dataset
.
This link contains:
Dataset annotations: all have ground-truth focal length, translation and smpl parameters.
Dataset images.
Pretrained backbone
hrnetw48_coco_pose.pth
resnet50_coco_pose.pth
Others
smpl_uv_decomr.npz
mesh_downsampling.npz
J_regressor_h36m.npy
SMPL skinning weights
root
โโโ body_models
โย ย โโโ smpl
| โโโ J_regressor_extra.npy
| โโโ J_regressor_h36m.npy
| โโโ mesh_downsampling.npz
| โโโ SMPL_FEMALE.pkl
| โโโ SMPL_MALE.pkl
| โโโ smpl_mean_params.npz
| โโโ SMPL_NEUTRAL.pkl
| โโโ smpl_uv_decomr.npz
โโโ cache
โโโ mmhuman_data
โย ย โโโ datasets
| โ โโโ coco
| โ โโโ h36m
| โ โโโ humman
| โ โโโ lspet
| โ โโโ mpii
| โ โโโ mpi_inf_3dhp
| โ โโโ pdhuman
| โ โโโ pw3d
| โ โโโ spec_mtp
โย ย โโโ preprocessed_datasets
| โโโ humman_test_p1.npz
| โโโ humman_train.npz
| โโโ pdhuman_test_p1.npz
| โโโ pdhuman_train.npz
| โโโ pw3d_train.npz
| โโโ pw3d_train_transl.npz
| โโโ spec_mtp.npz
| โโโ spec_mtp_p1.npz
โโโ pretrain
โโโ coco_pretrain
ย ย โโโ hrnetw48_coco_pose.pth
ย ย โโโ resnet50_coco_pose.pth
And change the root
in zolly/configs/base.py
sh train_bash.sh zolly/configs/zolly_r50.py $num_gpu$ --work-dir=$your_workdir$
E.g, you can use
sh train_bash.sh zolly/configs/zolly_r50.py 8 --work-dir=work_dirs/zolly
To resume training or finetune model:
sh train_bash.sh zolly/configs/zolly_r50.py 8 --work-dir=work_dirs/zolly --resume-from work_dirs/zolly/latest.pth
sh test_bash.sh zolly/configs/zolly/zolly_r50.py $num_gpu$ --checkpoint=$your_ckpt$ --data-name pw3d
For convenience, you can test the first 100 samples to evaluate your model.
sh test_bash.sh zolly/configs/zolly/zolly_r50.py $num_gpu$ --checkpoint=$your_ckpt$ --data-name pw3d --num-data 100
sh demo_bash.sh zolly/configs/zolly/zolly_h48.py $num_gpu$ --checkpoint=$your_ckpt$ --image_folder assets/demo_jpg --ext jpg --demo_root demo/
The output name will be like 56_789-0.00_586-1.91_pred.png
, which represent {raw_name}_{gt_f}-{gt_z}_{pred_f}-{pred_z}_pred.png
We have released our R50 and H48 model on huggingface: https://huggingface.co/WenjiaWang/Zolly_ckpts
You can use huggingface-cli download WenjiaWang/Zolly_ckpts --local-dir ckpts --repo-type model
to download the model. (Remember to login with you token firstly)
Method | PA-MPJPE | MPJPE | PVE |
---|---|---|---|
Zolly-R50 | 48.92๐ | 79.18๐ | 92.82 |
Zolly-R50 (ft) | 43.70๐ | 71.33๐ | 84.41 |
Zolly-H48 | 47.88๐ | 78.21 | 90.82 |
Zolly-H48(ft) | 39.09๐ | 64.44๐ | 75.78๐ |
Method | PA-MPJPE | MPJPE | PVE |
---|---|---|---|
Zolly-R50 | 75.34 | 126.66 | 140.69 |
Zolly-H48 | 67.47 | 115.74 | 127.96 |
Method | PA-MPJPE | MPJPE | PVE |
---|---|---|---|
Zolly-R50 | 24.57 | 35.88๐ | 43.49๐ |
Zolly-H48 | 22.94 | 33.39 | 37.93๐ |
Method | PA-MPJPE | MPJPE | PVE |
---|---|---|---|
Zolly-R50 | 56.75 | 79.83๐ | 91.93๐ |
Zolly-H48 | 46.53๐ | 67.86๐ | 77.77๐ |
zolly/models/heads
, and add it to zolly/models/builder.py
.zolly/models/architectures
, and add it to zolly/models/architectures/builder.py
.zolly/models/losses
, and add it to zolly/models/losses/builder.py
.zolly/configs/
, you can modify from zolly/configs/zolly_r50.py
. And remember to change the root
parameter in zolly/configs/base.py
, where your files should be put.If you find this project useful in your research, please consider citing us:
@inproceedings{wangzolly,
title={Zolly: Zoom Focal Length Correctly for Perspective-Distorted Human Mesh Reconstruction Supplementary Material},
author={Wang, Wenjia and Ge, Yongtao and Mei, Haiyi and Cai, Zhongang and Sun, Qingping and Wang, Yanjun and Shen, Chunhua and Yang, Lei and Komura, Taku},
booktitle={Proceedings of the IEEE International Conference on Computer Vision (ICCV)},
year={2023}
}
Emojis are collected from gist:7360908.
Some of the codes are based on MMHuman3D
, DecoMR
.
Feel free to contact me for other questions or cooperation: [email protected]