#ICML2022 Experimental codes of "On the Surrogate Gap between Contrastive and Supervised Losses"
MIT License
On the Surrogate Gap between Contrastive and Supervised Losses
pip install -r requirements.txt
git clone [email protected]:NVIDIA/apex.git
cd apex
git checkout 54b93919aadc117cbab1fe5a2af4664bb9842928
pip install -v --disable-pip-version-check --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./
Install an optional library for efficient experiments.
# Install gnu-parallel from source.
wget https://ftp.gnu.org/gnu/parallel/parallel-20210622.tar.bz2
tar -jxvf parallel-20210622.tar.bz2
cd parallel-20210622
./configure --prefix=~/bin
make
make install
code/jobs/toy/circle.sh
.{cifar10,cifar100}/contrastive/seed_*.sh
under code/jobs/vision
directory.code/jobs/language/wiki3029/contrastive.sh
.Note that this repo manages the experimental results by using wandb
. Please replace INPUT_YOUR_ENTITY
with your wandb username in codes that you would like to run.
Please run ./code/notebooks/extract_eval_weights_from_wandb.ipynb
.
{cifar10,cifar100}/mean_eval.sh
, {cifar10,cifar100}/linear_eval.sh
and {cifar10,cifar100}/contrastive_eval.sh
scripts under code/jobs/vision
directory.mean_eval.sh
, linear_eval.sh
and contrastive_eval.sh
scripts under code/jobs/language/wiki3029
directory.Please run the following notebooks to generate plots
scripts/plot_k_msuploss.py
: Figure 3 (a)scripts/plot_k_nceloss.py
: Figure 3 (b)scripts/compare_upper_bound.py
: Figure 4scripts/plot_toy_trajectory.py
: Figure 5scripts/make_toy_figure.py
: Figure 6Note that the scripts for Figures 5 and 6 require the results generated by code/jobs/toy/circle.sh
.
code/notebooks/bound.ipynb
: Figures 1 and 9code/notebooks/ck_heatmap.ipynb
Figures 7 and 8Note:: this codebase tracks experiments using Weights & Biases, but the default wandb might cause hanging at the beginning of training of distributed training code. To avoid this, please set an environment variable as follows:
WANDB_START_METHOD="thread"
@inproceedings{BNN2022,
title = {{On the Surrogate Gap between Contrastive and Supervised Losses}},
author = {Bao, Han and Nagano, Yoshihiro and Nozawa, Kento},
year = {2022},
booktitle = {ICML},
pages = {1585--1606},
}