Official code repository for the paper Integrating Semantics into Domain Translation without Supervision.
Dependencies are provided in requirement.txt
. A Dockerfile is provided for reproducing the same environment
used to run the experiments.
The models are in src/models
. Every model has 3 files:
__init__.py
: Defines the specific parameters of the models
model.py
Defines the architecture of the model
train.py
Defines the training algorithm of the model
and a folder evaluate
containing the evaluation scripts.
In general, a model can be trained by invoking main.py
, which also contains the general parameters shared among all the
models. The syntax for training a model is as follows:
python src/main.py [GENERAL PARAMETERS] [MODEL NAME] [SPECIFIC MODEL PARAMETERS]
Some models can be evaluated by using one of it's evaluation script in the folder evaluate
.
The syntax for evaluating a model is a follows:
python src/evaluate.py [MODEL NAME]-[EVALUATE SCRIPT NAME] [SPECIFIC EVALUATIONS PARAMETERS]
We presented results on the MNIST-SVHN dataset which can be themselves downloaded by the torchvision library. We also presented results on the Sketch->Real dataset which are a subset of the DomainNet dataset (https://arxiv.org/abs/1812.01754). The sketch dataset can be downloaded as follow:
./download_data.sh sketch
Similarly, the real dataset can be downloaded as follows:
./download_data.sh real
This repository is composed of the models that are useful for reproducing the results from the paper. We now explain how to train them.
This model is used for evaluating the translation MNIST<->SVHN. The classifier is a wide residual network (https://arxiv.org/abs/1605.07146) and the is code inspired from: https://github.com/szagoruyko/wide-residual-networks.
Classifying MNIST
python src/main.py --exp-name classifier --cuda --run-id mnist --train-batch-size 128 --valid-split 0.2 classifier --dataset mnist
Classifying SVHN
python src/main.py --exp-name classifier --cuda --run-id svhn --train-batch-size 128 --valid-split 0.2 classifier --dataset svhn_extra
Imsat (https://arxiv.org/abs/1605.07146) is a method for clustering using deep neural networks. In this work, we use it for clustering MNIST. We use a slighly different version than the one proposed in the original model. We explain the the differences in the appendix on the paper. But, the original imsat algorithm, or any other cluster algorithm which can cluster MNIST decently well could be used in place.
Clustering MNIST
python src/main.py --exp-name cluster --cuda --run-id mnist imsat
VMT (https://arxiv.org/abs/1905.04215) is an unsupervised domain adaptation method. In this work, we proposed to replace
the ground truth labels of the source domain with the learned cluster on the source domain. vmt_cluster
is a model
that applies the vmt method on images by considering the learned cluster rather than the ground truth labels.
Domain adaptation with clustering MNIST-SVHN
python src/main.py --exp-name vmt-cluster --cuda --run-id mnist-svhn vmt_cluster --dataset1 mnist --dataset2 svhn --cluster-model-path ./experiments/imsat/cluster_mnist-None --dw 0.01 --svw 1 --tvw 0.06 --tcw 0.06 --smw 1 --tmw 0.06
We also propose to apply VMT cluster on the representation learned by a representation learning model using the clusters learned themselves on the representation. In this work, we used the pre-trained MoCO-v2 (https://arxiv.org/abs/2003.04297) model, which can be downloaded as follow
wget https://dl.fbaipublicfiles.com/moco/moco_checkpoints/moco_v2_800ep/moco_v2_800ep_pretrain.pth.tar
We found that using this method for learning cross-domain clustering on sketch-real yielded better results. Learning the model can be done as follows:
Domain adaptation with clustering on Sketch-Real
python src/main.py --exp-name vmtc-repr --cuda --run-id sketch-real vmtc_repr --ss-path moco_v2_800ep_pretrain.pth.tar
We propose to use the learned semantics in a domain translation framework. In this work, we modified StarGAN-V2. The same framework can be used for both MNIST<->SVHN and Sketches->Reals.
Domain translation MNIST<->SVHN
python src/main.py --exp-name sg-sem --run-id mnist-svhn --cuda --train-batch-size 8 sg_sem --num_classes 10 --img_size 32 --dataset_loc data --dataset cond_mnist_svhn --cluster_type vmt_cluster --cluster_path experiments/vmt_cluster/vmt-cluster_mnist-svhn-None/
Domain translation Sketches->Reals
python src/main.py --exp-name sg_sem --run-id sketch-real --cuda --train-batch-size 8 sg_sem --dataset_loc data --dataset cond_visda --lambda_sty 0 --ss_path moco_v2_800ep_pretrain.pth.tar --cluster_path experiments/vmtc_repr/vmtc-repr_sketch-real-None/
To create the MNIST->SVHN grids, simply run the following command:
python src/evaluate.py sg_sem-fetch_results_ms --state-dict-path experiments/sg-sem_mnist-svhn-None/model/nets_ema:100000.ckpt --data-root-src data --dataset-src svhn --domain 0 --da-path ./experiments/vmt_cluster/vmt-cluster_mnist-svhn-None/ --save-name MNIST-SVHN
To create the Sketches->Reals grid, simply run the following command:
python src/evaluate.py sg_sem-fetch_results_sr --state-dict-path experiments/sg-sem_sketch-real-None/model/nets_ema:100000.ckpt --data-root-src data/test_all/sketch --domain 0 --ss-path moco_v2_800ep_pretrain.pth.tar --da-path experiments/vmtc_repr/vmtc-repr_sketch-real-None/ --save-name Sketch-Real --model-path experiments/sg-sem_sketch-real-None
To fetch results, use the script fetch_results.py
, provided. Similarly, one can compute the FID using the script compute_fid.py
.
Sketch->Real
MNIST->SVHN