Implement GANs with PyTorch.

Unconditional image generation (CIFAR-10):

• DCGAN (vanilla GAN)
• DCGAN + R1 regularization
• WGAN
• WGAN-GP
• SNGAN
• LSGAN

Conditional image generation (CIFAR-10):

• CGAN
• ACGAN

Unsupervised decomposition (MNIST, FFHQ):

• InfoGAN
• EigenGAN

Mode collapse study (Ring8, MNIST):

• GAN (vanilla GAN)
• GAN + R1 regularization
• WGAN
• WGAN-GP
• SNGAN
• LSGAN
• VEEGAN

Notes:

• SN stands for "Spectral Normalization".

• For simplicity, the network architecture in all experiments is SimpleCNN, namely a stack of nn.Conv2d or nn.ConvTranspose2d layers. The results can be improved by adding more parameters and using advanced architectures (e.g., residual connections), but I decide to use the simplest setup here.

• All models except LSGAN are trained for 40k generator update steps. However, the optimizers and learning rates are not optimized for each model, so some models may not reach their optimal performance.

Quantitative results:

• The FID is calculated between 50k generated samples and the CIFAR-10 training split (50k images).
• The Inception Score is calculated on 50k generated samples.

Visualization:

DCGAN	DCGAN + R1 reg	WGAN	WGAN-GP
SNGAN (vanilla loss)	SNGAN (hinge loss)	LSGAN

Notes:

• cBN stands for "conditional Batch Normalization"; SN stands for "Spectral Normalization"; AC stands for "Auxiliary Classifier"; PD stands for "Projection Discriminator".

Quantitative results:

• The FID is calculated between 50k generated samples (5k for each class) and the CIFAR-10 training split (50k images).
• The intra FID is calculated between 5k generated samples and CIFAR-10 training split within each class.
• The Inception Score is calculated on 50k generated samples.

Visualizations:

CGAN	CGAN (cBN)	ACGAN

InfoGAN

• Left: change the discrete latent variable, which corresponds to the digit type.
• Right: change one of the continuous latent variable from -1 to 1. However, the decomposition is not clear.
• Note: I found that batch normalization layers play an important role in InfoGAN. Without BN layers, the discrete latent variable tends to have a clear meaning as shown above, while the continuous variables have little effect. On the contrary, with BN layers, it's harder for the discrete variable to catch the digit type information and easier for continuous ones to find rotation in digits.

EigenGAN

Random samples (no truncation):

Traverse:

Mode collapse is a notorious problem in GANs, where the model can only generate a few modes of the real data. Various methods have been proposed to solve it. To study this problem, I experimented different methods on the following two datasets:

• Ring8: eight gaussian distributions lying on a ring.
• MNIST: handwritten digit dataset.

For simplicity, the model architecture in all experiments is SimpleMLP, namely a stack of nn.Linear layers, thus the quality of generated MNIST image may not be so good. However, this section aims to demonstrate the mode collapse problem rather than to achieve the best image quality.

GAN

200 steps	400 steps	600 steps	800 steps	1000 steps

1000 steps	2000 steps	3000 steps	4000 steps	5000 steps

On the Ring8 dataset, it can be clearly seen that all the generated data gather to only one of the 8 modes.

In the MNIST case, the generated images eventually collapse to 1.

GAN + R1 regularization

200 steps	400 steps	600 steps	800 steps	5000 steps

1000 steps	3000 steps	5000 steps	7000 steps	9000 steps

R1 regularization, a technique to stabilize the training process of GANs, can prevent mode collapse in vanilla GAN as well.

WGAN

200 steps	400 steps	600 steps	800 steps	5000 steps

1000 steps	3000 steps	5000 steps	7000 steps	9000 steps

WGAN indeed resolves the mode collapse problem, but converges much slower due to weight clipping.

WGAN-GP

200 steps	400 steps	600 steps	800 steps	5000 steps

1000 steps	3000 steps	5000 steps	7000 steps	9000 steps

WGAN-GP improves WGAN by replacing the hard weight clipping with the soft gradient penalty.

The pathological weights distribution in WGAN's discriminator does not appear in WGAN-GP, as shown below.

SNGAN

200 steps	400 steps	600 steps	800 steps	5000 steps

1000 steps	3000 steps	5000 steps	7000 steps	9000 steps

Note: The above SNGAN is trained with the vanilla GAN loss instead of the hinge loss.

SNGAN uses spectral normalization to control the Lipschitz constant of the discriminator. Even with the vanilla GAN loss, SNGAN can avoid mode collapse problem.

LSGAN

200 steps	400 steps	600 steps	800 steps	5000 steps

1000 steps	3000 steps	5000 steps	7000 steps	9000 steps

LSGAN uses MSE instead of Cross-Entropy as the loss function to overcome the vanishing gradients in vanilla GAN. However, it still suffers from the mode collapse problem. For example, as shown above, LSGAN fails to cover all 8 modes on the Ring8 dataset.

Note: Contrary to the claim in the paper, I found that LSGAN w/o batch normalization does not converge on MNIST.

VEEGAN

200 steps	400 steps	600 steps	800 steps	5000 steps

1000 steps	3000 steps	5000 steps	7000 steps	10000 steps

VEEGAN uses an extra network to reconstruct the latent codes from the generated data.

The checkpoints and training logs are stored in xyfJASON/GANs-Implementations on huggingface.

For GAN, WGAN-GP, SNGAN, LSGAN:

accelerate-launch scripts/train.py -c ./configs/xxx.yaml

For WGAN (weight clipping), InfoGAN, VEEGAN, CGAN, ACGAN and EigenGAN, use the scripts with corresponding name instead:

accelerate-launch scripts/train_xxxgan.py -c ./configs/xxx.yaml

Unconditional GANs:

accelerate-launch scripts/sample.py \
    -c ./configs/xxx.yaml \
    --weights /path/to/saved/ckpt/model.pt \
    --n_samples N_SAMPLES \
    --save_dir SAVE_DIR

Conditional GANs:

accelerate-launch scripts/sample_cond.py \
    -c ./configs/xxx.yaml \
    --weights /path/to/saved/ckpt/model.pt \
    --n_classes N_CLASSES \
    --n_samples_per_class N_SAMPLES_PER_CLASS \
    --save_dir SAVE_DIR

EigenGAN:

accelerate-launch scripts/sample_eigengan.py \
    -c ./configs/xxx.yaml \
    --weights /path/to/saved/ckpt/model.pt \
    --n_samples N_SAMPLES \
    --save_dir SAVE_DIR \
    --mode MODE

Sample images following the instructions above and use tools like torch-fidelity to calculate FID / IS.