Batches per epoch / training on half batches about keras-gan HOT 3 CLOSED

1. Epoch might be a slightly misleading variable name honestly. That's something I should change. For each iteration I randomly sample images with uniform probability, so when the number of iterations rises to a sufficiently large number the model will have seen each sample. Therefore I didn't see a reason to divide the training into epochs and batch iterations. MNIST consists of 60k samples, so iterating through the whole dataset during one epoch would take a lot of time. And the models I have implemented that are modelling MNIST converge relatively quickly, so therefore I randomly sample instead.

2. When BatchNormalization2d is used it's recommended to split the data into real and fake batches before feeding them to the discriminator. It's been a while since I read up on why but since batch norm keeps a running average of batch means and variances I guess mixing batches with data from two distributions messes with these running averages. There are a lot of tricks listed in this great repo: https://github.com/soumith/ganhacks. Dividing real and fake batches when batch norm is used is one trick listed there.

from keras-gan.

Thanks so much for the help Erik, super cool.

from keras-gan.

First, thank you for sharing this code, @eriklindernoren!

Regarding the misleading use of the word epoch, it is indeed confusing for newcomers. I guess people who run the code in your repository come from the Keras documentation, so they have this vocabulary in mind:

Epoch: an arbitrary cutoff, generally defined as "one pass over the entire dataset", used to separate training into distinct phases, which is useful for logging and periodic evaluation.

Batch: a set of N samples. The samples in a batch are processed independently, in parallel. If training, a batch results in only one update to the model.

Now, it makes more sense that the number of so-called epochs in your code is so large! And if I were to apply your code to another dataset of 30k samples, I might want to lower the value of the so-called epochs to account for the lower number of samples.

from keras-gan.