Training at half-/mixed-precision? about bmsg-gan HOT 4 OPEN

@BlindElephants,

That is a good suggestion. We could really use faster training. We are more focused on providing the empirical validation for the MSG-GAN technique in this repo. Contributions are most welcome 😄.

Best regards,
@akanimax

from bmsg-gan.

Happy to contribute on this front, as I have a real interest in seeing this happen. Can you comment a bit more about what is and isn't parallelizable in the current architecture and the reasons for those decisions?

I think your notes outlined some particulars that are not currently parallelized.

from bmsg-gan.

Awesome. I'll need to take a deeper look into the code for that. It's night here actually. Will do that tomorrow. Thanks! 👍

Cheers 🍻!
@akanimax

from bmsg-gan.

@BlindElephants

Okay, so the notes about parallelism are regarding the use of the dataParallel technique over mulitple GPUs. this comment is saying that all the computations till the final block of the Discriminator will take place over multiple GPUs but will then bring all the results till then to the first GPU. This is done to handle the operation of the MinBatchStd which requires the whole batch of the data to evaluate the statistic. This could also be parallelized for multi-gpus using the lower level constructs, but for our use-case, the current solution seemed fine.
Basically, if you are running on a single GPU, there are no such issues. PyTorch GPU acceleration is automatically supported.

Hope this info helps.
Please let me know if you need any more info.

Best regards,
@akanimax

from bmsg-gan.