Divergence of CD/PCD about normalized-autoencoders HOT 2 CLOSED

Dear @GloryyrolG , Sorry for the late update.

From my experience, NAE for image data could not be successfully trained using CD or PCD as in IGEBM.
My guess is that the energy landscape of NAE is significantly more rigged than that of IGEBM and have much more local optima.
Therefore, MCMC chain in the input space does not effectively explore possible samples from the model.
On-manifold initialization is hence essential for training NAE for complex, high-dimensional data like images.
Hope this helps.

from normalized-autoencoders.

I used standard hyperparam settings as in IGEBM, i.e., step = 60, step_size = 10, noise_std = 0.005, clip_langevin_grad = 0.01.

from normalized-autoencoders.