syiswell / mi_bounds_pytorch Goto Github PK

All those bounds were derived from the work of Ben Poole and al., On Variational Bounds of Mutual Information, 2019.

Their TensorFlow implementation, as well as useful insights, are available on their GitHub

In the jupyter notebook MI_bounds_pytorch.ipynb you'll find implementation and example with toy data of the following variational lower bounds on MI :

• NWJ / Mine-f lower bound. High variance, low bias. ref
• Noise Contrastive Estimation based (infoNCE bound). Low variance, high bias. ref
• Interpolated bound, control of bias-variance trade-off. ref

General guidelines from Ben Poole and al. :

• For representation learning purpose, use infoNCE lower bound
• For MI estimation purpose, use interpolated bound with a low alpha