The general arechitecture of an MLDL neural network.

Comparison of training processes of three autoencoders on the Spheres dataset.

This is a PyTorch implementation of the MLDL paper :

@article{Li-MarLip-2020,
  title={Markov-Lipschitz Deep Learning},
  author={Stan Z Li and Zelin Zang and Lirong Wu},
  journal={arXiv preprint arXiv:2006.08256},
  year={2020}
}

The main features of MLDL for manifold learning and generation in comparison to other popular methods are summarized below:

The code includes the following modules:

• Datasets (Swiss Roll, S-Curve, MNIST, Spheres)
• Training for ML-Enc and ML-AE (ML-Enc + ML-Dec)
• Test for manifold learning (ML-Enc)
• Test for manifold generation (ML-Dec)
• Visualization
• Evaluation metrics
• The compared methods include: AutoEncoder (AE), Topological AutoEncoder (TopoAE), Modified Locally Linear Embedding (MLLE), ISOMAP, t-SNE. (Note: We modified the original TopoAE source code to make it able to run the Swiss roll dataset by adding a swiss roll dataset generation function and modifying the network structure for fair comparison.)

• pytorch == 1.3.1
• scipy == 1.4.1
• numpy == 1.18.5
• scikit-learn == 0.21.3
• csv == 1.0
• matplotlib == 3.1.1
• imageio == 2.6.0

• main.py
  • SetParam() -- Parameters for training
  • Train() -- Train a new model (encoder and/or decoder)
  • Train_MultiRun() -- Run the training for multiple times, each with a different seed
  • Generation() -- Testing generation of new data of the learned manifold
  • Generalization() -- Testing dimension reduction from unseen data of the learned manifold
  • InlinePlot() -- Inline plot intermediate results during training
• dataset.py
  • LoadData() -- Load data of selected dataset
• loss.py
  • MLDL_Loss() -- Calculate six losses: ℒ_Enc, ℒ_Dec, ℒ_AE, ℒ_lis, ℒ_push, ℒ_ang
• model.py
  • Encoder() -- For latent feature extraction
  • Decoder() -- For generating new data on the learned manifold
• eval.py -- Calculate performance metrics from results, each being the average of 10 seeds
• utils.py
  • GIFPloter() -- Auxiliary tool for online plot
  • CompPerformMetrics() -- Auxiliary tool for evaluating metric
  • Sampling() -- Sampling in the latent space for generating new data on the learned manifold

1. Clone this repository

git clone https://github.com/westlake-cairi/Markov-Lipschitz-Deep-Learning

2. Install the required dependency packages

3. To get the results for 10 seeds, run

python main.py -MultiRun

4. To get the metrics for ML-Enc and ML-AE

python eval.py -M ML-Enc
python eval.py -M ML-AE

The evaluation metrics are available in ./pic/PerformMetrics.csv

5. To choose a dataset among SwissRoll, Scurve, MNIST, Spheres5500 and Spheres10000 for tow modes (ML-Enc and ML-AE)

python main.py -D "dataset name" -M "mode"

6. To test the generalization to unseen data

python main.py -M Test

The results are available in ./pic/file_name/Test.png

7. To test the manifold generation

python main.py -M Generation

The results are available in ./pic/file_name/Generation.png

• Swiss Roll and S-Curve

  A symbol √ or X represents a success or failure in unfolding the manifold. The methods in the upper-row ML-Enc succeed and by calculation, the ML-Enc best maintains the true aspect ratio.

• MNIST (10 digits)

• Spheres (data designed by the TopoAE project )

This table demonstrates that the ML-Enc outperforms all the other 6 methods in all the evaluation metrics, particularly significant in terms of the isometry (LGD, RRE, Cont and Trust) and Lipschitz (K-Min and K-Max) related metrics.

The learned ML-Enc network can unfold unseen data of the learned manifold, demonstrated using the Swiss-roll with a hole, whereas the compared methods cannot.

In the learning phase, the ML-AE taking (a) the training data as input, output (b) embedding in the learned latent space, and then reconstruct back (c). In the generation phase, the ML-Dec takes (d) random input samples in the latent space, and maps the samples to the manifold (e).

The ML-AE training gradually unfolds the manifold from input layer to the latent layer and reconstructs the latent embedding back to data in the input space.

If you have any issue about the implementation, please feel free to contact us by email:

• Zelin Zang: [email protected]
• Lirong Wu: [email protected]