mbaddar1 / genmodel Goto Github PK

View Code? Open in Web Editor NEW

1.0 1.0 0.0 23.79 MB

Generative Model Experiments

License: Apache License 2.0

Python 100.00%

genmodel's People

Contributors

Stargazers

Watchers

genmodel's Issues

Explore the DDPM pytorch repo

https://github.com/mbaddar1/denoising-diffusion-pytorch
Understand

UNet architecture
Loss function
The optimization
Try to adapt for simple cases like swissroll and conc circle
Understand the FID score

Explore the song score_sde_torch repo

https://github.com/mbaddar1/score_sde_pytorch/tree/cb1f359f4aadf0ff9a5e122fe8fffc9451fd6e44

Understand how Repo works
Plan a set of small experiments, and viz to test the points you have understood in #7
Document these experiments plan into #9

DiffModel Experminet 1 - UNet - Gradient Descent - L2 - MNIST

This is a preliminary experiment of the experiment set document in sec 8.7.1 here
https://github.com/mbaddar1/phd_latex/blob/c37daefa57518c5d4fa762a7cb254f1cbaa5c617/out/main.pdf

Experiment objectives :

Get hands on with DDPM PyTorch Repo https://github.com/mbaddar1/denoising-diffusion-pytorch
Understand the UNet architecture
Understand the DDPM L2 loss function
Plot the evolution of evaluation metric and loss function

Conduct an experiment as follows

Dataset :
for test :
1 ) mvn 2d as in my test code here
https://github.com/mbaddar1/genmodel/blob/master/sandbox/diffmodel/simple_diffusion_model.py#L33
and also similar to this mvn experiment here
dff6026
In the old experiment the code was using KL divergence and some different head and time variant tail as model, instead of UNet
In this experiment : we will use UNet and DDPM L2. Both will use Gradient Descent

https://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_swiss_roll.html

2 ) Circles
https://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_circles.html

similar to the previous experiment

f82e3f0

Model
Use UNet model as in this code part
https://github.com/mbaddar1/denoising-diffusion-pytorch/blob/main/denoising_diffusion_pytorch/denoising_diffusion_pytorch.py#L272

Loss function
Use the L2 Loss as in
https://arxiv.org/pdf/2006.11239.pdf eq 14 p5

Optimization Method
Gradient Descent as in
https://arxiv.org/pdf/2006.11239.pdf Algorithm 1 P4

Evaluation Metric
FID as in Fig 3 and 4 in https://arxiv.org/pdf/2006.11239.pdf

Reading Song Paper

https://arxiv.org/pdf/2011.13456.pdf

Main points to grasp

How it is compared to DDPM
Different Variants for Score Matching (SDE and ODE)
What is the Model architecture (UNET)
What is the loss function
How it is optimized

Read : High-dimensional density estimation with tensorizing flow

https://arxiv.org/pdf/2212.00759.pdf

Dev code for quality measure for generated data samples

Objective to have some function q-measure(sample) that gives a measure on the generated sample, to use along with L2 loss while training the diffusion model.
Dataset :

sklearn conc circles
" half moon
" blob

Focus mainly on the sinkhorn algorithm
https://github.com/fwilliams/scalable-pytorch-sinkhorn?tab=readme-ov-file

misc links
https://gist.github.com/atabakd/ed0f7581f8510c8587bc2f41a094b518
sinkhorn
https://amsword.medium.com/a-simple-introduction-on-sinkhorn-distances-d01a4ef4f085
https://papers.nips.cc/paper_files/paper/2013/hash/af21d0c97db2e27e13572cbf59eb343d-Abstract.html

Read : Tensor-Train Density Estimation

https://arxiv.org/pdf/2108.00089.pdf

DiffModel Experminet 1 - UNet - Gradient Descent - L2 - sklearn dataset

apply the DDPM code to sklearn swissroll, conc circles, and half moons dataset. find an objective eval measure

Plan Dump from Latex

\subsection{Work-Schedule}\label{subsec:time-table}
\begin{enumerate}
    \item Read about score/gradient matching based generative models~\cite{Generati61:online} 20-Nov-2023 \textcolor{cyan}{DONE}
    \item Read this comprehensive paper :\cite{song2021scorebased} 20 to 25 Nov 2023 \textcolor{green}{In progress}
    \begin{enumerate}
        \item Focus on section 2.1 (score-matching) \textcolor{cyan}{done}
        \item Focus on section 2.2 (DDPM) \textcolor{green}{in prgoress}
        \item Skim through the rest
    \end{enumerate}
    \item A quick overview for this diffusion model tutorial~\cite{weng2021diffusion} 22-Nov-2023 -> Focus on
    Loss function understanding and derivation , also on different variants of it
    \item Experiment these two simple implementation of DDPM~\cite{aju22DDP15:online,Generati57:online}.
        Have a look at this bigger one~\cite{lucidrai97:online}
    \item Hands on coding for the basic score matching~\cite{DBLP:journals/corr/abs-1907-05600}
    From scratch implementation can be good for learning~\cite{Benefits95:online}. \newline
    Try this tutorial \cite{Tutorial71:online} which I have found in this GitHub page~\cite{yangsong73:online}
    and blog post \cite{Generati61:online}
    =============================
    \item A quick overview for Deep Learning Book~\cite{jentzen2023mathematical}
    \textbf{Focus on Chapter 5 -> Gradient-FLow} 25-Nov-2023
    \item A quick overview Riemannian Optimization~\cite{smith2014optimization}
    \item Check this paper for TT-DNN \cite{qi2022exploiting} (Good One)
    \item Check this TT-DNN paper~\cite{qi2019riemannian} (Rejected one)
    \item TT-GP \cite{izmailov2018scalable}
    \item A quick overview for this Riemannian Optimization Autograd paper~\cite{novikov2021automatic} 26-Nov-2023
    \item Riemannian Optimization with T3F lib \cite{novikov2020tensor}
    \item A quick overview for this Diffusion Model survey paper~\cite{yang2023diffusion} 27-Nov-2023
    \item Try to make connections between topics from Riemannian Opt or DL Book to Diffusion Models 27 to 28 Nov 2023
    \item Experimentation the score sde method(\href{https://github.com/yang-song/score_sde}{code})
    and DDPM (\href{https://github.com/lucidrains/denoising-diffusion-pytorch}{code}).
    29 Nov 2023
\end{enumerate}
%%%
\subsubsection{Understanding Connection Between Diffusion Models, Langevin Dynamic and Differential Equations}
\label{subsubsec:conn-ddpm-lavdyn-diffeq}
\begin{itemize}
    \item write the sampling eqns of both langevin dyn , ddpm \cite{weng2021diffusion}
    \item compare {$\epsilon_{\theta}$} and  $\nabla_{x} \log p(x)$ .
    \item Highlight that the derivation of correspondence between DDPM and SMLD losses is heuristic based
    Eq (3) in \cite{song2021scorebased} and Eq(12) in \cite{ho2020_ddpm}
\end{itemize}