christiando / gaussian-toolbox Goto Github PK

So far just some moment matching (MM) classes are defined as approximate conditionals. However, there are other ways to define approximate affine operations, e.g. by

• Variational inference (VI)
• Laplace approximation (LA)
• Unscented transform (UT)

One should write a base class for each type of approximation and then implement some examples. It is expected that VI, LA, and UT can be more general than MM methods.

At the moment Gaussian (PDF&Measure) diagonals are still full matrices (matrix inversion done differently), which is computationally and memory-wise is not very efficient.

Like layers in deep nets, conditionals can be defined as a chain of many conditionals. While this does not make sense for linear Gaussian conditionals (since multiple linear conditionals can be written as one conditional), for non-linear ones, it does. Affine transformations can be implemented as forward (or backward) iterations over the conditional "layers". Integration over the log conditionals is also straightforward since it is just the sum of the integrals of layers of log conditionals.

Many operations like for Gaussian mixtures can be implemented by looping over Gaussian operations, especially the affine transformations. However, there are also some problems, e.g. how to implement integration over the log density of mixtures. There are two possibilities:

• Allow mixtures only in restricted mode. Straightforward to implement, but less valuable, since parameter learning in general will rely on terms that require integration of log-densities
• Come up with the implementation of log-density integration, to enable full usage of mixtures in the context of the toolbox.

While GT allows the implementation of all kinds of approximations for complicated log conditionals, it would be nice to have standardized quality assessments for how well different operations are approximated. To this end, one could design a certain test bench, which summarizes the quality of these operations (affine transformations, log integrations). However, metrics and test cases need to be defined.