korepwx / blog-comments Goto Github PK

https://ipwx.me/2017/12/31/2017%E5%B9%B4%E5%BA%A6%E6%80%BB%E7%BB%93/

转眼间 2017 年就这么过去了，回想一下的话时间过得还真快。去年的这个时候，我怀着五味杂陈的心情写了篇年度总结。那时候的我对于机器学习和深度学习这两者还没有完全领会神髓，对于数学的学习又处于零零碎碎不成体系的阶段，而做了好久的 Cake 项目又眼看着完全没法用，所以不能说心情很好。不过今年事情稍稍发生了一点变化，我各个方面都有了点“感觉”，无论是机器学习和深度学习，还是数学，亦或者是各种轮子的编程，我都有不小的收获。因此今年的这个总结，我总体来说还是挺开心的。

https://ipwx.me/about/

https://ipwx.me/2017/12/31/2017%E5%B9%B4%E5%BA%A6%E6%80%BB%E7%BB%93/

转眼间2017年就这么过去了，回想一下的话时间过得还真快。去年的这个时候，我怀着五味杂陈的心情写了篇年度总结。那时候的我对于机器学习和深度学习这两者还没有完全领会神髓，对于数学的学习又处于零零碎碎不成体系的阶段，而坐了好久的Cake项目又眼看着完全没法用，所以不能说心情很好。不过今年事情稍稍发生了一点变化，我各个方面都有了点“感觉”，无论是机器学习和深度学习，还是数学，亦或者是各种轮子的编程，我都有不小的收获。因此今年的这个总结，我总体来说还是挺开心的。

https://ipwx.me/links/

https://ipwx.me/2017/11/24/variational-inference/

OverviewSuppose there are two random variables ${ {\boldsymbol{\mathrm{ {x} } } } }$ and ${ {\boldsymbol{\mathrm{ {z} } } } }$, where ${ {\boldsymbol{\mathrm{ {x} } } } }$ is observed and ${ {\boldsymbol{\mathrm{ {z} } } } }$ is not. We assume these two variables are modeled with a neural network, with parameter ${ {\boldsymbol{\mathrm{ {\theta} } } } }$. The prior of ${ {\boldsymbol{\mathrm{ {z} } } } }$ is $p _ {\theta}({ {\boldsymbol{\mathrm{ {z} } } } })$, while the posterior of ${ {\boldsymbol{\mathrm{ {x} } } } }$ given ${ {\boldsymbol{\mathrm{ {z} } } } }$ is $p _ {\theta}({ {\boldsymbol{\mathrm{ {x} } } } } \vert { {\boldsymbol{\mathrm{ {z} } } } })$. Although computing $p _ {\theta}({ {\boldsymbol{\mathrm{ {x} } } } } \vert { {\boldsymbol{\mathrm{ {z} } } } })$ is straightforward in this scenario, the posterior of ${ {\boldsymbol{\mathrm{ {z} } } } }$ given ${ {\boldsymbol{\mathrm{ {x} } } } }$, i.e., $p _ {\theta}({ {\boldsymbol{\mathrm{ {z} } } } } \vert { {\boldsymbol{\mathrm{ {x} } } } })$, is intractable by analytic methods.While it is possible to use sampling based methods to evaluate $p _ {\theta}({ {\boldsymbol{\mathrm{ {x} } } } })$ and further obtain $p _ {\theta}({ {\boldsymbol{\mathrm{ {z} } } } } \vert { {\boldsymbol{\mathrm{ {x} } } } }) = p _ {\theta}({ {\boldsymbol{\mathrm{ {x} } } } },{ {\boldsymbol{\mathrm{ {z} } } } }) / p _ {\theta}({ {\boldsymbol{\mathrm{ {x} } } } })$, it often demands a large number of samples. Alternatively, one may fit another neural network $q _ {\phi}({ {\boldsymbol{\mathrm{ {z} } } } } \vert { {\boldsymbol{\mathrm{ {x} } } } })$ to approximate the true posterior $p _ {\theta}({ {\boldsymbol{\mathrm{ {z} } } } } \vert { {\boldsymbol{\mathrm{ {x} } } } })$, by using the variational inference techniques.In the context of deep learning, a variational inference algorithm that fits $q _ {\phi}({ {\boldsymbol{\mathrm{ {z} } } } } \vert { {\boldsymbol{\mathrm{ {x} } } } })$ should basically have two elements. One is the training objective, which we may call variational objective hereafter. The other is the gradient estimator, since almost the entire deep learning is based on gradient descent techniques.