Another viewpoint to interpret the score about sjc HOT 5 CLOSED

hi Yuanzhi, thanks for your interest in the work.

The benefits of writing the score as $\frac{D - x}{\sigma^2}$ include, but not limited to

1. You can visualize $D$. $D$ is a (potentially blurry) image. This is helpful for debugging and understanding.
2. It is clear that $D - x$ is mean-shift. It is more intuitive.
3. $D - x$ is also the gradient of a l2 reconstruction loss. This is helpful for understanding what's happening. When doing 3D optimization, 2D diffusion guidance is providing 1-step reconstruction targets.

And sry about the earlier deleted response. Since you are referring to score at time $t$ rather than at noise level $\sigma$, the expression you have is correct. $\nabla_x \log p_{\sigma}(x)$ and $\nabla_x \log p_{t}(x)$ are different and I was thinking of the former. I personally find it more helpful to think of things in noise-to-signal ratio.

from sjc.

Also $\epsilon$ parameterization is one particular way of parameterizing $D$.
The consensus on denoiser parametrization now leans towards $v$-param and Karras param. This is orthogonal to whether the SDE is VP or VE. $\epsilon$ is brittle when noise is large.

Since in the end it's all denoising, it seems easier to treat them all as $D$, and the parametrization is just an internal neural net black-box detail.

from sjc.

Hi Haochen, thank you so much for elaborating on the advantages of writing the score as $\frac{D - x}{\sigma^2}$ and of the new parametrization.

Here I have a follow-up question regarding the difference between $\nabla_x \log p_{\sigma}(x)$ and $\nabla_x \log p_{t}(x)$. aren't they the same as the noise schedule $\sigma$ is an injective function of $t$?

from sjc.

Scaling down increases the density. The denominator is $\sqrt{\frac{1 - \bar{\alpha}}{\bar{\alpha}}}$ (without scaling) or $\sqrt{1 - \bar{\alpha}}$ (with scaling, like the way you wrote).
If there is no trajectory scaling then $\sigma$ and $t$ are 1-to-1 and it's fine.
DDPM uses scaling to cap the variance (which, only in hindsight, appears unnecessary). VESDEs are easier to solve and the formula tends to be simpler.

from sjc.

Thank you so much for your clarification.

from sjc.