Why Hessian can get by activation ($H = XX^T$) ？ about sparsegpt HOT 4 CLOSED

SparseGPT operates in a layer-wise manner, solving the corresponding layer-wise pruning problem, given by Equation (1) in the paper. This is essentially a masked linear squared error problem of which the Hessian is indeed $XX^T$; you are right that the whole network's Hessian includes second-order derivatives, but the SparseGPT algorithm does not require this Hessian.

from sparsegpt.

SparseGPT operates in a layer-wise manner, solving the corresponding layer-wise pruning problem, given by Equation (1) in the paper. This is essentially a masked linear squared error problem of which the Hessian is indeed XXT; you are right that the whole network's Hessian includes second-order derivatives, but the SparseGPT algorithm does not require this Hessian.

Thanks for the answer, but why the Hessian for the masked linear squared error problem is $XX^T$ and how was this result derived?

from sparsegpt.

@Beatlesso , check the least squares estimator here: https://global.oup.com/booksites/content/0199268010/samplesec3

The first order derivative of the squared loss function is as shown in equation 3.7, the second order derivative is as shown in equation 3.10 (which is essentially the same as XX^T if you interchange the row and column dimension).

from sparsegpt.

@Beatlesso , check the least squares estimator here: https://global.oup.com/booksites/content/0199268010/samplesec3

The first order derivative of the squared loss function is as shown in equation 3.7, the second order derivative is as shown in equation 3.10 (which is essentially the same as XXT if you interchange the row and column dimension).

This is very useful, thanks!

from sparsegpt.