Gaussian elimination about sparsegpt HOT 1 CLOSED

Hi, this is due to some extra optimizations which we omitted during our main discussion (we mention them later sections and the pseudocode) of the algorithm to convey the high level ideas as clearly as possible. First, the inverse Hessian sequence is computed in advance via a Cholesky decomposition, which essentially performs Gaussian elimination. Second, we lazily batch weight updates together to improve GPU utilization; this is why we subselect an inverse Hessian block, process the corresponding columns, and then batch update the remainder of the weight matrix in the end. You can read more about these optimizations in our related GPTQ paper. Overall, these optimizations do not affect the algorithm (they are equivalent), but significantly improve speed and stability.

from sparsegpt.