Why LLaMa pruning is difficult? about sparsegpt HOT 4 CLOSED

Hi, could you share your pruning script? The llama model is similar with opt, so I think it should work

from sparsegpt.

Hi, @akisd2020 .LLaMA is more parameter efficient than OPT and bloom - hence It is natural that it is harder to maintain same performance using the same algorithm.

from sparsegpt.

@songlinlibit In the latest commit I've added LLaMA training script. Overall, all sparseGPT should be done for all CausalLM without significant changes

from sparsegpt.

In addition to what Denis said with LLaMa being more parameter efficient, the reconstruction process at the core of SparseGPT also appears to be less effective on LLaMa, either because there are less correlations between intermediate inputs that can be exploited and/or because the default calibration data that we used for OPT/BLOOM is not very well suited for LLaMa. In any case, improving SparseGPT specifically for pruning LLaMa is a great topic for further research. (Please also note that, in general, SparseGPT does work reasonably on LLaMa, but accuracy drops are generally a bit higher than for OPT/BLOOM; if you observe massive drops, then there might be some implementation problem.).

from sparsegpt.