why i can't reproduce the result of paper? about sparsegpt HOT 7 CLOSED

hi, i am very confused, I am reproducing opt-1.3b sparsity. the fact is that i can get the same dense model preplexity at 14.62 of wikitext2, but the preplexity after sparsing 50% is 26.71, which is higher than 17.46 in paper. And I didn't modify any code in this repo. I wonder if the hyperparams in paper's experiments are different? look forward to your reply.

@ilmarkov hi, could you reply to me in your spare time ? i am stuck in this problem. I don't know how i reproduce the result as showed in paper. please

from sparsegpt.

Hi, what command are you running? Generally, the code has been running fine for many people; however, it is possible that some recent HuggingFace update broke things.

from sparsegpt.

Hi, what command are you running? Generally, the code has been running fine for many people; however, it is possible that some recent HuggingFace update broke things.

Thank you for your reply. The command I run is the one in the readme. as follows:
python opt.py facebook/opt-1.3b wikitext2 --sparsity .5 --save ./out/opt-1.3b/sparse0.5/

and huggingface version i am using is 0.23.0
I can run your code succuessfully, but the results are slightly different from the results in paper:

look forward to your reply, Thanks

from sparsegpt.

Hi, in the paper we run with calibration data from c4 and not wikitext2; the former is not only more general but I think the latter can even sometimes lead to overfitting and correspondingly bad results (possibly this is what you are seeing?).

python opt.py facebook/opt-1.3b c4 --sparsity .5 --save ./out/opt-1.3b/sparse0.5/

from sparsegpt.

Hi, in the paper we run with calibration data from c4 and not wikitext2; the former is not only more general but I think the latter can even sometimes lead to overfitting and correspondingly bad results (possibly this is what you are seeing?).

python opt.py facebook/opt-1.3b c4 --sparsity .5 --save ./out/opt-1.3b/sparse0.5/

Thank you. To verify if the code is wrong, I test more opt models:

as you can see, Apart from opt-1.3b, the other model results differ only slightly from the paper, So I think the code is right and most reproduced results looks normal too .
That is weird. And I can confirm there is not any modify when i test opt-1.3b.
Besides, The table in your paper shows the result on wikitext2:

from sparsegpt.

We evaluate on wikitext but always use c4 as calibration data (this should be noted in the paper). Yes, this is a bit weird, however there can sometimes be outlier models which do not follow the trend.

from sparsegpt.

We evaluate on wikitext but always use c4 as calibration data (this should be noted in the paper). Yes, this is a bit weird, however there can sometimes be outlier models which do not follow the trend.

yes. i realize i made a mistake. i didn't use c4 as calibration data. Now I solve the problem, Thank you!!!!

from sparsegpt.