eval_math script outputs 0 accuracy about metamath HOT 4 CLOSED

Hi~ zhangir-azerbayev, Thanks for your attention!
have you fixed this error?
I retested it, and I am able to obtain the expected numbers. it works for both 1x A100 80GB and 8x A100 80GB
The environment I am using is:
torch==2.0.1+cu117
vllm==0.1.4
transformers==4.31.0

from metamath.

Hi~ zhangir-azerbayev, Thanks for your attention!

have you fixed this error?

I retested it, and I am able to obtain the expected numbers. it works for both 1x A100 80GB and 8x A100 80GB

The environment I am using is:

torch==2.0.1+cu117

vllm==0.1.4

transformers==4.31.0

Thanks for your response. I'll retry with those package versions and report back.

from metamath.

Thanks, was able to repro the results in this paper with vllm==0.1.4 and transformers==4.31.0. Will PR a requirements.txt file when I have time.

from metamath.

I ran the eval scripts with the weights from meta-math/MetaMath-7(13)B-V1.0 and obtained scores of 0.1948 for 7B and 0.2238 for 13B. However, when I �ran SFT on the metamathqa data for LLama2-7B/13B, I got 0 score for math and 0.71 for gsm8k. What could be the reason for the score of 0 in math, and how can I prevent it? (I don't think it's an issue with the package version.)

The environment I am using is:
torch==2.0.1+cu118
vllm==0.1.4
transformers==4.34.0

from metamath.