About USMLE results about lmflow HOT 1 CLOSED

Hi,
We have not fine-tuned the LLaMA-7B on the USMLE training data. It seems low with 30% accuracy on in-domain data. One possible reason is the training data of USMLE is small (around 10K), while the training data of PubMedQA and MedMCQA is about 400K although they have different data distributions (OOD).

Could you try our example first to see whether you could reproduce our results? We choose this setting because, in Galactica, they deem PubMedQA and MedMCQA as in-domain and USMLE as out-of-domain data.

Thanks!

from lmflow.