[Feature] Discussion of Evaluation Results for HumanEval Dataset about opencompass HOT 4 CLOSED

The discrepancy arises from variations in the prompts and inherent randomness. When you perform inference using vLLM or LMDeploy, the performance might exhibit a significant fluctuation, typically within a range of 3 to 5 points.

from opencompass.

Thanks for your reply.

Do you have any suggestions on how to stabilize the performance during evaluation or achieve relatively higher results?

I guess that LLM developers might report higher results in their findings. If possible, I'd like to explore how they achieve that.

from opencompass.

Prompt engineering is an effective approach; you can explore different prompts. OC also offers several alternative prompts, so feel free to try them out.
Like: https://github.com/open-compass/opencompass/blob/main/configs/datasets/humaneval/humaneval_openai_sample_evals_gen_159614.py

from opencompass.

Thanks for your reply.

I will have a try regarding the prompt engineering.

from opencompass.