GPT4 rate limit about alpaca_eval HOT 2 CLOSED

Great questions!

1. Concerning the rate limit, there are 3 ways to avoid/mitigate it.

   • Decreasing number of processes: by default we use 4 concurrent processes for querying GPT4. This will hit the rate limit faster. If that's an issue you can change it in the evaluators configs. e.g. add num_procs: 2 in the completion kwargs
   • Increasing sleep time: we are already adding a 2 seconds sleep time here. If this is not sufficient for you, you can change it in the evaluators configs. e.g. add sleep_time: 10 in the completion kwargs
   • Using multiple API keys: in case you have multiple API keys you can set them separated by commas OPENAI_API_KEYS=key1,key2 and we will switch between them when you get rate limited.

2. When you hit a rate limit we will simply retry (as seen in the warning) so that's not the issue. If there are missing evaluation results it typically means that the annotations could not be parsed. we also raise a warning for that so you see the reason in your terminal!

Feel free to reopen if you have further questions

from alpaca_eval.

For future reference, here's the way to deal with multiple keys now: https://github.com/tatsu-lab/alpaca_eval/tree/main/client_configs

one can also use Azure

from alpaca_eval.