[BUG] MLFlow Deployment Server for LLMs using chatCompletion on Azure OpenAI text-davinci-003 about mlflow HOT 4 CLOSED

it looks like the completions route is hardcoded in the deployments code to point to chat/completions, as it's the new standard set by OpenAI:

https://github.com/mlflow/mlflow/blob/master/mlflow/gateway/providers/openai.py#L266

given that the text-davinci-003 model will be deprecated and unavailable in a couple months, i'm not sure if we can prioritize a fix for this. as a workaround, gpt-35-turbo should work!

from mlflow.

Thanks for the answer. I know this for OpenAI models, and text-davinci-003 is indeed close to retirement. But have all completion models really disappeared, e.g. open source/huggingface, and every LLM supports chat/completions now and in the future?
This is not a big problem for us. As you said, we can switch to gpt-35-turbo (which is cheaper, anyway). However, this decision may affect the ability of the MLFlow deployment server to be fully generic.

from mlflow.

This issue only affects OpenAI endpoints!

If you want to deploy an open source model completions model using MLflow (there's a guide for doing that here), you shouldn't run into this issue. Other providers (e.g. Anthropic) also shouldn't have this issue, as the chat/completions code is specific to the OpenAI provider.

from mlflow.

Understood. Thank you for the clarification. Since it no longer seems relevant after your explanations, I'll close this issue.

from mlflow.