Possibility to specify custom API endpoint address? about llmtest_needleinahaystack HOT 4 OPEN

Hey! Thanks for the note - yep that is definitely doable we just need to add it to the repo. This isn't on our immediate development plans for the maintainers but if you want to throw up a PR we can take a look

from llmtest_needleinahaystack.

I also need that to be working.
Releasing that would be greatly appreciated.

from llmtest_needleinahaystack.

https://github.com/vllm-project/vllm/blob/main/examples/openai_chatcompletion_client.py
This vllm client example might worth to look at how they integrated custom url with openai.

from llmtest_needleinahaystack.

https://github.com/vllm-project/vllm/blob/main/examples/openai_chatcompletion_client.py
This vllm client example might worth to look at how they integrated custom url with openai.

I'm aware of this setting already, unfortunately things are a little more complex as they are using langchain openAI which complicates things a little more than that.

from llmtest_needleinahaystack.