Self-host llmapi server
llmapi-server is an abstract backend that encapsulates a variety of large language models (LLM, such as ChatGPT, GPT-3, GPT-4, etc.), and provides simple access services through OpenAPI
- run locally
# python >= 3.8
python3 -m pip install -r requirements.txt
mkdir logs
python3 run_api_server.py
- run with docker
./build_docker.sh
./start_docker.sh
- Use the
curl
command to access:
# 1. Start a new session
curl -X POST -H "Content-Type: application/json" -d '{"bot_type":"mock"}' http://127.0.0.1:5050/v1/chat/start
# response sample: {"code":0,"msg":"Success","session":"123456"}
# 2. chat with LLMs
curl -X POST -H "Content-Type: application/json" -d '{"session":"123456","content":"hello"}' http://127.0.0.1:5050/v1/chat/ask
# response sample: {"code":0,"msg":"Success","reply":"Text mock reply for your prompt:hello","timestamp":1678865301.0842562}
# 3. Close the session and end chat
curl -X POST -H "Content-Type: application/json" -d '{"session":"123456"}' http://127.0.0.1:5050/v1/chat/end
# response: {"code":0,"msg":"Success"}
- Using command line tools:llmapi_cli
llmapi_cli --host="http://127.0.0.1:5050" --bot=mock
- Integrate in your python code with llmapi_cli module
from llmapi_cli import LLMClient
client = LLMClient(host = "http://127.0.0.1:5050", bot = "mock")
rep = client.ask("hello")
print(rep)
- You need to create a new backend name in the backend directory (assumed to be
newllm
), you can directlycp -r mock newllm
- Referring to the implementation of
mock
, change the backend name tonewllm
- In the
newllm
directory, add the necessary dependencies, and all related development is bound to this directory - Add support for
newllm
inbackend.py