Comments (5)
entrypoints/api_server
is demowareentrypoints/openai/api_server
is production
from vllm.
This is a good idea.
Do you have capacity to implement it?
from vllm.
@robertgshaw2-neuralmagic Note that a previous PR for this functionality was rejected, see #1244.
I'm not sure if the goal of api_server
is just "demo ware you should fork", as suggested in that PR, or if it should be "simple, but production-ready".
from vllm.
In any case, a simple wiring of /health
AND /ready
also for the demoware doesn't hurt. It does not have to do much in the sense of actually determining some ready state. But having those "stub" endpoints and handlers implemented, is somewhat of an interface or blueprint for derived implementations to #put some code here#
Getting to the openai API server:
If we agree that this feature should be implemented somehow, what would be a good sources / starting point to determine vLLM is ready to receive requests? So to follow my initial thoughts, how do I determine if the model is downloaded and loaded / imported into the GPU, KV is setup?
Coming to your question (#6073 (comment)) @robertgshaw2-neuralmagic, I might be able to come up with a PR, but would love some more discussion on the implementation and "quality" of the reported status (be it vLLMs health or readiness). As for health, I'd like to expose as much of a proper health check as possible. Round-tripping a static API endpoint is nice, but does not go much beyond taking the existence of a running process as health check. What about running out of CUDA memory or other issues at runtime? Do they always ultimately cause the process to crash / end itself? Or would some more internal checking help to see if vLLM is actually still alive?
from vllm.
In any case, a simple wiring of
/health
AND/ready
also for the demoware doesn't hurt. It does not have to do much in the sense of actually determining some ready state. But having those "stub" endpoints and handlers implemented, is somewhat of an interface or blueprint for derived implementations to#put some code here#
Getting to the openai API server:
If we agree that this feature should be implemented somehow, what would be a good sources / starting point to determine vLLM is ready to receive requests? So to follow my initial thoughts, how do I determine if the model is downloaded and loaded / imported into the GPU, KV is setup?
Coming to your question (#6073 (comment)) @robertgshaw2-neuralmagic, I might be able to come up with a PR, but would love some more discussion on the implementation and "quality" of the reported status (be it vLLMs health or readiness). As for health, I'd like to expose as much of a proper health check as possible. Round-tripping a static API endpoint is nice, but does not go much beyond taking the existence of a running process as health check. What about running out of CUDA memory or other issues at runtime? Do they always ultimately cause the process to crash / end itself? Or would some more internal checking help to see if vLLM is actually still alive?
Regarding your question "So to follow my initial thoughts, how do I determine if the model is downloaded and loaded / imported into the GPU, KV is setup?", I have tried to used in this PR #7078 the "model_memory_usage" variable in model_runner object to determine the readiness once the model weights are loaded in the GPU memory.
from vllm.
Related Issues (20)
- [New Model]: Snowflake Arctic Embed (Family)
- [Bug]: FP8 Marlin fallback out of memory regression
- [Usage]: Is there any way to hook features inside vision-language model? HOT 1
- [Bug]: my vllm phi-3-vision server runs one request correctly then returns an error for the same request stating 2509 image tokens to 0 placeholders HOT 3
- Request support for the deepseek-gptq version HOT 1
- [Bug]: Running mistral-large results in an error related to NCCL
- [Bug]: Running mistral-large results in an error related to NCCL HOT 2
- [Bug]: ModuleNotFoundError: No module named 'openai.types' HOT 5
- [Bug]: falcon-40B model support
- [Usage]: How to generate independent samples for a given input?
- [Bug]: Llama 3 answers starting with <|start_header_id|>assistant<|end_header_id|>
- [Usage]: The seed in vllm.SamplingParams and vllm.LLM HOT 2
- [Bug]: Docker build for ROCm fails for latest release and main branch HOT 2
- [Bug]: Error: No available node types can fulfill resource request HOT 1
- [Usage]: When debugging with vLLM, a CUDA error occurs. HOT 2
- [Bug]: tool_calls parsing error with CPU
- [Bug]: Nemotron 340B does not generated EOS token
- [Bug]:
- [BUG]: Support AI21-Jamba-1.5-Large (and mini) HOT 2
- [Bug]: AI21-Jamba-1.5-Mini RuntimeError: Triton Error [CUDA]: an illegal memory access was encountered
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from vllm.