[Feature]: Add readiness endpoint /ready and return /health earlier (vLLM on Kubernetes) about vllm HOT 5 OPEN

• entrypoints/api_server is demoware
• entrypoints/openai/api_server is production

from vllm.

This is a good idea.

Do you have capacity to implement it?

from vllm.

@robertgshaw2-neuralmagic Note that a previous PR for this functionality was rejected, see #1244.

I'm not sure if the goal of api_server is just "demo ware you should fork", as suggested in that PR, or if it should be "simple, but production-ready".

from vllm.

In any case, a simple wiring of /health AND /ready also for the demoware doesn't hurt. It does not have to do much in the sense of actually determining some ready state. But having those "stub" endpoints and handlers implemented, is somewhat of an interface or blueprint for derived implementations to #put some code here#

Getting to the openai API server:

If we agree that this feature should be implemented somehow, what would be a good sources / starting point to determine vLLM is ready to receive requests? So to follow my initial thoughts, how do I determine if the model is downloaded and loaded / imported into the GPU, KV is setup?

Coming to your question (#6073 (comment)) @robertgshaw2-neuralmagic, I might be able to come up with a PR, but would love some more discussion on the implementation and "quality" of the reported status (be it vLLMs health or readiness). As for health, I'd like to expose as much of a proper health check as possible. Round-tripping a static API endpoint is nice, but does not go much beyond taking the existence of a running process as health check. What about running out of CUDA memory or other issues at runtime? Do they always ultimately cause the process to crash / end itself? Or would some more internal checking help to see if vLLM is actually still alive?

from vllm.

In any case, a simple wiring of /health AND /ready also for the demoware doesn't hurt. It does not have to do much in the sense of actually determining some ready state. But having those "stub" endpoints and handlers implemented, is somewhat of an interface or blueprint for derived implementations to #put some code here#

Getting to the openai API server:

If we agree that this feature should be implemented somehow, what would be a good sources / starting point to determine vLLM is ready to receive requests? So to follow my initial thoughts, how do I determine if the model is downloaded and loaded / imported into the GPU, KV is setup?

Coming to your question (#6073 (comment)) @robertgshaw2-neuralmagic, I might be able to come up with a PR, but would love some more discussion on the implementation and "quality" of the reported status (be it vLLMs health or readiness). As for health, I'd like to expose as much of a proper health check as possible. Round-tripping a static API endpoint is nice, but does not go much beyond taking the existence of a running process as health check. What about running out of CUDA memory or other issues at runtime? Do they always ultimately cause the process to crash / end itself? Or would some more internal checking help to see if vLLM is actually still alive?

Regarding your question "So to follow my initial thoughts, how do I determine if the model is downloaded and loaded / imported into the GPU, KV is setup?", I have tried to used in this PR #7078 the "model_memory_usage" variable in model_runner object to determine the readiness once the model weights are loaded in the GPU memory.

from vllm.