[Feature]: add local_files_only parameter about vllm HOT 4 CLOSED

@njhill thanks. it works.
version 0.4.2
llm = LLM(model= model_local_path, dtype='float16', HF_HUB_OFFLINE=1)

from vllm.

@yananchen1989 what version are you using? Changes were made recently to not require a connection to HF hub if the model is in the local cache. And you can prevent it from attempting to connect at all by setting HF_HUB_OFFLINE=1.

from vllm.

@yananchen1989 actually HF_HUB_OFFLINE=1 is an env variable, not an arg passed to the LLM constructor.

You should be able to remove that, it will still work because it should still fall back to local cache if it can't contact HF hub.

from vllm.

My vllm version is 0.5.0.post1, and it shows me an error when I don't use HF_HUB_OFFLINE=1. Maybe it is because I'm using a private repository. And I'm using AsyncLLMEngine, so it is another possibility that causes an error.

from vllm.