[Bug]: OOM when setting prompt_logprobs=1 about vllm HOT 3 OPEN

Something like this: model = LLM(..., enable_chunked_prefill=True, max_num_batched_tokens=512, gpu_memory_utilization=0.9)

Try smaller values of gpu_memory_utilization and/or max_num_batched_tokens if you still see OOM.

from vllm.

There is a pending PR trying to address this problem: #5355.

Meanwhile, you can try the chunked prefill feature which worked for me as a workaround: https://docs.vllm.ai/en/latest/models/performance.html#chunked-prefill.

from vllm.

would you mind sharing your code? let's say i have n_prompts=10, and set prompt_logprobs=0, i'd ideally get the logprobs for all 10 prompts using a single call model.generate(prompts=prompts, sampling_params=sampling_params).

from vllm.