[Bug]: prefix-caching: inconsistent completions about vllm HOT 5 OPEN

Thanks for the reply @cadedaniel.
I tried now with --use-v2-block-manager (version 0.5.0.post1) and it still happens unfortunately.

Edit: Tried also building current main branch (commit e2b85cf) where #5364 is already merged, and the issue still happens (also with --use-v2-block-manager)

from vllm.

Thanks @colefranks
I tried and seems that the workaround doesn't seem to help but it does change the behavior, tried several combinations (all with version 0.5.0.post1).

On first iteration, there is difference in outputs between VLLM_ATTENTION_BACKEND=XFORMERS and without. And if we assume that's ok, anyway when --enable-prefix-caching is used, than second iteration with --enable-prefix-caching differs from the first one.

from vllm.

We have an improved block manager which has better test coverage for prefix caching. We have tests which compare equality of prefix caching vs non-prefix caching -- so this case shouldn't happen // if it is happening, we can more easily diagnose the failure. Note the v2 block manager is not yet optimized for performance.

Can you see if it occurs with --use-block-manager-v2?

from vllm.

Built also the branch of #5188 and it doesn't resolve the issue

from vllm.

possible workaround #5376 (comment)

from vllm.