Comments (8)
youkaichao
commented on July 4, 2024
1
can you follow https://docs.vllm.ai/en/latest/getting_started/debugging.html to investigate which Python function causes the crash?
from vllm.
simon-mo
commented on July 4, 2024
1
Also please try the latest patch v0.5.0.post1 which might fix one of the root cause
from vllm.
mpoemsl
commented on July 4, 2024
Thanks for the quick response from both of you! Through experiments, I managed to further narrow the issue down - it only occurs when enable_prefix_caching=True and only the first time a prefix is re-used.
@simon-mo I'm already using the pip version v0.5.0.post1
@youkaichao Here's the output with the env vars from your link set:
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] Error executing method start_worker_execution_loop. This might cause deadlock in distributed execution.
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] Traceback (most recent call last):
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] File "/usr/local/lib/python3.11/dist-packages/vllm/worker/worker_base.py", line 140, in execute_method
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] return executor(*args, **kwargs)
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] ^^^^^^^^^^^^^^^^^^^^^^^^^
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] File "/usr/local/lib/python3.11/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] return func(*args, **kwargs)
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] ^^^^^^^^^^^^^^^^^^^^^
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] File "/usr/local/lib/python3.11/dist-packages/vllm/worker/worker.py", line 294, in start_worker_execution_loop
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] while self._execute_model_non_driver():
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] File "/usr/local/lib/python3.11/dist-packages/vllm/worker/worker.py", line 317, in _execute_model_non_driver
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] self.model_runner.execute_model(None, self.gpu_cache)
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] File "/usr/local/lib/python3.11/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] return func(*args, **kwargs)
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] ^^^^^^^^^^^^^^^^^^^^^
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] File "/usr/local/lib/python3.11/dist-packages/vllm/worker/model_runner.py", line 749, in execute_model
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] hidden_states = model_executable(
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] ^^^^^^^^^^^^^^^^^
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] return self._call_impl(*args, **kwargs)
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] return forward_call(*args, **kwargs)
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] File "/usr/local/lib/python3.11/dist-packages/vllm/model_executor/models/mixtral.py", line 535, in forward
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] hidden_states = self.model(input_ids, positions, kv_caches,
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] return self._call_impl(*args, **kwargs)
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] return forward_call(*args, **kwargs)
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] File "/usr/local/lib/python3.11/dist-packages/vllm/model_executor/models/mixtral.py", line 468, in forward
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] hidden_states, residual = layer(positions, hidden_states,
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] return self._call_impl(*args, **kwargs)
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] return forward_call(*args, **kwargs)
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] File "/usr/local/lib/python3.11/dist-packages/vllm/model_executor/models/mixtral.py", line 414, in forward
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] hidden_states = self.self_attn(
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] ^^^^^^^^^^^^^^^
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] return self._call_impl(*args, **kwargs)
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] return forward_call(*args, **kwargs)
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] File "/usr/local/lib/python3.11/dist-packages/vllm/model_executor/models/mixtral.py", line 363, in forward
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] attn_output = self.attn(q, k, v, kv_cache, attn_metadata)
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] return self._call_impl(*args, **kwargs)
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] return forward_call(*args, **kwargs)
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] File "/usr/local/lib/python3.11/dist-packages/vllm/attention/layer.py", line 89, in forward
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] return self.impl.forward(query, key, value, kv_cache, attn_metadata,
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] File "/usr/local/lib/python3.11/dist-packages/vllm/attention/backends/flash_attn.py", line 339, in forward
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] output[:num_prefill_tokens] = flash_attn_varlen_func(
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] ^^^^^^^^^^^^^^^^^^^^^^^
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] File "/usr/local/lib/python3.11/dist-packages/vllm_flash_attn/flash_attn_interface.py", line 1099, in flash_attn_varlen_func
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] return FlashAttnVarlenFunc.apply(
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] ^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] File "/usr/local/lib/python3.11/dist-packages/torch/autograd/function.py", line 598, in apply
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] return super().apply(*args, **kwargs) # type: ignore[misc]
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] File "/usr/local/lib/python3.11/dist-packages/vllm_flash_attn/flash_attn_interface.py", line 596, in forward
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] out, q, k, v, out_padded, softmax_lse, S_dmask, rng_state = _flash_attn_varlen_forward(
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] File "/usr/local/lib/python3.11/dist-packages/vllm_flash_attn/flash_attn_interface.py", line 88, in _flash_attn_varlen_forward
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] out, q, k, v, out_padded, softmax_lse, S_dmask, rng_state = flash_attn_cuda.varlen_fwd(
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] RuntimeError: CUDA error: an illegal memory access was encountered
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14863) INFO 06-14 17:52:48 model_runner.py:965] Graph capturing finished in 129 secs. [repeated 2x across cluster]
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743) [rank2]:[E ProcessGroupNCCL.cpp:1414] [PG 2 Rank 2] Process group watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743) Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743)
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743) Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743) frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f8140773897 in /usr/local/lib/python3.11/dist-packages/torch/lib/libc10.so)
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743) frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7f8140723b25 in /usr/local/lib/python3.11/dist-packages/torch/lib/libc10.so)
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743) frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7f81580c5718 in /usr/local/lib/python3.11/dist-packages/torch/lib/libc10_cuda.so)
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743) frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7f51b1978e36 in /usr/local/lib/python3.11/dist-packages/torch/lib/libtorch_cuda.so)
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743) frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0x58 (0x7f51b197cf38 in /usr/local/lib/python3.11/dist-packages/torch/lib/libtorch_cuda.so)
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743) frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x77c (0x7f51b19825ac in /usr/local/lib/python3.11/dist-packages/torch/lib/libtorch_cuda.so)
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743) frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7f51b198331c in /usr/local/lib/python3.11/dist-packages/torch/lib/libtorch_cuda.so)
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743) frame #7: <unknown function> + 0xd6df4 (0x7f816b330df4 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743) frame #8: <unknown function> + 0x8609 (0x7f816d250609 in /usr/lib/x86_64-linux-gnu/libpthread.so.0)
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743) frame #9: clone + 0x43 (0x7f816d38a353 in /usr/lib/x86_64-linux-gnu/libc.so.6)
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743)
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743) [2024-06-14 17:57:30,390 E 14743 15025] logging.cc:101: Unhandled exception: N3c1016DistBackendErrorE. what(): [PG 2 Rank 2] Process group watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743) Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743)
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743) Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743) frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f8140773897 in /usr/local/lib/python3.11/dist-packages/torch/lib/libc10.so)
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743) frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7f8140723b25 in /usr/local/lib/python3.11/dist-packages/torch/lib/libc10.so)
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743) frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7f81580c5718 in /usr/local/lib/python3.11/dist-packages/torch/lib/libc10_cuda.so)
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743) frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7f51b1978e36 in /usr/local/lib/python3.11/dist-packages/torch/lib/libtorch_cuda.so)
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743) frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0x58 (0x7f51b197cf38 in /usr/local/lib/python3.11/dist-packages/torch/lib/libtorch_cuda.so)
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743) frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x77c (0x7f51b19825ac in /usr/local/lib/python3.11/dist-packages/torch/lib/libtorch_cuda.so)
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743) frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7f51b198331c in /usr/local/lib/python3.11/dist-packages/torch/lib/libtorch_cuda.so)
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743) frame #7: <unknown function> + 0xd6df4 (0x7f816b330df4 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743) frame #8: <unknown function> + 0x8609 (0x7f816d250609 in /usr/lib/x86_64-linux-gnu/libpthread.so.0)
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743) frame #9: clone + 0x43 (0x7f816d38a353 in /usr/lib/x86_64-linux-gnu/libc.so.6)
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743)
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743) Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1418 (most recent call first):
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743) frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f8140773897 in /usr/local/lib/python3.11/dist-packages/torch/lib/libc10.so)
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743) frame #1: <unknown function> + 0xe32e33 (0x7f51b1605e33 in /usr/local/lib/python3.11/dist-packages/torch/lib/libtorch_cuda.so)
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743) frame #2: <unknown function> + 0xd6df4 (0x7f816b330df4 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743) frame #3: <unknown function> + 0x8609 (0x7f816d250609 in /usr/lib/x86_64-linux-gnu/libpthread.so.0)
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743) frame #4: clone + 0x43 (0x7f816d38a353 in /usr/lib/x86_64-linux-gnu/libc.so.6)
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743)
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743) [2024-06-14 17:57:30,413 E 14743 15025] logging.cc:108: Stack trace:
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743) /usr/local/lib/python3.11/dist-packages/ray/_raylet.so(+0x101867a) [0x7f816c45e67a] ray::operator<<()
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743) /usr/local/lib/python3.11/dist-packages/ray/_raylet.so(+0x101b138) [0x7f816c461138] ray::TerminateHandler()
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743) /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xaa37c) [0x7f816b30437c]
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743) /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xaa3e7) [0x7f816b3043e7]
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743) /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xaa36f) [0x7f816b30436f]
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743) /usr/local/lib/python3.11/dist-packages/torch/lib/libtorch_cuda.so(+0xe32ee4) [0x7f51b1605ee4] c10d::ProcessGroupNCCL::ncclCommWatchdog()
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743) /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xd6df4) [0x7f816b330df4]
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743) /usr/lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f816d250609] start_thread
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743) /usr/lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7f816d38a353] __clone
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743)
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743) *** SIGABRT received at time=1718387850 on cpu 221 ***
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743) PC: @ 0x7f816d2ae00b (unknown) raise
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743) @ 0x7f816d2ae090 2192 (unknown)
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743) @ 0x7f816b30437c (unknown) (unknown)
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743) @ 0x7f816b304090 (unknown) (unknown)
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743) [2024-06-14 17:57:30,413 E 14743 15025] logging.cc:365: *** SIGABRT received at time=1718387850 on cpu 221 ***
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743) [2024-06-14 17:57:30,413 E 14743 15025] logging.cc:365: PC: @ 0x7f816d2ae00b (unknown) raise
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743) [2024-06-14 17:57:30,414 E 14743 15025] logging.cc:365: @ 0x7f816d2ae090 2192 (unknown)
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743) [2024-06-14 17:57:30,414 E 14743 15025] logging.cc:365: @ 0x7f816b30437c (unknown) (unknown)
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743) [2024-06-14 17:57:30,414 E 14743 15025] logging.cc:365: @ 0x7f816b304090 (unknown) (unknown)
llm-inference-mixtral-20240612-2-1 | (RayWorkerWrapper pid=14743) Fatal Python error: Aborted
from vllm.
youkaichao
commented on July 4, 2024
@robertgshaw2-neuralmagic for prefix caching.
from vllm.
mpoemsl
commented on July 4, 2024
Will try this when I have the chance: #5376 (comment)
from vllm.
ashgold
commented on July 4, 2024
This is happening to me as well.
GPU: H100
tp: 4
engine: ray
model: llama-65b fine-tuned model
In vLLM v0.5.0.post1, applying fp8 quantized model and --enable-prefix-caching at the same time causes this symptom.
Using only fp8 quantized model, or non quantized model and --enable-prefix-caching option does not cause this symptom.
Also, it doesn't happen 100% of the time on every inference, it happens suddenly in the middle of a load.
Here's the error log I have.
[rank2]:[E ProcessGroupNCCL.cpp:1414] [PG 2 Rank 2] Process group watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fdd9c09e897 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7fdd9c04eb25 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7fdd9c176718 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7fdd4fe4ae36 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0x58 (0x7fdd4fe4ef38 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x77c (0x7fdd4fe545ac in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7fdd4fe5531c in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #7: <unknown function> + 0xdc253 (0x7fdd9b8b0253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
frame #8: <unknown function> + 0x94ac3 (0x7fdd9cd50ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #9: <unknown function> + 0x126850 (0x7fdd9cde2850 in /usr/lib/x86_64-linux-gnu/libc.so.6)
[rank1]:[E ProcessGroupNCCL.cpp:1414] [PG 2 Rank 1] Process group watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fb3ecc9e897 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7fb3ecc4eb25 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7fb3ecd76718 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7fb3a0a4ae36 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0x58 (0x7fb3a0a4ef38 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x77c (0x7fb3a0a545ac in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7fb3a0a5531c in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #7: <unknown function> + 0xdc253 (0x7fb3ec4b0253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
frame #8: <unknown function> + 0x94ac3 (0x7fb3ed957ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #9: <unknown function> + 0x126850 (0x7fb3ed9e9850 in /usr/lib/x86_64-linux-gnu/libc.so.6)
terminate called after throwing an instance of 'c10::DistBackendError'
what(): [PG 2 Rank 2] Process group watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fdd9c09e897 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7fdd9c04eb25 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7fdd9c176718 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7fdd4fe4ae36 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0x58 (0x7fdd4fe4ef38 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x77c (0x7fdd4fe545ac in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7fdd4fe5531c in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #7: <unknown function> + 0xdc253 (0x7fdd9b8b0253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
frame #8: <unknown function> + 0x94ac3 (0x7fdd9cd50ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #9: <unknown function> + 0x126850 (0x7fdd9cde2850 in /usr/lib/x86_64-linux-gnu/libc.so.6)
Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1418 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fdd9c09e897 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0xe32e33 (0x7fdd4fad7e33 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #2: <unknown function> + 0xdc253 (0x7fdd9b8b0253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
frame #3: <unknown function> + 0x94ac3 (0x7fdd9cd50ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #4: <unknown function> + 0x126850 (0x7fdd9cde2850 in /usr/lib/x86_64-linux-gnu/libc.so.6)
terminate called after throwing an instance of 'c10::DistBackendError'
what(): [PG 2 Rank 1] Process group watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fb3ecc9e897 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7fb3ecc4eb25 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7fb3ecd76718 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7fb3a0a4ae36 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0x58 (0x7fb3a0a4ef38 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x77c (0x7fb3a0a545ac in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7fb3a0a5531c in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #7: <unknown function> + 0xdc253 (0x7fb3ec4b0253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
frame #8: <unknown function> + 0x94ac3 (0x7fb3ed957ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #9: <unknown function> + 0x126850 (0x7fb3ed9e9850 in /usr/lib/x86_64-linux-gnu/libc.so.6)
Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1418 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fb3ecc9e897 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0xe32e33 (0x7fb3a06d7e33 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #2: <unknown function> + 0xdc253 (0x7fb3ec4b0253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
frame #3: <unknown function> + 0x94ac3 (0x7fb3ed957ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #4: <unknown function> + 0x126850 (0x7fb3ed9e9850 in /usr/lib/x86_64-linux-gnu/libc.so.6)
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] Exception in worker VllmWorkerProcess while processing method start_worker_execution_loop: CUDA error: an illegal memory access was encountered
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] , Traceback (most recent call last):
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] File "/usr/local/lib/python3.10/dist-packages/vllm/executor/multiproc_worker_utils.py", line 223, in _run_worker_process
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] output = executor(*args, **kwargs)
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] return func(*args, **kwargs)
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker.py", line 294, in start_worker_execution_loop
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] while self._execute_model_non_driver():
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker.py", line 317, in _execute_model_non_driver
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] self.model_runner.execute_model(None, self.gpu_cache)
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] return func(*args, **kwargs)
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] File "/usr/local/lib/python3.10/dist-packages/vllm/worker/model_runner.py", line 749, in execute_model
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] hidden_states = model_executable(
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] return self._call_impl(*args, **kwargs)
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] return forward_call(*args, **kwargs)
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/llama.py", line 371, in forward
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] hidden_states = self.model(input_ids, positions, kv_caches,
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] return self._call_impl(*args, **kwargs)
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] return forward_call(*args, **kwargs)
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/llama.py", line 288, in forward
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] hidden_states, residual = layer(
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] return self._call_impl(*args, **kwargs)
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] return forward_call(*args, **kwargs)
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/llama.py", line 227, in forward
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] hidden_states = self.self_attn(
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] return self._call_impl(*args, **kwargs)
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] return forward_call(*args, **kwargs)
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/llama.py", line 161, in forward
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] attn_output = self.attn(q, k, v, kv_cache, attn_metadata)
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] return self._call_impl(*args, **kwargs)
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] return forward_call(*args, **kwargs)
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] File "/usr/local/lib/python3.10/dist-packages/vllm/attention/layer.py", line 89, in forward
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] return self.impl.forward(query, key, value, kv_cache, attn_metadata,
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] File "/usr/local/lib/python3.10/dist-packages/vllm/attention/backends/flash_attn.py", line 339, in forward
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] output[:num_prefill_tokens] = flash_attn_varlen_func(
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] File "/usr/local/lib/python3.10/dist-packages/vllm_flash_attn/flash_attn_interface.py", line 1099, in flash_attn_varlen_func
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] return FlashAttnVarlenFunc.apply(
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] File "/usr/local/lib/python3.10/dist-packages/torch/autograd/function.py", line 598, in apply
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] return super().apply(*args, **kwargs) # type: ignore[misc]
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] File "/usr/local/lib/python3.10/dist-packages/vllm_flash_attn/flash_attn_interface.py", line 596, in forward
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] out, q, k, v, out_padded, softmax_lse, S_dmask, rng_state = _flash_attn_varlen_forward(
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] File "/usr/local/lib/python3.10/dist-packages/vllm_flash_attn/flash_attn_interface.py", line 88, in _flash_attn_varlen_forward
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] out, q, k, v, out_padded, softmax_lse, S_dmask, rng_state = flash_attn_cuda.varlen_fwd(
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] RuntimeError: CUDA error: an illegal memory access was encountered
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]
[rank3]:[E ProcessGroupNCCL.cpp:1414] [PG 2 Rank 3] Process group watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f600937a897 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7f600932ab25 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7f60097ac718 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7f5fbd04ae36 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0x58 (0x7f5fbd04ef38 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x77c (0x7f5fbd0545ac in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7f5fbd05531c in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #7: <unknown function> + 0xdc253 (0x7f6008ab0253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
frame #8: <unknown function> + 0x94ac3 (0x7f600a0beac3 in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #9: <unknown function> + 0x126850 (0x7f600a150850 in /usr/lib/x86_64-linux-gnu/libc.so.6)
terminate called after throwing an instance of 'c10::DistBackendError'
what(): [PG 2 Rank 3] Process group watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f600937a897 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7f600932ab25 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7f60097ac718 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7f5fbd04ae36 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0x58 (0x7f5fbd04ef38 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x77c (0x7f5fbd0545ac in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7f5fbd05531c in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #7: <unknown function> + 0xdc253 (0x7f6008ab0253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
frame #8: <unknown function> + 0x94ac3 (0x7f600a0beac3 in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #9: <unknown function> + 0x126850 (0x7f600a150850 in /usr/lib/x86_64-linux-gnu/libc.so.6)
Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1418 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f600937a897 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0xe32e33 (0x7f5fbccd7e33 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #2: <unknown function> + 0xdc253 (0x7f6008ab0253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
frame #3: <unknown function> + 0x94ac3 (0x7f600a0beac3 in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #4: <unknown function> + 0x126850 (0x7f600a150850 in /usr/lib/x86_64-linux-gnu/libc.so.6)
ERROR 06-20 14:08:58 async_llm_engine.py:535] Engine iteration timed out. This should never happen!
ERROR 06-20 14:08:58 async_llm_engine.py:52] Engine background task failed
ERROR 06-20 14:08:58 async_llm_engine.py:52] Traceback (most recent call last):
ERROR 06-20 14:08:58 async_llm_engine.py:52] File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 506, in engine_step
ERROR 06-20 14:08:58 async_llm_engine.py:52] request_outputs = await self.engine.step_async()
ERROR 06-20 14:08:58 async_llm_engine.py:52] File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 235, in step_async
ERROR 06-20 14:08:58 async_llm_engine.py:52] output = await self.model_executor.execute_model_async(
ERROR 06-20 14:08:58 async_llm_engine.py:52] File "/usr/local/lib/python3.10/dist-packages/vllm/executor/distributed_gpu_executor.py", line 166, in execute_model_async
ERROR 06-20 14:08:58 async_llm_engine.py:52] return await self._driver_execute_model_async(execute_model_req)
ERROR 06-20 14:08:58 async_llm_engine.py:52] File "/usr/local/lib/python3.10/dist-packages/vllm/executor/multiproc_gpu_executor.py", line 149, in _driver_execute_model_async
ERROR 06-20 14:08:58 async_llm_engine.py:52] return await self.driver_exec_model(execute_model_req)
ERROR 06-20 14:08:58 async_llm_engine.py:52] asyncio.exceptions.CancelledError
ERROR 06-20 14:08:58 async_llm_engine.py:52]
ERROR 06-20 14:08:58 async_llm_engine.py:52] During handling of the above exception, another exception occurred:
ERROR 06-20 14:08:58 async_llm_engine.py:52]
ERROR 06-20 14:08:58 async_llm_engine.py:52] Traceback (most recent call last):
ERROR 06-20 14:08:58 async_llm_engine.py:52] File "/usr/lib/python3.10/asyncio/tasks.py", line 456, in wait_for
ERROR 06-20 14:08:58 async_llm_engine.py:52] return fut.result()
ERROR 06-20 14:08:58 async_llm_engine.py:52] asyncio.exceptions.CancelledError
ERROR 06-20 14:08:58 async_llm_engine.py:52]
ERROR 06-20 14:08:58 async_llm_engine.py:52] The above exception was the direct cause of the following exception:
ERROR 06-20 14:08:58 async_llm_engine.py:52]
ERROR 06-20 14:08:58 async_llm_engine.py:52] Traceback (most recent call last):
ERROR 06-20 14:08:58 async_llm_engine.py:52] File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 42, in _log_task_completion
ERROR 06-20 14:08:58 async_llm_engine.py:52] return_value = task.result()
ERROR 06-20 14:08:58 async_llm_engine.py:52] File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 532, in run_engine_loop
ERROR 06-20 14:08:58 async_llm_engine.py:52] has_requests_in_progress = await asyncio.wait_for(
ERROR 06-20 14:08:58 async_llm_engine.py:52] File "/usr/lib/python3.10/asyncio/tasks.py", line 458, in wait_for
ERROR 06-20 14:08:58 async_llm_engine.py:52] raise exceptions.TimeoutError() from exc
ERROR 06-20 14:08:58 async_llm_engine.py:52] asyncio.exceptions.TimeoutError
ERROR:asyncio:Exception in callback functools.partial(<function _log_task_completion at 0x7f8796cbea70>, error_callback=<bound method AsyncLLMEngine._error_callback of <vllm.engine.async_llm_engine.AsyncLLMEngine object at 0x7f884971aad0>>)
handle: <Handle functools.partial(<function _log_task_completion at 0x7f8796cbea70>, error_callback=<bound method AsyncLLMEngine._error_callback of <vllm.engine.async_llm_engine.AsyncLLMEngine object at 0x7f884971aad0>>)>
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 506, in engine_step
request_outputs = await self.engine.step_async()
File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 235, in step_async
output = await self.model_executor.execute_model_async(
File "/usr/local/lib/python3.10/dist-packages/vllm/executor/distributed_gpu_executor.py", line 166, in execute_model_async
return await self._driver_execute_model_async(execute_model_req)
File "/usr/local/lib/python3.10/dist-packages/vllm/executor/multiproc_gpu_executor.py", line 149, in _driver_execute_model_async
return await self.driver_exec_model(execute_model_req)
asyncio.exceptions.CancelledError
from vllm.
robertgshaw2-neuralmagic
commented on July 4, 2024
@ashgold does the model have quantized KV cache or just the layers?
from vllm.
ashgold
commented on July 4, 2024
@robertgshaw2-neuralmagic
just layers, not with --kv-cache-dtype argument.
from vllm.
Related Issues (20)
- [Bug]: Model "talking to itself" and ignoring `<|im_end|>` HOT 13
- [New Model]: Florence-2
- Virtual Office Hours: July 9 and July 25
- [Bug]: Illegal memory access for MoE kernel with large workloads HOT 1
- [Bug]: "work_use_ray" not work anymore in the latest version HOT 2
- [Usage]: can I save log to a file? HOT 1
- [Usage]: 是否可以多节点多CPU推理 HOT 1
- [Feature]: Way to using LLM's last hidden state embedding vector
- [Bug]: Can't support Phi-3-medium-* models with more than 2 GPUs
- [Bug]: Chunked prefill vs. non-chunked output is different for a long prompt HOT 1
- Gemma2 models from google HOT 3
- [Bug]: qwen1.5-32b-chat no response HOT 1
- [Feature]: `/info` endpoint for OpenAI-compatible API Server HOT 1
- [Bug]: vLLM crash when running Phi-3-small-8k-instruct with enable-chunked-prefill HOT 3
- [Bug]: Phi-3 vision crash: TypeError: only integer tensors of a single element can be converted to an index HOT 2
- [Bug]: New bug in last few days for phi-3-vision. The model's max seq len (131072) is larger than the maximum number of tokens that can be stored in KV cache (50944) HOT 5
- [Bug]: AttributeError: 'NoneType' object has no attribute 'prefill_metadata' HOT 1
- [Bug]: TypeError: FlashAttentionMetadata.__init__() missing 10 required positional arguments HOT 5
- about the RotaryEmbedding
- [New Model]: support for BartForSequenceClassification
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from vllm.