Comments (15)
youkaichao
commented on June 26, 2024
1
Quote from issue templates:
Please set the environment variable export VLLM_LOGGING_LEVEL=DEBUG to turn on more logging to help debugging potential issues.
If you experienced crashes or hangs, it would be helpful to run vllm with export VLLM_TRACE_FUNCTION=1 . All the function calls in vllm will be recorded. Inspect these log files, and tell which function crashes or hangs.
from vllm.
jayteaftw
commented on June 26, 2024
1
Quote from issue templates:
Please set the environment variable export VLLM_LOGGING_LEVEL=DEBUG to turn on more logging to help debugging potential issues.
If you experienced crashes or hangs, it would be helpful to run vllm with export VLLM_TRACE_FUNCTION=1 . All the function calls in vllm will be recorded. Inspect these log files, and tell which function crashes or hangs.
Thank I updated the env variables
I see 4 different log files
VLLM_TRACE_FUNCTION_for_process_1_thread_140370429153728_at_2024-06-09_00:57:22.494711.log VLLM_TRACE_FUNCTION_for_process_75_thread_139899367563712_at_2024-06-09_00:57:24.545374.log
VLLM_TRACE_FUNCTION_for_process_74_thread_140356664910272_at_2024-06-09_00:57:24.580614.log VLLM_TRACE_FUNCTION_for_process_76_thread_140102125220288_at_2024-06-09_00:57:24.507486.log
VLLM_TRACE_FUNCTION_for_process_76_thread.log
This is the output from towards the end ofVLLM_TRACE_FUNCTION_for_process_76_thread_140102125220288_at_2024-06-09_00:57:24.507486.log
2024-06-09 00:57:25.047238 Return from is_initialized in /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:975 to _get_default_group in /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:1007
2024-06-09 00:57:25.047261 Call to WORLD in /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:588 from _get_default_group in /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:1012
2024-06-09 00:57:25.047279 Call to default_pg in /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:460 from WORLD in /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:590
2024-06-09 00:57:25.047296 Return from default_pg in /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:468 to WORLD in /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:590
2024-06-09 00:57:25.047315 Return from WORLD in /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:590 to _get_default_group in /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:1012
2024-06-09 00:57:25.047332 Call to not_none in /usr/local/lib/python3.10/dist-packages/torch/utils/_typing_utils.py:10 from _get_default_group in /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:1012
2024-06-09 00:57:25.047350 Return from not_none in /usr/local/lib/python3.10/dist-packages/torch/utils/_typing_utils.py:13 to _get_default_group in /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:1012
2024-06-09 00:57:25.047367 Return from _get_default_group in /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:1012 to get_rank in /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:1746
2024-06-09 00:57:25.047385 Return from get_rank in /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:1748 to _get_msg_dict in /usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py:56
2024-06-09 00:57:25.047412 Call to version in /usr/local/lib/python3.10/dist-packages/torch/cuda/nccl.py:34 from _get_msg_dict in /usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py:59
2024-06-09 00:57:25.047440 Return from version in /usr/local/lib/python3.10/dist-packages/torch/cuda/nccl.py:41 to _get_msg_dict in /usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py:59
2024-06-09 00:57:25.047460 Call to <genexpr> in /usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py:60 from _get_msg_dict in /usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py:60
2024-06-09 00:57:25.047478 Return from <genexpr> in /usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py:60 to _get_msg_dict in /usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py:60
2024-06-09 00:57:25.047495 Call to <genexpr> in /usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py:60 from _get_msg_dict in /usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py:60
2024-06-09 00:57:25.047512 Return from <genexpr> in /usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py:60 to _get_msg_dict in /usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py:60
2024-06-09 00:57:25.047529 Call to <genexpr> in /usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py:60 from _get_msg_dict in /usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py:60
2024-06-09 00:57:25.047546 Return from <genexpr> in /usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py:60 to _get_msg_dict in /usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py:60
2024-06-09 00:57:25.047564 Call to <genexpr> in /usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py:60 from _get_msg_dict in /usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py:60
2024-06-09 00:57:25.047582 Return from <genexpr> in /usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py:60 to _get_msg_dict in /usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py:60
2024-06-09 00:57:25.047601 Return from _get_msg_dict in /usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py:66 to wrapper in /usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py:92
2024-06-09 00:57:25.047719 Return from wrapper in /usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py:96 to init_distributed_environment in /usr/local/lib/python3.10/dist-packages/vllm/distributed/parallel_state.py:104
2024-06-09 00:57:25.047884 Call to is_available in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:105 from init_distributed_environment in /usr/local/lib/python3.10/dist-packages/vllm/distributed/parallel_state.py:120
2024-06-09 00:57:25.047911 Call to _is_compiled in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:96 from is_available in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:107
2024-06-09 00:57:25.047931 Return from _is_compiled in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:98 to is_available in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:107
2024-06-09 00:57:25.047950 Call to _nvml_based_avail in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:101 from is_available in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:109
2024-06-09 00:57:25.047974 Return from _nvml_based_avail in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:102 to is_available in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:109
2024-06-09 00:57:25.047993 Return from is_available in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:118 to init_distributed_environment in /usr/local/lib/python3.10/dist-packages/vllm/distributed/parallel_state.py:120
2024-06-09 00:57:25.048472 Call to wrapper in /usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py:72 from init_distributed_environment in /usr/local/lib/python3.10/dist-packages/vllm/distributed/parallel_state.py:122
2024-06-09 00:57:25.048510 Call to all_reduce in /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:2146 from wrapper in /usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py:75
2024-06-09 00:57:25.048537 Call to _check_single_tensor in /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:860 from all_reduce in /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:2195
2024-06-09 00:57:25.048558 Return from _check_single_tensor in /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:862 to all_reduce in /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:2195
2024-06-09 00:57:25.048577 Call to _rank_not_in_group in /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:752 from all_reduce in /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:2196
2024-06-09 00:57:25.048595 Return from _rank_not_in_group in /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:755 to all_reduce in /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:2196
2024-06-09 00:57:25.048704 Call to _get_default_group in /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:1005 from all_reduce in /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:2208
2024-06-09 00:57:25.048723 Call to is_initialized in /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:973 from _get_default_group in /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:1007
2024-06-09 00:57:25.048742 Call to WORLD in /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:588 from is_initialized in /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:975
2024-06-09 00:57:25.048762 Call to default_pg in /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:460 from WORLD in /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:590
2024-06-09 00:57:25.048781 Return from default_pg in /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:468 to WORLD in /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:590
2024-06-09 00:57:25.048797 Return from WORLD in /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:590 to is_initialized in /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:975
2024-06-09 00:57:25.048815 Return from is_initialized in /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:975 to _get_default_group in /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:1007
2024-06-09 00:57:25.048835 Call to WORLD in /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:588 from _get_default_group in /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:1012
2024-06-09 00:57:25.048852 Call to default_pg in /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:460 from WORLD in /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:590
2024-06-09 00:57:25.048869 Return from default_pg in /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:468 to WORLD in /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:590
2024-06-09 00:57:25.048886 Return from WORLD in /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:590 to _get_default_group in /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:1012
2024-06-09 00:57:25.048903 Call to not_none in /usr/local/lib/python3.10/dist-packages/torch/utils/_typing_utils.py:10 from _get_default_group in /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:1012
2024-06-09 00:57:25.048921 Return from not_none in /usr/local/lib/python3.10/dist-packages/torch/utils/_typing_utils.py:13 to _get_default_group in /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:1012
2024-06-09 00:57:25.048937 Return from _get_default_group in /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:1012 to all_reduce in /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:2208
2024-06-09 00:57:25.048956 Call to pg_coalesce_state in /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:543 from all_reduce in /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:2210
2024-06-09 00:57:25.048975 Return from pg_coalesce_state in /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:545 to all_reduce in /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:2210
2024-06-09 00:57:25.182103 Return from all_reduce in /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:2224 to wrapper in /usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py:75
2024-06-09 00:57:25.182185 Return from wrapper in /usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py:75 to init_distributed_environment in /usr/local/lib/python3.10/dist-packages/vllm/distributed/parallel_state.py:122
2024-06-09 00:57:25.182215 Call to is_available in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:105 from init_distributed_environment in /usr/local/lib/python3.10/dist-packages/vllm/distributed/parallel_state.py:123
2024-06-09 00:57:25.182237 Call to _is_compiled in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:96 from is_available in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:107
2024-06-09 00:57:25.182267 Return from _is_compiled in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:98 to is_available in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:107
2024-06-09 00:57:25.182288 Call to _nvml_based_avail in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:101 from is_available in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:109
2024-06-09 00:57:25.182328 Return from _nvml_based_avail in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:102 to is_available in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:109
2024-06-09 00:57:25.182351 Return from is_available in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:118 to init_distributed_environment in /usr/local/lib/python3.10/dist-packages/vllm/distributed/parallel_state.py:123
2024-06-09 00:57:25.182372 Call to synchronize in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:782 from init_distributed_environment in /usr/local/lib/python3.10/dist-packages/vllm/distributed/parallel_state.py:124
2024-06-09 00:57:25.182396 Call to _lazy_init in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:263 from synchronize in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:790
2024-06-09 00:57:25.182416 Call to is_initialized in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:216 from _lazy_init in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:265
2024-06-09 00:57:25.182435 Return from is_initialized in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:218 to _lazy_init in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:265
2024-06-09 00:57:25.182454 Return from _lazy_init in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:266 to synchronize in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:790
2024-06-09 00:57:25.182476 Call to __init__ in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:360 from synchronize in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:791
2024-06-09 00:57:25.182498 Call to _get_device_index in /usr/local/lib/python3.10/dist-packages/torch/cuda/_utils.py:9 from __init__ in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:361
2024-06-09 00:57:25.182522 Call to is_scripting in /usr/local/lib/python3.10/dist-packages/torch/_jit_internal.py:1120 from _get_device_index in /usr/local/lib/python3.10/dist-packages/torch/cuda/_utils.py:35
2024-06-09 00:57:25.182540 Return from is_scripting in /usr/local/lib/python3.10/dist-packages/torch/_jit_internal.py:1139 to _get_device_index in /usr/local/lib/python3.10/dist-packages/torch/cuda/_utils.py:35
2024-06-09 00:57:25.182560 Call to _get_device_index in /usr/local/lib/python3.10/dist-packages/torch/_utils.py:759 from _get_device_index in /usr/local/lib/python3.10/dist-packages/torch/cuda/_utils.py:38
2024-06-09 00:57:25.182581 Call to is_scripting in /usr/local/lib/python3.10/dist-packages/torch/_jit_internal.py:1120 from _get_device_index in /usr/local/lib/python3.10/dist-packages/torch/_utils.py:793
2024-06-09 00:57:25.182599 Return from is_scripting in /usr/local/lib/python3.10/dist-packages/torch/_jit_internal.py:1139 to _get_device_index in /usr/local/lib/python3.10/dist-packages/torch/_utils.py:793
2024-06-09 00:57:25.182618 Call to _get_current_device_index in /usr/local/lib/python3.10/dist-packages/torch/_utils.py:733 from _get_device_index in /usr/local/lib/python3.10/dist-packages/torch/_utils.py:796
2024-06-09 00:57:25.182638 Call to _get_device_attr in /usr/local/lib/python3.10/dist-packages/torch/_utils.py:721 from _get_current_device_index in /usr/local/lib/python3.10/dist-packages/torch/_utils.py:735
2024-06-09 00:57:25.182657 Call to _get_available_device_type in /usr/local/lib/python3.10/dist-packages/torch/_utils.py:708 from _get_device_attr in /usr/local/lib/python3.10/dist-packages/torch/_utils.py:722
2024-06-09 00:57:25.182675 Call to is_available in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:105 from _get_available_device_type in /usr/local/lib/python3.10/dist-packages/torch/_utils.py:709
2024-06-09 00:57:25.182692 Call to _is_compiled in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:96 from is_available in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:107
2024-06-09 00:57:25.182710 Return from _is_compiled in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:98 to is_available in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:107
2024-06-09 00:57:25.182728 Call to _nvml_based_avail in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:101 from is_available in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:109
2024-06-09 00:57:25.182751 Return from _nvml_based_avail in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:102 to is_available in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:109
2024-06-09 00:57:25.182771 Return from is_available in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:118 to _get_available_device_type in /usr/local/lib/python3.10/dist-packages/torch/_utils.py:709
2024-06-09 00:57:25.182789 Return from _get_available_device_type in /usr/local/lib/python3.10/dist-packages/torch/_utils.py:710 to _get_device_attr in /usr/local/lib/python3.10/dist-packages/torch/_utils.py:722
2024-06-09 00:57:25.182808 Call to <lambda> in /usr/local/lib/python3.10/dist-packages/torch/_utils.py:735 from _get_device_attr in /usr/local/lib/python3.10/dist-packages/torch/_utils.py:724
2024-06-09 00:57:25.182827 Call to current_device in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:776 from <lambda> in /usr/local/lib/python3.10/dist-packages/torch/_utils.py:735
2024-06-09 00:57:25.182845 Call to _lazy_init in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:263 from current_device in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:778
2024-06-09 00:57:25.182863 Call to is_initialized in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:216 from _lazy_init in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:265
2024-06-09 00:57:25.182881 Return from is_initialized in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:218 to _lazy_init in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:265
2024-06-09 00:57:25.182900 Return from _lazy_init in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:266 to current_device in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:778
2024-06-09 00:57:25.182922 Return from current_device in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:779 to <lambda> in /usr/local/lib/python3.10/dist-packages/torch/_utils.py:735
2024-06-09 00:57:25.182939 Return from <lambda> in /usr/local/lib/python3.10/dist-packages/torch/_utils.py:735 to _get_device_attr in /usr/local/lib/python3.10/dist-packages/torch/_utils.py:724
2024-06-09 00:57:25.182956 Return from _get_device_attr in /usr/local/lib/python3.10/dist-packages/torch/_utils.py:724 to _get_current_device_index in /usr/local/lib/python3.10/dist-packages/torch/_utils.py:735
2024-06-09 00:57:25.182974 Return from _get_current_device_index in /usr/local/lib/python3.10/dist-packages/torch/_utils.py:735 to _get_device_index in /usr/local/lib/python3.10/dist-packages/torch/_utils.py:796
2024-06-09 00:57:25.182991 Return from _get_device_index in /usr/local/lib/python3.10/dist-packages/torch/_utils.py:801 to _get_device_index in /usr/local/lib/python3.10/dist-packages/torch/cuda/_utils.py:38
2024-06-09 00:57:25.183009 Return from _get_device_index in /usr/local/lib/python3.10/dist-packages/torch/cuda/_utils.py:38 to __init__ in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:361
2024-06-09 00:57:25.183030 Return from __init__ in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:362 to synchronize in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:791
2024-06-09 00:57:25.183050 Call to __enter__ in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:364 from synchronize in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:791
2024-06-09 00:57:25.183071 Return from __enter__ in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:365 to synchronize in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:791
Unsure of where the error would be
from vllm.
youkaichao
commented on June 26, 2024
1
seems to be a pytorch/cuda initialization problem. you can try to run:
# test.py
import torch
import torch.distributed as dist
dist.init_process_group(backend="nccl")
data = torch.ByteTensor([1,] * 128).to(f"cuda:{dist.get_rank()}")
dist.all_reduce(data, op=dist.ReduceOp.SUM)with
export NCCL_DEBUG=TRACE
torchrun --nproc-per-node=4 test.pyto see if it works.
from vllm.
jayteaftw
commented on June 26, 2024
1
Okay I ran the test.py in the container and got this output
W0609 01:26:10.258000 140697551208896 torch/distributed/run.py:757]
W0609 01:26:10.258000 140697551208896 torch/distributed/run.py:757] *****************************************
W0609 01:26:10.258000 140697551208896 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
W0609 01:26:10.258000 140697551208896 torch/distributed/run.py:757] *****************************************
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:428:428 [0] NCCL INFO Bootstrap : Using eth0:10.128.0.68<0>
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:428:428 [0] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:428:428 [0] NCCL INFO cudaDriverVersion 12040
NCCL version 2.20.5+cuda12.4
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:431:431 [3] NCCL INFO cudaDriverVersion 12040
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:431:431 [3] NCCL INFO Bootstrap : Using eth0:10.128.0.68<0>
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:431:431 [3] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:430:430 [2] NCCL INFO cudaDriverVersion 12040
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:430:430 [2] NCCL INFO Bootstrap : Using eth0:10.128.0.68<0>
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:430:430 [2] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:429:429 [1] NCCL INFO cudaDriverVersion 12040
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:429:429 [1] NCCL INFO Bootstrap : Using eth0:10.128.0.68<0>
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:429:429 [1] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:428:452 [0] NCCL INFO Failed to open libibverbs.so[.1]
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:428:452 [0] NCCL INFO NET/Socket : Using [0]eth0:10.128.0.68<0>
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:428:452 [0] NCCL INFO Using non-device net plugin version 0
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:428:452 [0] NCCL INFO Using network Socket
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:431:453 [3] NCCL INFO Failed to open libibverbs.so[.1]
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:431:453 [3] NCCL INFO NET/Socket : Using [0]eth0:10.128.0.68<0>
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:431:453 [3] NCCL INFO Using non-device net plugin version 0
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:431:453 [3] NCCL INFO Using network Socket
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:430:454 [2] NCCL INFO Failed to open libibverbs.so[.1]
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:430:454 [2] NCCL INFO NET/Socket : Using [0]eth0:10.128.0.68<0>
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:430:454 [2] NCCL INFO Using non-device net plugin version 0
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:430:454 [2] NCCL INFO Using network Socket
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:429:455 [1] NCCL INFO Failed to open libibverbs.so[.1]
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:429:455 [1] NCCL INFO NET/Socket : Using [0]eth0:10.128.0.68<0>
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:429:455 [1] NCCL INFO Using non-device net plugin version 0
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:429:455 [1] NCCL INFO Using network Socket
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:430:454 [2] NCCL INFO comm 0x5563a3113b20 rank 2 nranks 4 cudaDev 2 nvmlDev 2 busId 61000 commId 0x81e31ac233832a8e - Init START
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:429:455 [1] NCCL INFO comm 0x55eeb2ddb740 rank 1 nranks 4 cudaDev 1 nvmlDev 1 busId 41000 commId 0x81e31ac233832a8e - Init START
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:428:452 [0] NCCL INFO comm 0x55738de34760 rank 0 nranks 4 cudaDev 0 nvmlDev 0 busId 21000 commId 0x81e31ac233832a8e - Init START
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:431:453 [3] NCCL INFO comm 0x5592b925a920 rank 3 nranks 4 cudaDev 3 nvmlDev 3 busId 81000 commId 0x81e31ac233832a8e - Init START
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:429:455 [1] NCCL INFO Setting affinity for GPU 1 to ffffffff,00000000,ffffffff
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:429:455 [1] NCCL INFO NVLS multicast support is not available on dev 1
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:430:454 [2] NCCL INFO Setting affinity for GPU 2 to ffffffff,00000000,ffffffff
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:430:454 [2] NCCL INFO NVLS multicast support is not available on dev 2
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:428:452 [0] NCCL INFO Setting affinity for GPU 0 to ffffffff,00000000,ffffffff
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:428:452 [0] NCCL INFO NVLS multicast support is not available on dev 0
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:431:453 [3] NCCL INFO Setting affinity for GPU 3 to ffffffff,00000000,ffffffff,00000000
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:431:453 [3] NCCL INFO NVLS multicast support is not available on dev 3
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:430:454 [2] NCCL INFO comm 0x5563a3113b20 rank 2 nRanks 4 nNodes 1 localRanks 4 localRank 2 MNNVL 0
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:431:453 [3] NCCL INFO comm 0x5592b925a920 rank 3 nRanks 4 nNodes 1 localRanks 4 localRank 3 MNNVL 0
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:430:454 [2] NCCL INFO Trees [0] 3/-1/-1->2->1 [1] 3/-1/-1->2->1
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:429:455 [1] NCCL INFO comm 0x55eeb2ddb740 rank 1 nRanks 4 nNodes 1 localRanks 4 localRank 1 MNNVL 0
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:431:453 [3] NCCL INFO Trees [0] -1/-1/-1->3->2 [1] -1/-1/-1->3->2
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:430:454 [2] NCCL INFO P2P Chunksize set to 131072
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:431:453 [3] NCCL INFO P2P Chunksize set to 131072
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:429:455 [1] NCCL INFO Trees [0] 2/-1/-1->1->0 [1] 2/-1/-1->1->0
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:428:452 [0] NCCL INFO comm 0x55738de34760 rank 0 nRanks 4 nNodes 1 localRanks 4 localRank 0 MNNVL 0
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:429:455 [1] NCCL INFO P2P Chunksize set to 131072
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:428:452 [0] NCCL INFO Channel 00/02 : 0 1 2 3
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:428:452 [0] NCCL INFO Channel 01/02 : 0 1 2 3
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:428:452 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:428:452 [0] NCCL INFO P2P Chunksize set to 131072
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:429:455 [1] NCCL INFO Channel 00/0 : 1[1] -> 2[2] via P2P/CUMEM
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:428:452 [0] NCCL INFO Channel 00/0 : 0[0] -> 1[1] via P2P/CUMEM
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:430:454 [2] NCCL INFO Channel 00/0 : 2[2] -> 3[3] via P2P/CUMEM
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:429:455 [1] NCCL INFO Channel 01/0 : 1[1] -> 2[2] via P2P/CUMEM
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:428:452 [0] NCCL INFO Channel 01/0 : 0[0] -> 1[1] via P2P/CUMEM
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:431:453 [3] NCCL INFO Channel 00/0 : 3[3] -> 0[0] via P2P/CUMEM
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:430:454 [2] NCCL INFO Channel 01/0 : 2[2] -> 3[3] via P2P/CUMEM
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:431:453 [3] NCCL INFO Channel 01/0 : 3[3] -> 0[0] via P2P/CUMEM
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:428:452 [0] NCCL INFO Connected all rings
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:431:453 [3] NCCL INFO Connected all rings
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:431:453 [3] NCCL INFO Channel 00/0 : 3[3] -> 2[2] via P2P/CUMEM
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:429:455 [1] NCCL INFO Connected all rings
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:430:454 [2] NCCL INFO Connected all rings
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:431:453 [3] NCCL INFO Channel 01/0 : 3[3] -> 2[2] via P2P/CUMEM
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:429:455 [1] NCCL INFO Channel 00/0 : 1[1] -> 0[0] via P2P/CUMEM
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:430:454 [2] NCCL INFO Channel 00/0 : 2[2] -> 1[1] via P2P/CUMEM
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:429:455 [1] NCCL INFO Channel 01/0 : 1[1] -> 0[0] via P2P/CUMEM
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:430:454 [2] NCCL INFO Channel 01/0 : 2[2] -> 1[1] via P2P/CUMEM
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:428:452 [0] NCCL INFO Connected all trees
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:428:452 [0] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 512 | 512
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:428:452 [0] NCCL INFO 2 coll channels, 0 collnet channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:429:455 [1] NCCL INFO Connected all trees
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:429:455 [1] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 512 | 512
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:429:455 [1] NCCL INFO 2 coll channels, 0 collnet channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:431:453 [3] NCCL INFO Connected all trees
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:431:453 [3] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 512 | 512
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:431:453 [3] NCCL INFO 2 coll channels, 0 collnet channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:430:454 [2] NCCL INFO Connected all trees
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:430:454 [2] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 512 | 512
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:430:454 [2] NCCL INFO 2 coll channels, 0 collnet channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:430:454 [2] NCCL INFO comm 0x5563a3113b20 rank 2 nranks 4 cudaDev 2 nvmlDev 2 busId 61000 commId 0x81e31ac233832a8e - Init COMPLETE
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:428:452 [0] NCCL INFO comm 0x55738de34760 rank 0 nranks 4 cudaDev 0 nvmlDev 0 busId 21000 commId 0x81e31ac233832a8e - Init COMPLETE
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:429:455 [1] NCCL INFO comm 0x55eeb2ddb740 rank 1 nranks 4 cudaDev 1 nvmlDev 1 busId 41000 commId 0x81e31ac233832a8e - Init COMPLETE
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:431:453 [3] NCCL INFO comm 0x5592b925a920 rank 3 nranks 4 cudaDev 3 nvmlDev 3 busId 81000 commId 0x81e31ac233832a8e - Init COMPLETE
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:430:457 [2] NCCL INFO [Service thread] Connection closed by localRank 2
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:429:456 [1] NCCL INFO [Service thread] Connection closed by localRank 1
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:431:458 [3] NCCL INFO [Service thread] Connection closed by localRank 3
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:428:459 [0] NCCL INFO [Service thread] Connection closed by localRank 0
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:430:470 [0] NCCL INFO comm 0x5563a3113b20 rank 2 nranks 4 cudaDev 2 busId 61000 - Abort COMPLETE
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:429:468 [0] NCCL INFO comm 0x55eeb2ddb740 rank 1 nranks 4 cudaDev 1 busId 41000 - Abort COMPLETE
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:431:469 [0] NCCL INFO comm 0x5592b925a920 rank 3 nranks 4 cudaDev 3 busId 81000 - Abort COMPLETE
mixtral-8x7b-instruct-tgi-deploy-76b99c95f9-nb7tf:428:471 [0] NCCL INFO comm 0x55738de34760 rank 0 nranks 4 cudaDev 0 busId 21000 - Abort COMPLETE
from vllm.
youkaichao
commented on June 26, 2024
1
Your log file only shows 1 second. How long does it hang?
from vllm.
youkaichao
commented on June 26, 2024
1
I don't think this is vllm's problem. From vllm side, I can tell there are something wrong with the environment, so the distributed environment cannot be set up.
You can try to switch different factors, e.g. changing the gpu model you use, the gpu driver version you use, and try to use a physical machine rather than k8s container. I don't know anything about Openshift, so I cannot help here.
from vllm.
thobicex
commented on June 26, 2024
Did you went through any trail this year?
Yes / No
from vllm.
thobicex
commented on June 26, 2024
Okay, I'll look into it by next week.
This storm will get to an end once I get my work address fixed on my profile.
from vllm.
thobicex
commented on June 26, 2024
I will figure it out in two business working days, thank you for your advice.
I will let you know once I figure it out by Monday.
from vllm.
thobicex
commented on June 26, 2024
The issues is minority report, of which I tired to fill out the information needed.
Thank you all for your time, I'm going offline now. Till business working days. Have a great weekend everyone.
from vllm.
jayteaftw
commented on June 26, 2024
Your log file only shows 1 second. How long does it hang?
If you are talking about the vllm instance, it is still hanging (so about 34 minutes since creation)
The test.py went through
from vllm.
youkaichao
commented on June 26, 2024
this is the start line:
2024-06-09 00:57:24.508092 Call to <module> in /usr/local/lib/python3.10/dist-packages/vllm/worker/worker.py:1 from _call_with_frames_removed in <frozen importlib._bootstrap>:241
this is the end line:
2024-06-09 00:57:25.183071 Return from __enter__ in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:365 to synchronize in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:791
only one second elapsed.
if it is still hanging, it is hanging inside pytorch code.
from vllm.
jayteaftw
commented on June 26, 2024
this is the start line:
2024-06-09 00:57:24.508092 Call to <module> in /usr/local/lib/python3.10/dist-packages/vllm/worker/worker.py:1 from _call_with_frames_removed in <frozen importlib._bootstrap>:241this is the end line:
2024-06-09 00:57:25.183071 Return from __enter__ in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:365 to synchronize in /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:791only one second elapsed.
if it is still hanging, it is hanging inside pytorch code.
Okay, diagnosis wise what does that mean?
Also It actually has only been alive 10m. I think the pod restarts after it hangs for that much time
from vllm.
jayteaftw
commented on June 26, 2024
Is VLLM unable to run in a container with 4 l40s hook to it through the nvidia runtime?
from vllm.
Related Issues (20)
- [Performance]: gptq and awq quantization do not improve the performance HOT 4
- [Installation]: Compiling VLLM for cpu only. HOT 2
- [New Model]: mistralai/Codestral-22B-v0.1 HOT 7
- [Usage]: Streaming Response from vLLM 0.4.2 -> 0.4.3 HOT 2
- [Usage]: Function calling for mistral v0.3
- [Bug]: vLLM does not support virtual GPU
- [Bug]: Unexpected prompt token logprob behaviors of llama 2 when setting echo=True for openai-api server HOT 1
- [Bug]: Getting an empty string ('') for every call on fine-tuned Code-Llama-7b-hf model HOT 1
- [Bug]: non-deterministic Python gc order leads to flaky tests HOT 13
- [Usage]: Howto quiet the terminal 'Info' outputs in vllm HOT 1
- [Performance]: [Automatic Prefix Caching] When hitting the KV cached blocks, the first execute is slow, and then is fast. HOT 3
- [Speculative decoding]: `AttributeError: 'NoneType' object has no attribute 'numel'` when exceeding draft context length HOT 3
- [Bug]: Qwen2 MoE: AttributeError: 'MergedColumnParallelLinear' object has no attribute 'weight'. Did you mean: 'qweight'? HOT 3
- [Bug]: with `--enable-prefix-caching` , `/completions` crashes server with `echo=True` above certain prompt length HOT 2
- [RFC]: Refactor MoE
- [Bug]: TorchSDPAMetadata is out of date HOT 2
- [Bug]: Falcon fails if `trust_remote_code=True`
- [Bug]: load nvidia/Llama3-ChatQA-1.5-8B model 15 min HOT 4
- [Bug]: Docker image starts vllm.entrypoints.openai.api_server , Docker opens port 8000 but vllm isn't listening on 8000 HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from vllm.