[BUG] 开启vllm加速后，客户端多并发调用服务时，出现Connection broken: InvalidChunkLength报错 about langchain-chatchat HOT 3 CLOSED

To address the issues you're experiencing with Langchain-ChatGLM, particularly after enabling VLLM acceleration, consider the following steps:

1. Model Loading: Ensure the model is fully loaded before it starts serving requests. Implement a health check that confirms the model's readiness. This can help avoid initial call failures due to the model not being loaded.

2. Concurrency Settings: The InvalidChunkLength errors suggest a need to adjust how concurrent requests are handled. Experiment with the limit_worker_concurrency parameter to find an optimal setting for your workload.

3. VLLM Configuration: Since enabling VLLM acceleration introduces the issue, closely review and adjust VLLM-related configurations such as tensor_parallel_size, block_size, swap_space, and gpu_memory_utilization.

4. System Resources: Monitor and ensure your system resources (CPU, GPU, memory) are not overwhelmed by the load, as this can lead to errors.

5. Error Handling and Retries: Implement error handling and retry mechanisms for intermittent InvalidChunkLength errors, which can improve application stability.

6. Software Updates: Consider updating to the latest version of Langchain-ChatGLM if available, as it may contain fixes and improvements for your issues.

These steps are derived from analyzing the provided configuration and the issues you're facing. Adjusting these areas should help mitigate the problems with initial call failures and intermittent InvalidChunkLength errors.

To continue the conversation, mention @dosu.

from langchain-chatchat.

问题1同。只有第一次调用会有这个问题。

from langchain-chatchat.

这个项目没有做并发。相关问题需要自己兼容

from langchain-chatchat.