chatGLM3-6b-128k的模型使用lora微调后，进行推理的时候卡住怎么回事？ about langchain-chatchat HOT 4 CLOSED

To address the issue of your model inference getting stuck, especially with longer prompts, consider implementing the following optimizations and strategies:

1. PagedAttention: This method reduces memory waste by dividing the KV cache into blocks for efficient retrieval during attention computation. It's particularly effective in managing the large memory footprint associated with KV caches.

2. Efficient Memory Sharing with PagedAttention: By sharing blocks among sequences through a block table, this approach can significantly reduce memory overhead, leading to improved throughput.

3. Optimize CUDA Kernels and Implement Continuous Batching: Focus on CUDA kernel optimizations and use continuous batching to improve GPU utilization by dynamically inserting new sequences into the batch based on resource availability.

4. FlashAttention for Attention Mechanism Optimization: FlashAttention optimizes the attention mechanism by introducing tiling and recomputation strategies, reducing the need for large intermediate attention matrices and speeding up the computation.

5. Dynamic and Inflight Batching for Scheduling Optimization: Use dynamic batching and inflight batching to refine scheduling and improve GPU resource utilization by addressing "bubbles" during inference.

6. Memory Management in Conversation History: Ensure the conversation history buffer does not exceed a predefined maximum token limit to prevent memory overflow issues.

7. GPU Memory Usage Management: Manage GPU memory usage through parameters such as gpu_memory_utilization, swap_space, max_num_batched_tokens, and max_num_seqs to control memory allocation during inference.

These strategies are derived from practices in managing large model inference processes and should be adjusted based on your specific setup and requirements. If the problem persists, profiling the memory usage during inference might help identify any bottlenecks or areas of excessive memory use.

To continue the conversation, mention @dosu.

from langchain-chatchat.

我在微调b6的时候也遇到了这个问题，是模型出现了复读现象，可以使用stream_chat看下是不是在正常输出。GPU占用率没有掉下去就没有卡死，只是复读造成生成的文本过长，在等待生成完成

from langchain-chatchat.

估计显卡没有用到，你检查显卡推理吧，另外这个模型应该有问题，不建议调这个

from langchain-chatchat.

估计显卡没有用到，你检查显卡推理吧，另外这个模型应该有问题，不建议调这个

显卡用到了，显存占用着呢，这个模型有什么问题啊，可以说说吗

from langchain-chatchat.