访问次数多了以后显存不释放 about chinese-llama-alpaca-2 HOT 6 CLOSED

api server的日志输出没有任何报错

from chinese-llama-alpaca-2.

经过一晚上又降回了17显存

from chinese-llama-alpaca-2.

大概每50-100次就会遇到下面这个报错，torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 3.51 GiB (GPU 0; 23.65 GiB total capacity; 17.10 GiB already allocated; 2.61 GiB free; 20.28 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALL
OC_CON

对于4090这样的显卡运行 Chinese-Alpaca-2 7B 这样的模型，显存占用是合理的吗？

from chinese-llama-alpaca-2.

模型本身权重就有14G左右，17G显存占用是合理的。运行中显存占用升高应该是每次推理缓存没有清理的原因，可以试试每次推理完将之前的记录等都清理了。脚本本身是指导性的作用，不一定那么完善，欢迎修改完善。

from chinese-llama-alpaca-2.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your consideration.

from chinese-llama-alpaca-2.

Closing the issue, since no updates observed. Feel free to re-open if you need any further assistance.

from chinese-llama-alpaca-2.