Memory requirements about koboldai-client HOT 5 CLOSED

Running the models in half precision should work and make neo 2.7B be able to run in 8GB with barely any impact on accuracy. My transformers fork should already be casting the model to half precision automatically as well, unless it's getting cast back to fp32 after loading somehow.

The vanilla model's weights are fp32 though, so while loading it will use the full amount of system memory. There's a pre-halved version torrent floating around somewhere.

from koboldai-client.

Check your installed version of PyTorch. Models in KoboldAI are initialized through transformers pipelines; while the GPT-2 variants will load and run without PyTorch installed at all, GPT-Neo requires torch even through transformers. As for total memory usage, I'd recommend installing finetune's transformers branch, it'll reduce VRAM usage for Neo-2.7B from 12GB down to about 6.5-7.

pip install https://github.com/finetuneanon/transformers/archive/refs/heads/gpt-neo-dungeon-localattention1.zip

from koboldai-client.

pip install https://github.com/finetuneanon/transformers/archive/refs/heads/gpt-neo-dungeon-localattention1.zip

Interesting. I ran that, and then tried again. This time, with GPT-Neo 2.7b, it immediately started loading it into GPU memory, and in the end successfully loaded on my system without causing it to hang, almost identical to CloverEdition loading GPT-Neo 2.7b. Awesome!
Just one issue, though: Where CloverEdition's system memory usage dropped back down to 1.2 GB once it finished loading the AI model, KoboldAI's system memory usage is staying up at nearly 11 GB still allocated to it. Between it and Windows itself, that's not leaving much available for other stuff.

EDIT:
OK, I just noticed that memory usage did finally drop down to less than 1 GB system memory used by KoboldAI, I guess it just took a few minutes? Not sure why it took so long, but I guess it's not a problem since it did eventually free that huge chunk of system memory up.

from koboldai-client.

I definitely think that should be added to the project readme, it makes the larger models playable on a much wider range of hardware with the greatly reduced memory requirements.

from koboldai-client.

0.16 has been released with breakmodel allowing you to further manage memory, the readme has also been overhauled so people are pointed towards a version most suitable for them which is Finetuneanon's (newer) version for GPU users.

from koboldai-client.