Use Petals without sharing GPU about xmtf HOT 11 OPEN

Not sure about that, maybe one of @borzunov @justheuristic @mryab knows?

from xmtf.

Hi @raihan0824,

Your GPU is not shared when you use a Petals client to run inference or fine-tuning. The GPU is only shared when you run a Petals server.

from xmtf.

yes, but I want to run bloom in petals with my own GPU, not others. Is that possible?

from xmtf.

Hi, do you mean you want to use Petals with your GPU, but don't want to let the others use it? I think you can set up a private swarm using these instructions. If you run into any troubles, the tutorial has a link to the Discord server, where we (and other users of Petals) can help you with technical issues.

Please keep in mind that you'll need around 176GB of GPU memory just for 8-bit parameters though; if you only have a single GPU, your best bet is offloading or joining the public swarm.

from xmtf.

Well noted.

Is it possible to do prompt tuning with that private swarm? also what if I want to use the smaller bloom model such as bloomz-7b1-mt?

My goal is to do prompt tuning on bloomz-7b1-mt.

from xmtf.

Yes, it is possible — you just need to specify a different set of initial peers in DistributedBloomConfig when you're creating DistributedBloomForCausalLM from the tutorial. By default, the config (and thus the model) will connect to peers from the public swarm — you need to change these to the addresses of your peers in the private swarm.

However, I'd say that for bloomz-7b1, you might not even need Petals (depends on your GPU setup, obviously). A reasonably new GPU should be able to host the whole model, so you'll be able to run it just with standard Transformers/PEFT. Do you have any specific reasons why you want to use Petals for this task?

from xmtf.

The reason why I want to use petals is because it can be used to do prompt tuning, instead of fine-tuning. I can't find other sources that provides prompt tuning for BLOOM

from xmtf.

Have you checked out https://github.com/huggingface/peft#use-cases? I think PEFT even showcases bigscience/bloomz-7b1, and the model support matrix includes BLOOM for prompt tuning

from xmtf.

Thank you for the info! Will check it out.

So I want to confirm my initial question:
It's possible to use petals with my own GPU to do inference and prompt-tuning on bigscience/bloomz-7b1 model.
Is that correct?

from xmtf.

Yes, it is possible, but not necessary: with PEFT, you are likely to get the same result with fewer intermediate steps for setup.

from xmtf.

Thank you very much 🙏

from xmtf.