CPU instead of GPU about fauxpilot HOT 4 CLOSED

You won't be able to do that with this particular codebase (which is heavily dependent on FasterTransformer so that it can get decent latency), but this gist from @nforest shows how to use any model from Huggingface Transformers with Copilot, and Transformers supports doing CPU-only:

https://gist.github.com/nforest/d1432b917468f5ad24b83954c98e67b1

You should be able to pass in pretrained=Salesforce/codegen-16B-multi there and device='cpu' to run everything on the CPU instead.

I do warn you that it will be a lot slower!

from fauxpilot.

Thank you!

Edit: Running off CPU makes this more accessible by moving the barrier to entry from a $3000 rig to a $1000 rig.

from fauxpilot.

@TechnologyClassroom Did you tried it? How long did it take for a completion on what cpu? 😃

from fauxpilot.

@1muen I didn't try it yet. I commented asking about licensing and have not heard back. (That project really deserves to be a full repo too)

from fauxpilot.