Issue when running transformers from GPU / CPU about optimum HOT 3 CLOSED

Thanks for reporting this bug @axsaucedo ! We have a fix in #162 that should solve the issue :)

Regarding your question:

For completeness I am getting the notice The model 'ORTModelForCausalLM' is not supported for text-generation. - is that expected or does that suggest my transformers/hugging-face library is also not compatible with the latest live branch of optimum?

This is indeed expected because we're wrapping the pipeline() function from transformers which is only aware of the model classes in that library. We will have a look at providing a better warning for users, so thanks for flagging this!

We'll also look at adding support to set the target device in the transformers.pipeline() function by adding a ORTModel.to() method :)

from optimum.

Thank you for the fast turnaround @lewtun - I just tested it and works perfectly. Also thank you for the clarification on the warning, makes sense - I have added #163 for context.

I still cannot define device=X in the pipeline parametres, however I can confirm that it runs automatically in GPU, so it's not necessary. Would it still be required to ensure that the ORT models support the to(...) as well as other params like config? It's not currently a blocker from our side but just wanted to highlight.

from optimum.

I still cannot define device=X in the pipeline parametres, however I can confirm that it runs automatically in GPU, so it's not necessary. Would it still be required to ensure that the ORT models support the to(...) as well as other params like config? It's not currently a blocker from our side but just wanted to highlight.

Yes, currently the device arg can't be set in the pipeline() function for ORTModelForXxx classes - it's on our TODO list and will hopefully be added soon :)

And thanks for opening a new issue about the warning - that really helps for tracking!

from optimum.