add `stream` to pipeline parameters about transformers HOT 6 OPEN

Personally would love to have streaming support in pipelines - it’s the one missing feature. Currently, streaming is quite difficult to use, but this would make it so much easier.

from transformers.

Hi @not-lain, thanks for opening a feature request!

using tokenizer.apply_chat_template then other stuff then model.generate is pretty repetitive

Could you elaborate on this a bit e.g. with a code snippet? Is is the streaming feature when generating you wish to be able to use?

from transformers.

@amyeroberts
normally when someone wants to stream their output (example: https://huggingface.co/spaces/ysharma/Chat_with_Meta_llama3_8b) they need to apply all that code, and this has been quite a repetitive process for AI models, and I thought we can implement this within the transformers library.

from transformers.

I was thinking about integrating this with only text-generation models, but I think we can do that too with image-to-text models.

this is a good resource for that: https://huggingface.co/blog/idefics#getting-started-with-idefics

from transformers.

Thanks for sharing an example!

I'm not sure this is really something we want to add to the pipelines. Pipelines are intended to be simple objects which enable users to get predictions in one line, they're not intended to support all transformers' functionality. In this case, I think it makes sense to leave streaming outside as it enables the user to have full control of the threads and yielding logic.

cc @Rocketknight1 @gante for your thoughts

from transformers.

Yeah, I'm on @amyeroberts's side here - pipelines are (imo) a sort of high-level "on-ramp" API for transformers, which make it easy for users to quickly get outputs from common workflows. We definitely don't want to pack them full of features to handle every use-case - that's what the lower-level API is for! If we make pipelines very feature-heavy, then they become very big and confusing for new users, which defeats their purpose.

Once users are streaming output and working with threads/yielding/async/etc. they're probably advanced enough that they don't need the pipelines anyway.

from transformers.