the onnx in optimum can not accelerate the speed of batch feature extraction. about optimum HOT 3 CLOSED

It's slower than the pytorch version when using batch decoding in optimum onnx. If I use origin onnx, It's faster than the pytorch version.I also find other user meet the same problem.
like:
#1337
#626
#1217
etc.
I think they are all the reason the optimum onnx can't support batch decoding.

from optimum.

I test use this script. https://github.com/amulil/vector_by_onnxmodel/blob/main/generate_optimum.py

It seems it's not a bug of optimum, It maybe the bug of sentence_transformer, I'm not sure.

Because I test transformers pipeline('feature_extraction', onnx=true), slow again. And I use the sentence_transformer pipeline SentenceEmbeddingPipeline, also slow again.

from optimum.

I found the reason of it. If you want to use the pipeline on batch decoding, you must use the arg batch_size. Therefore, It's not a bug of optimum.

from optimum.