Unsupported ONNX type 10 for FP16 about deepsparse HOT 5 CLOSED

This command for export transformers to onnx model
!optimum-cli export onnx --model /content/bert-base-indonesian-1.5G-sentiment-analysis-smsa bert-sentiment-onnx-fp16-opset/ --opset 13 --task text-classification --optimize 'O1' --device 'cuda' --fp16

from deepsparse.

HI @farizalmustaqim fp16 ONNX models aren't supported in DeepSparse or CPU runtimes generally, so please try your command with these edits

optimum-cli export onnx --model /content/bert-base-indonesian-1.5G-sentiment-analysis-smsa bert-sentiment-onnx-fp32-opset/ --opset 13 --task text-classification

from deepsparse.

Oh really, but I was able to do yolov8 inference using the onnx model with the f16 option to reduce the model size in the deepsparse pipeline. Is that not possible for NLP?

from deepsparse.

@farizalmustaqim That is interesting to hear, I guess it might be possible it would just run in a naive backend for sure. Even in the optimum codebase, they raise an exception if you try to export fp16 on a CPU device https://github.com/huggingface/optimum/blob/5017d06603488f396537e69ff77055907fae79d0/optimum/exporters/onnx/__main__.py#L295

from deepsparse.

Hi @farizalmustaqim
As some time has passed with no further updates, I am going to go ahead and close out this issue.
Please re-open if you want to continue the conversation.
Best, Jeannie / Neural Magic

from deepsparse.