invalid converting fp32 to fp16 about onnxruntime HOT 4 CLOSED

A possible cause: microsoft/onnxconverter-common#271

If you find two Cast nodes linked like:

A walkaround is to remove extra Cast nodes like

import onnx
from onnxruntime.transformers.onnx_model import OnnxModel
onnx_model=OnnxModel(onnx.load("path/to/model_fp16.onnx"))
onnx_model.remove_cascaded_cast_nodes()
onnx_model.save_model_to_file("path/to/model_fp16_v2.onnx", use_external_data_format=False, all_tensors_to_one_file=True)

from onnxruntime.

@tianleiwu Thank you that comments me
I checked the cast nodes through netron app

This is "model_fp16.onnx"

and then I just executed what you said
However, It doesn't works for me sadly
this is "model_fp16_v2.onnx"

For reference, I think that onnxruntime or onnxconverter_common are not invalid.
Because other models are works well. except for the probelm models (the problem model is bigger than other models)
Do you have any hint can I do by anychance?

from onnxruntime.

Not every model can run in fp16. Sometime, some nodes need to upcast to fp32 to avoid overflow.

You can add --cmake_extra_defines onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS=1 in build command line to build a package from source code. Set environment variable ORT_DEBUG_NODE_IO_DUMP_OUTPUT_DATA to be 1 before running your application. In this way, you can see of output of each node. Compare the stdout of two runs (fp32 model vs fp16 model) can find out the root cause. See https://onnxruntime.ai/docs/build/eps.html#cuda for more information.

from onnxruntime.

okay Thank you reply @tianleiwu
I'll keep in track this problem and I will share with you if I found the reason why the model had didn't work

from onnxruntime.