Bulk Prediction for multi input in C# about onnxruntime HOT 5 CLOSED

There are no special capabilities for bulk predictions. If you model is constructed in a way that it would accept the shapes in your example, then you feed that data. C# is just a thin layer on top of the native library.

There is no difference between python, C# in terms of the capabilities.

from onnxruntime.

The input/output dimensions of your model dictate what you can process.
This does not change with the API you choose to use, the model remains the same so is the implementation behind the API.

You also need to stop leaking memory in a loop. Please, read https://onnxruntime.ai/docs/tutorials/csharp/basic_csharp.html

from onnxruntime.

I guess I'm asking how to predict in batches, rather than one at a time in a loop. Thanks.

from onnxruntime.

I currently save the model

that has a last layer:
model4 = keras.models.Model(inputs=[spectra1, spectra2, ratioAProtS,ratioBProtS,isoS,hitA,hitB,sizeS], outputs=output)

as so:

import tf2onnx
import tensorflow as tf
import onnx
input_signature = ([tf.TensorSpec((1, (1001)), tf.float32) ,tf.TensorSpec((1, (1001)), tf.float32)
,tf.TensorSpec((1, (1001)), tf.float32) ,tf.TensorSpec((1),
tf.float32),tf.TensorSpec((1), tf.float32) ,
tf.TensorSpec((1), tf.float32) ,
tf.TensorSpec((1), tf.float32) ,
tf.TensorSpec((1), tf.float32) ,tf.TensorSpec((1), tf.float32) ])

onnx_model, _ = tf2onnx.convert.from_keras(model4 , input_signature, opset=18)
onnx.save(onnx_model, "../../Desktop/testModel.onnx")

how could I change it for multibatch? thanks

from onnxruntime.

Beautiful, @yuslepukhin . Thanks. I figured it out.

For future people, what yuslepukhin says is key. There's no difference.

You need a mix of these two threads:

#9867
onnx/onnx#2182

Just need to change to N, and then provide what you need.

Got it working in python, should be able to in C# now. Will report back if not, but otherwise closed.

from onnxruntime.