ONNX does not work for BMM function about opencv HOT 8 OPEN

@fengyuentau, that's definitely 3D issue with last dimension used as channels. The data layout should be fine.
Probably, we can make a fix in C++ version of setInput which changes Mat header in case of channels() != 1 to it's ND version.

from opencv.

I am speculating about internals of OpenCV now, I apologize if I misunderstand things, please correct me where my knowledge is incorrect.

As I understand it, 3d numpy arrays are translated into multichannel matrices in the python bindings, and changing that behavior might break code that depends on this behavior. However, this does not prevent the ONNX inference layer to convert all multichannel matrices (a.k.a. blobs) into single channel by adding an extra dimension. If any downstream code depends on the faulty conversion of numpy arrays into opencv multichannel matrices I would argue that they depend on unspecified behavior, hence it is OK to break those consumers (that is, if an ONNX API consumer depends on the behavior of translating a 3d array into a 2d matrix, they are ok to break, because relying on that behavior is nonsensical).

This is just for the 3d case, I understand that 0d/1d is trickier since matrices needs at least 2 dimensions?

Again, I am just speculating about the internals as a user of the opencv library, but this is how I perceive this issue and how would expect it to work.

Basically you are correct. As @dkurt 's comment, it is better to make a fix in setInput to convert Mat of channels!=1 to one of channels=1.

from opencv.

If you look closely to the error message, it says:

[ERROR:[email protected]] global net_impl.cpp:1168 getLayerShapesRecursively     input[0] = [ 3 5 ]
[ERROR:[email protected]] global net_impl.cpp:1168 getLayerShapesRecursively     input[1] = [ 3 5 ]

Both the two tensors loses the last dimension. It was due to a legacy problem where OpenCV Python interface converts a three dimensional array/tensor into a Mat with channel (channel is not part of Mat shape). See more detail #23242.

We should fix this problem in 5.x. cc @vpisarev

from opencv.

Yeah, this is probably just a side effect of #25763 . Can a fix be included in 4.11 as well? It would be sad to leave 4.x hanging in a broken state.

from opencv.

By the way, the issue referenced by you is a duplicate of #19091.
It seems that this was closed by this PR in 2021: #20462, so it is surprising to see these issues arise for 3D inputs in 2024 with latest OpenCV version.

from opencv.

Can a fix be included in 4.11 as well? It would be sad to leave 4.x hanging in a broken state.

It was legacy. I guess this cannot be fixed as some functions depand on this functionality which transforms a numpy array to a Mat object that is used in the underlying c++ code.

@dkurt @opencv-alalek Could you add comments and verify my point?

from opencv.

I am speculating about internals of OpenCV now, I apologize if I misunderstand things, please correct me where my knowledge is incorrect.

As I understand it, 3d numpy arrays are translated into multichannel matrices in the python bindings, and changing that behavior might break code that depends on this behavior. However, this does not prevent the ONNX inference layer to convert all multichannel matrices (a.k.a. blobs) into single channel by adding an extra dimension. If any downstream code depends on the faulty conversion of numpy arrays into opencv multichannel matrices I would argue that they depend on unspecified behavior, hence it is OK to break those consumers (that is, if an ONNX API consumer depends on the behavior of translating a 3d array into a 2d matrix, they are ok to break, because relying on that behavior is nonsensical).

This is just for the 3d case, I understand that 0d/1d is trickier since matrices needs at least 2 dimensions?

Again, I am just speculating about the internals as a user of the opencv library, but this is how I perceive this issue and how would expect it to work.

from opencv.

@dkurt Thank you so much for adding information!

Speaking of make a fix in C++ version of setInput, I would say it should be better to make a fix in the Python binding of setInput to convert a Mat from channels!=1 to channels=1, since we do not have such problems with c++ version and blobFromImage can handle any inputs in C++ already.

from opencv.