onnx / backend-scoreboard Goto Github PK

I see there have been new tests added for ops under the ml namespace, see this PR onnx/onnx#4881.

The current scoreboard setup will automatically pick up all tests under the backend/test/case/node directory. Since most backend providers don't support the ml ops, should such tests be excluded from the scoreboard? eg. by specifying -k "not test_ai_onnx_ml_".

There are no ORT nightly build packages any more (microsoft/onnxruntime#11791)

If we want that for our analysis we would have to build it by hand.

Hi,
using the following combination of dependencies

RUN pip3 install
onnx==1.12.0
onnxruntime==1.12.0
protobuf==3.20.1
numpy==1.22.4

results in a lot of successful tests...

= 102 failed, 979 passed, 1081 deselected, 1474 warnings in 154.80s (0:02:34) ==

I wonder what the further reasonable proceeding would be?