Hi @whk6688 can you please share the source from which this command is taken from? It is missing --execution-mode=valid flag which is needed to do a valid submission run (default is test run). Also, --target_qps option should be avoided or the actual target_qps must be input during a valid run. (When we do a test run, target_qps should be automatically determined by the workflow and later used in the valid run).

cmr "run mlperf inference generate-run-cmds _submission" --submitter="MLCommons" --hw_name=default --model=bert-99 --implementation=reference --backend=pytorch --device=cpu --scenario=Offline --adr.compiler.tags=gcc --category=edge --division=open --execution-mode=valid

from ck.

i found the command in mlcommons project. i will send to you if i find again. besides, i saw the performance report:

i run the script:
cm run script --tags=generate-run-cmds,inference,_find-performance,_all-scenarios --model=bert-99 --implementation=reference --device=cpu --backend=onnxruntime --category=edge --division=open --quiet --rerun

================================================
onnx 模式下 bert 的性能

SUT name : PySUT
Scenario : Offline
Mode : PerformanceOnly
Samples per second: 0.670674
Result is : VALID
Min duration satisfied : Yes
Min queries satisfied : Yes
Early stopping satisfied: Yes

================================================
Additional Stats

Min latency (ns) : 10300619946
Max latency (ns) : 14910371096
Mean latency (ns) : 11759452531
50.00 percentile latency (ns) : 10690117315
90.00 percentile latency (ns) : 14910371096
95.00 percentile latency (ns) : 14910371096
97.00 percentile latency (ns) : 14910371096
99.00 percentile latency (ns) : 14910371096
99.90 percentile latency (ns) : 14910371096

Samples per second: 0.670674 ?????
Is there some thing wrong?
thanks!

from ck.

With --device=cpu option the run happens on the CPU. Please follow this link to run MLPerf on Nvidia Jetson Orin
https://github.com/mlcommons/ck/blob/master/docs/mlperf/setup/setup-nvidia-jetson-orin.md

from ck.