Automation and reproducibility for MLPerf Inference v3.1 about ck HOT 6 CLOSED

We got a feedback from submitters to create a GUI that can generate CM commands. I opened a ticket: #1070

from ck.

[20240130] We had lots of great feedback and improved both generic CM automation recipes and CM workflows for MLPerf inference:

• #1078
• #1079
  We are preparing CM v1.6.1 for the release ...

from ck.

We started discussing a proposal for MLPerf reproducibility badges similar to ACM/IEEE/NeurIPS conferences: #1080 - feedback is welcome!

from ck.

Following the feedback from the MLPerf submitters, we have developed a prototype of a GUI to generate a command line to run MLPerf inference benchmarks for all main implementations (reference, Intel, Nvidia, Qualcomm, MIL and DeepSparse) and automate submissions. You can check it here. The long-term goal is to aggregate and encode all MLPerf submission rules and notes for all models, categories and divisions in this GUI.

We have also developed a prototype of a reproducibility infrastructure to keep track of successful MLPerf inference benchmark configurations across different MLPerf versions, hardware, implementations, models and backends based on the ACM/IEEE/cTuning reproducibility methodology and badging. You can see the last results here - we will continue adding more tests based on your suggestions including GPT-J, LLAMA2 and Stable Diffusion.

Our goal is to test as many v4.0 submissions as possible and add them to the above GUI to make it easier for the community to rerun experiments after the publication date. If some configurations are not working, we plan to help submitters fix issues.

from ck.

We improved CM automation for Intel, Nvidia and Qualcomm and added to GUI: https://access.cknowledge.org/playground/?action=howtorun&bench_uid=39877bb63fb54725 . We can re-run most of them now.

from ck.

We now have relatively stable common CM interface to rerun above submissions and reproduce key results. I close this ticket - we will open a similar ticket for inference v4.0 after publication. Huge thanks to colleagues from Intel, Qualcomm and Nvidia for their help and suggestions!

from ck.