Suggestions to improve CM-MLPerf inference after SCC'23 about ck HOT 4 CLOSED

More feedback from SCC'23 teams about CM-MLPerf automation:

• Provide a tutorial how to debug CM scripts/workflows - maybe we can make a simple video tutorial?
• Provide a list of all main flags for CM script to run MLPerf inference benchmark
• Describe all compatible flags and variations for CM script to run MLPerf inference benchmark
• Provide a description of a used model (and/or a link to original paper) and how to optimize it (quantize/prune/etc)
• Improve logging in CM (particularly when CM is in interactive mode): #1017
• Visualize all dependencies in a CM-MLPerf script: #1018

from ck.

Another idea is to stop CM just before running the command and show how we prepared the command line and which cache entries are used (with inference sources, model, dataset, loadgen, etc) ... This is what --debug flag already does but we may mention it explicitly.

Also, we should simplify tutorial and only describe the CM command to run full MLPerf inference benchmark out-of-the-box with all the flags that can customize execution - we plan to prepare a higher-level CM-MLPerf launcher to do that.

from ck.

Yet another suggestion is to make it clear that CM is not a "reproducibility" tool but a workflow automation tool that attempts adapt benchmark/applications to continuously changing software and hardware. When combined with Docker, we can ensure that we can run a benchmark (even though it doesn't guarantee reproducibility of performance numbers since we may not have a control to pin threads to specific cores, control NUMA, set frequency of different cores, etc). But CM users want to run MLPerf with new software/hardware even if it fails, because we can then collaboratively improve CM scripts and make them more portable and reproducible as a community effort - we should highlight it in the tutorial!

from ck.

We added support for most features from here in CM v2+. I am closing this ticket and we will open a new one with a few pending features if needed.

from ck.