aicoe / benchmarks-tekton Goto Github PK

In tf_cnn/task/benchmark.yaml:
resources:
limits:
memory: "12Gi"
nvidia.com/gpu: 4
requests:
memory: "10Gi"
nvidia.com/gpu: 4

This is used even during CPU-only runs

When doing multiple runs, I noticed that as long as the ImageStream has a tag present, it will run the tests regardless of whether or not the new image build has completed. That's not a problem per se, but thought it was odd behavior

When using CPU-only benchmarking, it sets num_gpus, however setting it this way causes an error. In addition, setting it to 0 causes a different error due to:
https://github.com/tensorflow/benchmarks/blob/master/scripts/tf_cnn_benchmarks/benchmark_cnn.py#L750,L752

It uses that for the number of CPUs to run

recent change to support gpus has broken the master branch.
There is no easy fix other than reverting the commit to 485736f

• Remove buildconfig tasks

• Refactor cpu and gpu benchmark tasks

When attempting to run the benchmarks, one issue I found was that the ImageStream tag and the tag referenced in the BuildConfig do not match:

ImageStream Tag: tfc-benchmark
BuildConfig Uses: tf-cnn-benchmark:latest

Addressed by my PR, but issue here for tracking

Use the models in the following repo https://github.com/IntelAI/models/tree/master/benchmarks
to create Tekton Pipelines which use TensorFlow models into tekton pipelines.

When running the dockerfile build, it fails on the line:

RUN source scl_source enable devtoolset-7 rh-python36 && \
    pip install --upgrade pip setuptools wheel && \

Saying devtoolset-7 is invalid, I removed it and it appears to work

When attempting the benchmark, I was hitting an almost immediate OOM error. This is despite nodes having more than enough free memory to satisfy the 12Gb limit

Almost immediately after this, it fails with either return code 137 or 255, the pod listing OOMKilled as the reason

According to the README.md at https://github.com/tensorflow/benchmarks/blob/master/scripts/tf_cnn_benchmarks/README.md it seems the repo is no longer maintained, and directs users over to https://github.com/tensorflow/models/tree/master/official

Should I look at refactoring the tests to use maintained models?

Use the following repo https://github.com/mlperf/training_results_v0.6/tree/master/NVIDIA/benchmarks
to create a tekton pipeline for GNMT model.
The pipeline should use 1-4 GPUs.