Benchmark fails almost immediately on training start with OOM error about benchmarks-tekton HOT 7 CLOSED

As a note, the pod is only consuming ~3-4Gb of RAM at the time it exits as well

from benchmarks-tekton.

    - restartCount: 0
      started: false
      ready: false
      name: step-run
      state:
        terminated:
          exitCode: 255
          reason: OOMKilled
          message: >-
            [{"key":"StartedAt","value":"2020-06-02T19:58:04Z","resourceRef":{}}]
          startedAt: '2020-06-02T19:57:56Z'
          finishedAt: '2020-06-02T20:01:02Z'
          containerID: >-
            cri-o://6f06a17d4da67d79f47359bece32163709441544350486a1ea3eb9bca04ac8c8
      imageID: >-
        image-registry.openshift-image-registry.svc:5000/mlperf/tfc-benchmark@sha256:7a78f8322d30a34f1a4da53833728a91de887c1ccc21f0cea77c0457974063ca
      image: >-
        image-registry.openshift-image-registry.svc:5000/mlperf/tfc-benchmark:cpu
      lastState: {}
      containerID: 'cri-o://6f06a17d4da67d79f47359bece32163709441544350486a1ea3eb9bca04ac8c8'

from benchmarks-tekton.

This run took up a bit more RAM, but still below 12Gb

from benchmarks-tekton.

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale

from benchmarks-tekton.

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

/lifecycle rotten

from benchmarks-tekton.

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

from benchmarks-tekton.

@sesheta: Closing this issue.

from benchmarks-tekton.