Failed to run tensorflow mnist example about tony HOT 3 CLOSED

That is the VMEM used by your TF jobs, tf uses minimal vmem. If you use GPU for your training, you're guaranteed to use all a huge amount of vmem.

from tony.

In our deep learning cluster, we've disabled YARN's vmem check by setting

<property>
  <name>yarn.nodemanager.vmem-check-enabled</name>
  <value>false</value>
  <description>Whether virtual memory limits will be enforced for containers.</description>
</property>

Ref: https://stackoverflow.com/questions/14110428/am-container-is-running-beyond-virtual-memory-limits

from tony.

In our deep learning cluster, we've disabled YARN's vmem check by setting

<property>
  <name>yarn.nodemanager.vmem-check-enabled</name>
  <value>false</value>
  <description>Whether virtual memory limits will be enforced for containers.</description>
</property>

Ref: https://stackoverflow.com/questions/14110428/am-container-is-running-beyond-virtual-memory-limits

Thanks!It solves my problem.

from tony.