Memory error on tensorflow directml about tensorflow-directml HOT 3 CLOSED

This is something we're working on. We didn't work on optimizing the memory allocator yet, so large batch sizes or big images that work on CUDA devices may not be supported on DML devices yet, depending on the device. This is also due to the fact that we don't have complete operator coverage yet, so copies between CPU/GPU happen more often than they should, therefore utilizing a lot of memory.

In your case, reducing the batch size when working with bigger images will probably let you run the model without errors.

from tensorflow-directml.

It seems to me that the Conv2D layer has some sort of memory allocation problem, I ran a Dense model with more than 100 million parameters without a problem, but even a 50 thousand params Conv2D model has memory issues, so my conclusion is that maybe some memory leak is happening.

from tensorflow-directml.

We just released tensorflow-directml 1.15.3.dev200911 with many improvements to the memory allocator. You can try it out and tell us how it goes!

Also, since we have now open-sourced our fork, new tensorflow-directml issues should be opened over here.

from tensorflow-directml.