Failed to allocate A device memory about ml-suite HOT 9 CLOSED

from ml-suite.

I'm actually hitting this using the demo caffemodel as well, not just our custom model.

from ml-suite.

I did see the same error when I ran YOLO example. It happened when I ran YOLO twice and the second run ended with the error.
So, I stopped F1 instance and then stated F1. Then it runs.
There might be something missing to de-allocate/release resources or cleanup steps after run.

... ...
[XDNN] kernel configuration
[XDNN]   num cores       : 1
[XDNN]   dsp array width : 56
[XDNN]   img mem size    : 5 MB
[XDNN]   version         : 2.2
[XDNN]   8-bit mode      : 0
[XDNN]   Max Image W/H   : 1023
[XDNN]   Max Image Depth : 4095

Loading weights/bias/quant_params to FPGA...
ERROR: Failed to allocate A device memory

from ml-suite.

from ml-suite.

I was able to run with the demo yolo model after restarting my F1 instance as well.

from ml-suite.

Some more information. After restarting my instance I am able to run the demo with the default model successfully multiple times in succession. But if I try to use my custom yolo model, I get the error no matter what (after a restart and after successful default model run). Once I hit the error using my custom model, any subsequent runs (default or custom) will also hit the error.

My guess is that there is an issue with the custom model that is causing this failure which puts the system in a bad state which is resolved by a restart. So really there are two issues here. 1) What is wrong with our model that is breaking the demo. 2) Why is the initial failure causing all subsequent runs to fail.

from ml-suite.

from ml-suite.

We had an issue with the caffemodel datadir having an extra directory in the path when using our custom model. The result was that the weights weren't being found. That's why we would see the issue initially. Still not sure why the initial failure causes all subsequent runs to fail.

from ml-suite.

I recently hit this as well experimenting on the yolo demo. I'm not sure if my situation is the same as others but running fpga-clear-local-image -S 0 cleared the loaded AFI image from the FPGA and fixed it without needing to restart the whole instance. (Note: I only have one FPGA in my instance and -S 0 clears the FPGA at the first slot. If you have multiple FPGAs, you might need to adjust the slot number)

from ml-suite.