Memory allocation best practice about tch-rs HOT 4 CLOSED

Glad that you were able to track the issue down.
There are two ways to use no_grad with the rust bindings, either using no_grad with a closure:

    tch::no_grad(|| {
        // gradients are not tracked here
    });

Or using a no_grad_guard, in this cases gradient are not tracked until the guard reaches the end of its scope and is dropped.

{
    let guard = tch::no_grad_guard();
    // gradients are not tracked here
}

from tch-rs.

Would you have a repro for this ?
I just pushed a small example to try to replicate your issue. When running on gpu mode, my gpu memory stays constant a bit below 500MB. This is large but I think caused by pytorch internal caching, anyway I don't see the memory increasing.

As per your other questions:

• The way to feed datasets depend a lot on how your dataset are structured. Reading files and then using of_slice should be reasonable. There are some helper functions in the vision module to help with image datasets.
• Deallocation should be automatic when reaching the scope end (which is one of rust big advantage).
• The memory consumption of forward_t depends a lot of what the model is doing. It can certainly double memory consumption and even more.

from tch-rs.

It turns out the overly large pre-fetch buffer exhausted the main memory. It's not tch's fault.
Besides, I find Torch forum suggests no_grad() to disable gradient engine. It helps reducing memory footprint.

from tch-rs.

Thanks for your reply. The thread can be closed.

from tch-rs.