NPT-14389: `.neptune` folder is not cleaned up if multiple PyTorch Dataloaders are used about neptune-client HOT 6 CLOSED

Hey @NiklasKappel 👋

We just released neptune v1.10.0, and from my tests, the issue seems to have been fixed.

Could you please upgrade to the latest version of Neptune and give it a shot? 🙏

from neptune-client.

Hey, I just tried the latest version and the issue seems to be fixed for me as well. .neptune/async/ is now empty after a training run. :)

from neptune-client.

Hey @NiklasKappel 👋

We have started looking into this, and initial investigations suggest that Lightning is doing some kind of “instant kill” signals/overriding the signals in general to all the dataloader workers, preventing us from calling the post-execution cleaning logic.

While we see if this can be addressed from our end, can you try using persistent_workers=True in your DataLoaders? This will reduce the number of exec-* folders to ~num_workers.

from neptune-client.

Thanks, I can confirm that persistent_workers=True limits the number of exec-* folders to ~num_workers, which is already satisfactory as a workaround. :)

from neptune-client.

@NiklasKappel - great! Thanks for confirming!

Meanwhile we will continue investigating if this can be fixed from our end, or if there needs to be an issue created in Lightning's repo

from neptune-client.

Perfect 🎉

Thanks for confirming!

from neptune-client.