Ddip in Fastai2 about ddip HOT 4 CLOSED

Hello @bowenroom,

I have released another library mpify, to support distributed training for fastai2 in Jupyter notebook.

Here is the link: https://github.com/philtrade/mpify/tree/v0.1.0

If you have any question, please raise issues in the above new repo. Thank you for your patience.

from ddip.

Have you ever tested on fastai2? look forward to it!

from ddip.

Hello @bowenroom, thank you for the inquiry.

The support for fastai2 is very much a work-in-progress, some pieces work already, but not all. I can't promise when it'll be ready, but it's on my radar 😅

from ddip.

Update: Checked in a prototype notebook for fastaiv2:v2_multiprocess_ddp_05_pet_breeds.ipynb that allows distributed-data-parallel in Jupyter notebook.

It is an entirely different design from the current Ddip. It does not use the %load_ext Ddip mechanism, nor does it depends on ipyparallel.

It only needs a module called multiprocess (install by pip install multiprocess), which handles object and function serialization more properly than the default python multiprocessing, and torch.multiprocessing, specifically in the Jupyter notebook environment. See the excellent explanation here.

This on-demand distributed training within the interactive notebook environment may be more suitable for v2 because of many hidden race conditions outside of the learner.fit() loop, that the old design (the big hammer of a persistent ipyparallel cluster) has to wrestle with. E.g. in learning rate finder, or anything that may involve file system read/write.

The middle of the notebook shows the raw steps: a basic "create a group of new distributed learners on the fly using explicitly dataloader creation functions etc.., and train for N epochs", all in one cell. To Do: allow restoration from checkpoints weights/learner states, then train, and refactor these into easier to use api, possibly a context manager.

from ddip.