Add More Discussions on Multiprocessing about lecture-python-programming.myst HOT 4 CLOSED

My idea is that we could also restructure our current lecture on parallelization into two large sections -- multiprocessing and multithreading. Then, we can introduce them separately, compare the differences, and group these functions into corresponding categories like Numba and JAX, implicit multithreading in NumPy, distributed scheduler (multiprocessing/multithreading) in Dask... so that the conceptual differences between them will be clear to readers.

I think you're right. Let's go with this approach for now unless @mmcky objects.

Regarding the section on multiprocessing, my first instinct is that we should just stick to dask. Thoughts on that @mmcky ? Also, we should point out that multiprocessing can be used on top of multithreading --- for example, with a cluster of machines or GPUs.

from lecture-python-programming.myst.

I agree @HumphreyYang and @jstac I think dask is the current clear pick. There are some others like multiprocessing but they aren't as general or as simple as dask (except for really simple parallelisation).

from lecture-python-programming.myst.

Hi @HumphreyYang and @mmcky , thanks for the input. Let's focus on dask while at least mentioning the existence of other options.

Second, with apologies, I want to reverse myself and request a separate lecture on dask / multiprocessing. One reason is that I envisage adding applications to the numba lecture and I don't want it to get too long.

I suggest we title the two lectures

• Parallelization Part I: Multithreading
• Parallelization Part II: Multiprocessing

If you guys agree then @HumphreyYang and @mmcky , it would be great if you could take the lead together on Part II. In the meantime I'm going to write up the JAX lecture and add one application to the numba lecture.

Regards, John.

from lecture-python-programming.myst.

Close with #263

from lecture-python-programming.myst.