Large ops with multiprocess fail with internal error about mlx HOT 3 OPEN

Hmm this has nothing to do with distributed processing unfortunately. The following also fails with the same error

for i in {0..3}; do python <path to script> & done; wait

from mlx.

I guess since this is somewhat rare.. and there isn't much if anything we can do in MLX, I will close this for now :.

from mlx.

Some possible work-arounds:

The internal error signifies that the GPU kernel timed out. One way to fix that is to decrease the size of the operations:

For example if you are using LoRA, try

• decreasing the batch size
• decreasing the sequence length
• decrease the model size

Another possible fix is to decrease the number of operations per command buffer, this may work in some cases but may also slow things down. You can do that like so:

MLX_MAX_OPS_PER_BUFFER=1 <training command here>

from mlx.