Omnistaging (still) breaks LSTM Model about dm-haiku HOT 4 CLOSED

I suspect this is an XLA compile time bug. Since omnistaging stages out bigger programs to XLA, it has a tendency to hit more compile time issues.

I think the best next step is to open a bug with the XLA team so that they can investigate. In some cases they can fix the issue causing the long compile time, but if not then we might need to brainstorm a workaround of some kind (no idea what that would look like until we understand the issue though!).

I'll reopen google/jax#4448, which may have been closed prematurely :P

Thanks for bringing this to our attention @n2cholas (and for pinging me @petebu)!

from dm-haiku.

Hi @n2cholas, I also get an OOM when running with OmniStaging on the public GPU colab kernel (12GB RAM). But by reducing the model size (e.g. vocab_size = 10, seq_length = 8) or using a bigger GPU (32GB RAM) I can compile it ok, so I don't think this is a bug in Haiku.

I'm going to close this for now but please reopen if you'd like to follow up.

from dm-haiku.

Thanks for the response @petebu. In my view, this is still a regression bug, since this model worked fine before the new 0.2.0 JAX release (which automatically enabled omnistaging). Also, the model I have in the gist is pretty small for a language model (vocab of only 1000 and sequence length of 80), so it should work no problem. Before the 0.2.0 release, a similar but larger model with a vocab size of 10,000 and sequence length of 256 worked fine, so this smaller one shouldn't OOM.

I would love to help fix this issue, but I'm not familiar enough with the internals. If you could provide some suggestions on what to look into, I would be happy to help.

from dm-haiku.

I think the issue here is with core Jax (rather than Haiku) since OmniStaging can change compilation. @mattjj Any ideas?

from dm-haiku.