Document which architectures have been modified and are incompatible with the original architecture about neosr HOT 4 CLOSED

Hi, good suggestion. I wrote here when they are not compatible. As for ATD, I also added the empty tensor, so spandrel could detect it.

what your take is on neosr models being incompatible with original arch code?

If the changes contributes that much, yes, I would accept it. Minor changes should be an option though, as we have been doing with rgb normalization. For example, in some networks I added flash attention, because it improves ~40% training speeds. However, I made it a bool param, so anybody can enable/disable it.

from neosr.

Just one thing: you forgot SPAN :)

Added it

Maybe it's detectable?

Not sure, but probably yes. I put 2 models in this zip, one with flash_attn: true and other without it, if you want to take a look at some point: flash_attn.zip

from neosr.

I put 2 models in this zip

Thanks! That'll help when I get to it.

Since all architectures modifications seems to have been documented, I consider this issue closed now. We have to investigate those modifications for spandrel, but that isn't an issue with neosr.

Thanks musl!

from neosr.

I wrote here when they are not compatible.

Lots of valuable information. Thanks! Just one thing: you forgot SPAN :)

I'll have some testing to do to see how compatible everything is with spandrel.

As for ATD, I also added the empty tensor, so spandrel could detect it.

Yup, I saw. I'll add support for this soon.

I added flash attention

I wonder how compatible that is. You don't seem too optimistic in the docs. Maybe it's detectable? Then it won't be an issue. Well, I guess I'll just have to intestate and see.

from neosr.