[Bug] XL-recurrence with AlibiPositionalBias and mems not working correctly about x-transformers HOT 17 OPEN

@pfeatherstone that's from the author of alibi. of course they would say it is great

from x-transformers.

@pfeatherstone yea i could make it work

however, i think alibi is really bad and probably should be removed. i would never use it for any serious model

from x-transformers.

@pfeatherstone maybe i'll just make it all work as a personal challenge. i can also fix dynamic pos bias in the presence of memory as well

from x-transformers.

Basically i found rotary positional embedding to not length-extrapolate at all. Like, it's really bad. I'm doing some tests with XPOS and though I haven't got a complete model yet (still training), it looks a bit better. However I'm nervous about limiting the context length.

from x-transformers.

@pfeatherstone yea i could make it work

however, i think alibi is really bad and probably should be removed. i would never use it for any serious model

is this based on personal tests? According to the paper it's the best thing since sliced bread

from x-transformers.

Basically i found rotary positional embedding to not length-extrapolate at all. Like, it's really bad. I'm doing some tests with XPOS and though I haven't got a complete model yet (still training), it looks a bit better. However I'm nervous about limiting the context length.

that's well known, i think i even mention it in the readme. however, there's a lot of research going into fine tuning trained rotary models to longer context, so it is not a big deal

from x-transformers.

i wouldn't use xpos either.. it suffers from the same issues as alibi. i really should start removing features i no longer believe in

from x-transformers.

@pfeatherstone yea i could make it work
however, i think alibi is really bad and probably should be removed. i would never use it for any serious model

is this based on personal tests? According to the paper it's the best thing since sliced bread

which paper?

from x-transformers.

https://ofir.io/train_short_test_long.pdf, the one you reference in your readme. I have to admit I haven't read it in great detail but they suggest AliBI is great.

from x-transformers.

Basically I need a positional embedding that length-extrapolates well, works with memories, and flash attention. Do you have any suggestions?

from x-transformers.

Basically I need a positional embedding that length-extrapolates well, works with memories, and flash attention. Do you have any suggestions?

these days, i would stick with rotary, given the amount of research now going into it

curriculum learn to longer sequence lengths while tuning the rotary theta value (and whatever new tricks recent papers have discovered)

from x-transformers.

What do you mean by curriculum learn to longer sequence lengths? Sorry if my questions are dumb.

from x-transformers.

@pfeatherstone ah, curriculum learning is just a fancy way of saying making training increasingly harder over time, like how you design a curriculum for a student. so start with a small sequence length and slowly increase to your desired length

from x-transformers.