Sequence to Sequence? about rwkv-lm HOT 3 CLOSED

make a standard encoder-decoder architecture with RWKV

I think it's possible with this approach:

1. Run RWKV to encode context into hidden states, and store them.

2. Currently the RNN decoder will only use the hidden states of last token, but you can mix it with the encoder hidden states (and learn the layerwise channelwise mixing weights), then the decoder is using encoder output.

is there any report on in-context-learning/FSL capability of the latest trained model?

RWKV is good at zero-shot and you are welcome to test few-shot :) Let me know if you run into any issues.

from rwkv-lm.

Ahh, thank you for your reply. I don't understand completely when you say: mix it with the encoder hidden states (and learn the layerwise channelwise mixing weights) but will try looking at code first.

PS: Please excuse late reply, was out of loop from this for a while. Will try some few-shot with this!

from rwkv-lm.

Originally RWKV RNN is only using [hidden state after last token].

But you can concat that with [hidden state after encoder].

Then all tokens in decoder always have access to both [hidden state after encoder] & [hidden state after last token].

Then the model can learn how to use them.

Please join our Discord for more discussions with everyone :)

from rwkv-lm.