Finetune with transformer-xl pretrained models about transformer-xl HOT 5 OPEN

Hi!

Thank you so much for your time! I'm trying to learn more about transformers-xl. Could you please help answer this beginner question?

Is it possible to change the loss function and fine tune the transformer-xl model to do classification, similar to BERT? So for the transformer-xl model, there isn't one general pre-trained model like in BERT, but instead there are many pre-trained models based on different tasks?

Thank you so much for your time and God bless!

from transformer-xl.

Yes I believe it's a good direction. Transformer-XL is presumably good for document-level representations due to the ability of handling long context. On short text, Transformer-XL might also have an edge (see results on One Billion Word).

from transformer-xl.

@BoPengGit did you manage to create a transfoXL based classifier like Bert ?

from transformer-xl.

I don't remember, this was a long time ago.

from transformer-xl.

Is it possible to classify document of length 30k tokens/words, using transformer-XL?

from transformer-xl.