thunlp / plmpapers Goto Github PK

Thanks! This is a great resource. Could you add our recent MultiFiT (Eisenschlos et al., EMNLP 2019) that extends ULMFiT for multilingual applications to the diagram?

For relatively new conference papers like ICRL2020, are you sure about the quality? I think there are a lot of articles recently added. After all, the title is a must-read thesis. I hope to control it and only keep the essence.thanks

Hi,

We have released the code of TinyBERT here (https://github.com/huawei-noah/Pretrained-Language-Model/tree/master/TinyBERT). Could you please add it?

Thanks

Just a small typo. LXMBERT should be LXMERT in the diagram.

DistilBERT is proposed by Hugging Face and implemented in pytorch-transformers. I think this work is worth adding to the repo.

You might consider adding context2vec, which was one of the predecessors to ELMo and inspired much of the work on LM pre-training: https://www.aclweb.org/anthology/K16-1006.pdf

Hi,

Wonderful paper list! Could you please update the new link (https://arxiv.org/pdf/1909.10351v2.pdf) of TinyBERT?

Thanks

GPT到BERT中的BidirectionalLM是指双向LSTM吗？ 如果是的话，是不是应该 ELMo才是BidirectionalLM，然后指向BERT啊。

BERT和GPT都是Transformer，GPT指向BERT是Transformer。

Transformer Dissection: An Unified Understanding for Transformer’s Attention via the Lens of Kernel
https://arxiv.org/abs/1908.11775

Another perspective to understand the attention formulation of Transformer.

Hi,
I'm the first author of a paper mentioned in this list, "Thieves on Sesame Street! Model Extraction of BERT-based APIs". Firstly, thanks so much for the mention!

I feel the paper might be more appropriate under the Knowledge Distillation & Model Compression section or the Analysis section, since it's not really a new model.