A list of pretrained Transformer models for the Russian language (including multilingual models).

Code for the model using and visualisation is from the following repos:

• https://github.com/huggingface/pytorch-transformers
• https://github.com/jessevig/bertviz

There are models form:

• DeepPavlov project
• Hugging Face repository
• Facebook research
• Facebook's fairseq
• Denis Antyukhov Google Colab code
• Russian RuBERTa

Downloading and converting the DeepPavlov model:

$ wget 'http://files.deeppavlov.ai/deeppavlov_data/bert/rubert_cased_L-12_H-768_A-12_v1.tar.gz'
$ tar -xzf rubert_cased_L-12_H-768_A-12_v1.tar.gz
$ python3 convert_tf_checkpoint_to_pytorch.py \
    --tf_checkpoint_path rubert_cased_L-12_H-768_A-12_v1/bert_model.ckpt \
    --bert_config_file rubert_cased_L-12_H-768_A-12_v1/bert_config.json \
    --pytorch_dump_path rubert_cased_L-12_H-768_A-12_v1/bert_model.bin

There are scripts to train and evaluate models on the Sber SQuAD dataset for the russian language [download dataset].

Comparision of BERT models trained on the Sber SQuAD dataset:

The attention-head view visualization from BertViz:

[Notebook]

The model view visualization from BertViz:

[Notebook]

The neuron view visualization from BertViz:

[Notebook]

Code: https://github.com/mgrankin/ru_transformers

Download models:

pip install awscli
aws s3 sync --no-sign-request s3://models.dobro.ai/gpt2/ru/unfreeze_all gpt2

• Code: https://github.com/vlarine/ruGPT2
• Model: gpt2_345m.tgz (4,2G)

There are some RNN models for russian language.

• ELMo on Russian Wikipedia: [config], [model]
• ELMo on Russian WMT News: [config], [model]
• ELMo on Russian Twitter: [config], [model]

• RNC and Wikipedia. December 2018 (tokens): [model]
• RNC and Wikipedia. December 2018 (lemmas): [model]
• Taiga 2048. December 2019 (lemmas): [model]

• Pavel Pleskov model: [model]
• mamamot model: [model]