• Improve the method for distilling a dataset (scripts/distillation.py).
• Push the datasets trained in this way on the hugginface hub in order to load them for future use (NAR models benefit greatly from sequence-level knowledge distillation).

Change the codebase format in order to make it more readable and closer to the PEP8 standard.

Train a new sentencepiece tokenizer with the following requirements:

• Use SentencepieceBPETokenizer from huggingface.
• Train on the cc100 dataset possibly.
• All the languages must have the same number of sentences during training.
• It should have a <length> token as cls (for the CMLM model).
• It should have tokens mainly from english, french, german and spanish (romanian and chinese can also be considered).
• The special lang tokens should only be linked to the previously listed languages (this may be optional if the tokenizer doesn't allow it).
• Write a method to automatize the previous points.

The training batches can be either heterogeneous (filled with sentence pairs from different translations directions) or homogeneous (one translations direction per batch). There seems to be no clear winner in literature, so both should be tested.

• Implement the continual learning strategy of replay, each experience is made of two language directions ($en \Leftrightarrow x$), where $x \in \set{de, fr, es}$.
• Let the user decide the buffer size and the experience order.

• The trainer should take care of the train and validation steps which are currently located inside train.py.
• The logging on the tensorboard should be also managed by the trainer (if the user requests it).
• The train.yaml file should be used instead of passing arguments by terminal and it must have a simple structure.

The models can be trained on only one translation direction (e.g.: en->es), since the aim is to have multilingual models it should be possibile to allow more directions during the training phase.

• The training is performed by using the script train.py right now and the user must manually change the model, dataset collators and callbacks. A trainer class should be built in order to have a more general and automatized approach.
• There is still an issue during the validation loss logging (but this does not impact the model's performance) while using gradient accumulation, this should be solved.
• The progress bar used during training should show the actual number of stepping batches instead of the estimated ones, this implies that a custom progress bar must be built.
• It must be possible to not only train on a single direction (e.g.: en->es) but also on both ways given a pair of languages (e.g.: en<->es).
• If more than two languages are passed, then the user should have the choice on the translation directions.

It should be possible to compute the BLEU score on the validation sets instead of the usal loss.

Implement the GLAT model, the architecture is the same as the CMLM.

Improve train.py and let the user define all the arguments inside a .yaml file.

As a last step, it should be possibile to use the models through a gradio app.

Implement the mask-predict decoding algorithm for the CMLM model.

Upload the results obtained by the models in three different training settings:

• Multilingual.
• Incremental, to show the effects of catastrophic forgetting.
• Contintual, buffer size set to 5% of the entire training set.

Beam search should be implemented in order to obtain better results for the auto-regressive models.