Improve the method for distilling a dataset (scripts/distillation.py).
Push the datasets trained in this way on the hugginface hub in order to load them for future use (NAR models benefit greatly from sequence-level knowledge distillation).
The training batches can be either heterogeneous (filled with sentence pairs from different translations directions) or homogeneous (one translations direction per batch). There seems to be no clear winner in literature, so both should be tested.
Implement the continual learning strategy of replay, each experience is made of two language directions ($en \Leftrightarrow x$), where $x \in \set{de, fr, es}$.
Let the user decide the buffer size and the experience order.
The models can be trained on only one translation direction (e.g.: en->es), since the aim is to have multilingual models it should be possibile to allow more directions during the training phase.
The training is performed by using the script train.py right now and the user must manually change the model, dataset collators and callbacks. A trainer class should be built in order to have a more general and automatized approach.
There is still an issue during the validation loss logging (but this does not impact the model's performance) while using gradient accumulation, this should be solved.
The progress bar used during training should show the actual number of stepping batches instead of the estimated ones, this implies that a custom progress bar must be built.
It must be possible to not only train on a single direction (e.g.: en->es) but also on both ways given a pair of languages (e.g.: en<->es).
If more than two languages are passed, then the user should have the choice on the translation directions.