This is the official code implementation for the LEPISZCZE benchmark experiments. "This is the way: designing and compiling LEPISZCZE, a comprehensive NLP benchmark for Polish" (NeurIPS 2022) (Łukasz Augustyniak, Kamil Tagowski, Albert Sawczyn, Denis Janiak, Roman Bartusiak, Adrian Szymczak, Marcin Wątroba, Arkadiusz Janz, Piotr Szymański, Mikołaj Morzy, Tomasz Kajdanowicz, Maciej Piasecki).

LEPISZCZE benchmark resources

@inproceedings{augustyniak2022lepiszcze,
 author = {Augustyniak, Lukasz and Tagowski, Kamil and Sawczyn, Albert and Janiak, Denis and Bartusiak, Roman and Szymczak, Adrian and Janz, Arkadiusz and Szyma\'{n}ski, Piotr and W\k{a}troba, Marcin and Morzy, Miko\l aj and Kajdanowicz, Tomasz and Piasecki, Maciej},
 booktitle = {Advances in Neural Information Processing Systems},
 editor = {S. Koyejo and S. Mohamed and A. Agarwal and D. Belgrave and K. Cho and A. Oh},
 pages = {21805--21818},
 publisher = {Curran Associates, Inc.},
 title = {This is the way: designing and compiling LEPISZCZE, a comprehensive NLP benchmark for Polish},
 url = {https://proceedings.neurips.cc/paper_files/paper/2022/file/890b206ebb79e550f3988cb8db936f42-Paper-Datasets_and_Benchmarks.pdf},
 volume = {35},
 year = {2022}
}

In case of any question or concerns about LEPISZCZE benchmark feel free to contact us:

• Łukasz [email protected]
• Kamil [email protected]
• Albert [email protected]
• Denis [email protected]

DVC Repository Access Due to the size of pipeline outputs data, we do not provide public access to our DVC Remote Repository. However, if you are interested in any kinds of data artifacts, don't hesitate to get in touch with us.

Repository can be setup via poetry or via docker.

Prerequisites:

• Python: 3.9+
• Poetry [LINK].
• CUDA 11.3+ for GPU support (Recommended)

Installation

poetry install

For GPU support

poetry run poe force-torch-cuda

Building image

docker build . -f docker/Dockerfile -t LEPISZCZE

After the container setup use conda env LEPISZCZE

conda activate LEPISZCZE

Our experiments can be easily reproduced with DVC repro & W&B logging. Using dvc repro command and with W&B token setup.

DISCLAIMER Reproduction of full pipeline could take above 2000 hours to compelete on a single GPU device. We advise to execute stages in parallel on mutiple GPU computing devices.

Experiments configs can be found under configs

DISCLAIMER For some of the dataset we had to limit manually maximum sequence length to 512 for Hyper Parameter Search.

Models hyperparameters configuations can be accessed via W&B dashboard. Example: [LINK]