This repository contains the code used to produce the results in the paper "Detecting intersectionality in NER models: A data-driven approach" by Lassen et al. (2023).

The project investigates the effect of intersectional biases in Danish language models (see list) used for Named Entity Recognition (NER). This is achieved by applying a data augmentation technique, namely augmenting all names in the DaNe testset on gender-divided name lists for both majority and minority names.

For instructions on how to reproduce the results, please refer to the Pipeline section.

The repository has the following directory structure:

The following models are evaluated:

• ScandiNER
• DaCy models
  • DaCY large (da_dacy_large_trf-0.1.0)
  • DaCy medium (da_dacy_medium_trf-0.1.0)
  • DaCY small (da_dacy_small_trf-0.1.0)
• DaNLP BERT
• Flair
• NERDA
• Spacy models
  • SpaCy large (da_core_news_lg-3.4.0)
  • SpaCy medium (da_core_news_md-3.4.0)
  • SpaCy small (da_core_news_sm-3.4.0)
• Polyglot

The pipeline has been built on Ubuntu (UCloud).

For all models except polyglot, first run the setup.sh

bash setup.sh

This will create a virtual environment (env) and install the necessary packages.

To then evaluate all models, run:

bash run-models.sh

To setup and evaluate polyglot, run:

sudo bash polyglot.sh

NB! Notice that it is necessary to run this code with sudo as the setup requires certain devtools that will not be installed otherwise. Run at own risk!

The polyglot.sh script will both install devtools, packages and run the evaluation of the model in a seperately created environment called polyenv.

The name augmentation was performed using augmenty and model evaluation was performed using the DaCy framework.