jduhamel / cdeid Goto Github PK

A framework for training de-identification models to automatically remove protected health information (PHI) from the free text.

cDeid is a customized de-identification method. The users can easily train their own de-identification Models on the data sets which are extracted from their own free text corpus. cDeid is based on 3 popular NLP toolkits: spaCy, Stanza and FLAIR.

This project is based on Python 3.7+. Please install it if you do not have. PyTorch is used by FLAIR and Stanza toolkits. It needs to be installed from here before you install this project.

pip install cdeid

We are using the pre-trained word2vec embeddings released from the CoNLL 2017 Shared Task. It is important to specify the customized PHI types in the corpus otherwise it will cause runtime error during training the models.

from cdeid.models.trainer import Trainer

phi_types = ['PHONE', 'PERSON', 'ADDRESS', 'IDN', 'DOB']
nlp = Trainer("C:/data", "C:/workspace", phi_types, "C:/wordvec/English/en.vectors.xz")
nlp.train()

from cdeid.deidentifier.phi_deid import PHIDeid

deider = PHIDeid("C:/workspace", "C:/output")
doc = deider("C:/raw/example.txt")
deider.output(doc)

python -m cdeid --command train --workspace C:/workspace --data_dir C:/data --phi_types PHONE PERSON ADDRESS IDN DOB --wordvec_file C:/wordvec/English/en.vectors.xz

python -m cdeid --command deid --workspace C:/workspace --deid_output_dir C:/output --deid_file C:/raw/example.txt

• 0.1.1
  • The first release
• 0.1.2
  • Modify Readme and Setup
• 0.1.3
  • Update model trainers

Leibo Liu - initial work - leiboliu

Apache License, Version 2.0