With the development of Medical Artificial Intelligence (AI) System, Natural Language Processing (NLP) has played an essential role to process medical texts and build intelligent machines. Named Entity Recognition (NER), one of the most basic NLP tasks, is primarily studied since it is the cornerstone of the following NLP downstream tasks, e.g., Relation Extraction. In this work, a character-level Bidirectional Long-short Term Memory (BiLSTM)-based models were introduced to tackle the challenge of medical texts. The input character embedding vectors were randomly initialized and then updated during training. The character-level BiLSTM extracted features from medical order-matter sequential data. The followed Conditional Random Field (CRF) predicted the final entity tag. Results have shown that the presented method took advantage of the recurrent architecture and achieved competitive performances for medical texts. Promising results paved the road towards building robust and powerful medical AI engines.

Task: Named Entity Recognition (NER) implemented using PyTorch

Background: Medical & Clinical Healthcare

Level: Character (and Word) Level

Data Annotation: BIOES tagging Scheme

Method:

1. CRF++

2. Character-level BiLSTM + CRF

3. Character-level BiLSTM + Word-level BiLSTM + CRF

4. Character-level BiLSTM + Word-level CNN + CRF

Results:

 Results of this work can be downloaded here.

Prerequisities:

 For Word-level models:

 The pre-trained word vectors can be downloaded here.

def load_word_vector(self):
    """
    Load word vectors
    """
    print("Start to load pre-trained word vectors!!")
    pre_trained = {}
    for i, line in enumerate(codecs.open(self.model_path + "word_vectors.vec", 'r', encoding='utf-8')):
        line = line.rstrip().split()
        if len(line) == self.word_dim + 1:
            pre_trained[line[0]] = np.array([float(x) for x in line[1:]]).astype(np.float32)
    return pre_trained

 For Character-level models:

 The Embeddings of characters are randomly initialized and updated by a PyTorch Function, i.e., (nn.Embedding).

self.char_embed = nn.Embedding(num_embeddings=vocab_size, embedding_dim=self.char_dim)

Number of entities: 34

train data: 6494   vocab size: 2258   unique tag: 74  

dev data: 865   vocab size: 2258   unique tag: 74  

data: number of sentences vocab: character vocabulary unique tag: number of (prefix + entities)

At the root of the project, you will see:

├── data
|  └── train # Training set 
|  └── val # Validation set 
|  └── test # Testing set 
├── models
|  └── data.pkl # Containing all the used data, e.g., look-up table
|  └── params.pkl # Saved PyTorch model
├── preprocess-data.py # Preprocess the original dataset
├── data_manager.py # Load the train/val/test data
├── model.py # BiLSTM-CRF with Attention Model
├── main.py # Main codes for the training and prediction
├── utils.py # Some functions for prediction stage and evaluation criteria
├── config.yml # Contain the hyper-parameters settings

    Character Input
          |                         
     Lookup Layer  <----------------|    Update Character Embedding
          |                         |
     Bi-LSTM Model  <---------------|        Extract Features
          |                         |     Back-propagation Errors
     Linear Layer  <----------------|   Update Trainable Parameters
          |                         |
       CRF Model  <-----------------|    
          |                         |
Output corresponding tags  ---> [NLL Loss] <---  Target tags

1. Currently only support CPU training

   GPU is much more slower than the CPU as a result of the viterbi decode's FOR LOOP.

2. Cannot recognize entities with fewer examples (< 500 samples)

Overall F1 score on 18 entities:

Separate F1 score on each entity:

The BiLSTM + CRF model has been deployed using Docker + Flask as a webapp.

The codes and demos were open-sourced in this repo.

1. Neural Architectures for Named Entity Recognition

2. Log-Linear Models, MEMMs, and CRFs

3. Named Entity Recognition with Bidirectional LSTM-CNNs

4. A Survey on Deep Learning for Named Entity Recognition

1. Lattice LSTM

2. 中文NER的正确打开方式: 词汇增强方法总结 (从Lattice LSTM到FLAT)

3. 工业界如何解决NER问题

MIT Licence