• A CNTK(Microsoft deep learning toolkit) implementation of S-NET: FROM ANSR EXTRACTION TO ANSWER GENERATION FOR MACHINE READING COMPREHENSION extraction part with some modifications.
• This project is designed for the MSMARCO dataset
• Code structure is based on CNTK BIDAF Example
• Support MSMARCO V1 and V2!

Here are some required libraries for training and evaluations.

• python3.6
• cuda-9.0 (CNTK required)
• openmpi-1.10 (CNTK required)
• gcc >= 6 (CNTK required)

• Please refer requirements.txt

This repo provides pretrained model and pre-processed validation dataset for testing the performance

Please download pretrained model and pre-processed data and put them on the MSMARCO/data and MSMARCO root directory respectively, then decompress them at the right places.

The code structure should be like

MSMARCO
├── data
│   ├── elmo_embedding.bin
│   ├── test.tsv
│   ├── vocabs.pkl
│   ├── data.tar.gz
│   └── ... others
├── model
│   ├── pm.model
│   ├── pm.model.ckp
│   └── pm.model_out.json
└── ... others

After decompressing,

cd Evaluation
sh eval.sh

then you should get the generated answer and rough-l score.

Download MSMARCO v1 dataset, GloVe embedding.

cd data
python3.6 download.py v1

Convert raw data to tsv format

python3.6 convert_msmarco.py v1 --threads=`nproc` 

Convert tsv format to ctf(CNTK input) format and build vocabs dictionary

python3.6 tsv2ctf.py

Generate elmo embedding

sh elmo.sh

Download MSMARCO v2 dataset, GloVe embedding.

cd data
python3.6 download.py v2

Convert raw data to tsv format

python3.6 convert_msmarco.py v2 --threads=`nproc`

Convert tsv format to ctf(CNTK input) format and build vocabs dictionary

python3.6 tsv2ctf.py

Generate elmo embedding

sh elmo.sh

cd ../script
mkdir log
sh run.sh

cd Evaluation
sh eval.sh v1

cd Evaluation
sh eval.sh v2

• Multi-threads preprocessing
• Elmo-Embedding
• Evaluation script
• MSMARCO v2 support
• Reasonable metrics