Implementation of A Compare-Aggregate Model with Dynamic-Clip Attention for Answer Selection on TrecQA and WikiQA using Keras

• Python 2.x
• Theano 0.8
• Keras 1.0

• WikiQA: A Challenge Dataset for Open-Domain Question Answering
• TrecQA: Answer Selction Task
• GloVe: Global Vectors for Word Representation

First, you need to install dependencies, then clone this repo:

git clone https://github.com/wjbianjason/Dynamic-Clip-Attention

I have uploaded my prepocess result of WikiQA, you can repeat the procedure as follow:

Note: dowload "WikiQACorpus.zip" to the path "./data/raw_data/WikiQA/".

WikiQACorpus.zip download link: https://www.microsoft.com/en-us/download/details.aspx?id=52419

sh preprocess.sh wikiqa

Note: If you don't have svn command, you can copy the directory TrecQA_of_CIKM2016_Rao to our path "./data/raw_data/"

sh preprocess.sh trecqa

Because I have uploaded my preprocess data of WikiQA, if you want to cast a glance at our model, you can skip above operations.

usage: main.py [-h] [-t TASK] [-m MODEL] [-d HIDDEN_DIM] [-e EPOCH] [-l LR]
               [-k_q K_VALUE_QUES] [-k_a K_VALUE_ANS] [-b BATCH_SIZE]
               [-p PRE_TRAIN]

Basic approach: listwise

python main.py -t wikiqa -m listwise -d 300 -e 10 -l 0.001 -b 5

Note: k_max and k_threshold need basic approach trained model to init weights. So please running basic approach first.
Second approach: k_max

python main.py -t wikiqa -m k_max -d 300 -e 5 -l 0.001 -b 5 -k_q 5 -k_a 10 -p 5

Third approach: k_threshold

python main.py -t wikiqa -m k_threshold -d 300 -e 4 -l 0.001 -b 6 -k_q 0.08 -k_a 0.08 -p 1

Basic approach: listwise

python main.py -t trecqa -m listwise -d 300 -e 15 -l 0.001 -b 3 

Note: k_max and k_threshold need basic approach trained model to init weights. So please running basic approach first.
Second approach: k_max

python main.py -t trecqa -m k_max -d 300 -e 5 -l 0.001 -b 3 -k_q 3 -k_a 16 -p 5

Third approach: k_threshold
Note: -1 represents the reciprocal of answer length

python main.py -t trecqa -m k_threshold -d 300 -e 5 -l 0.001 -b 3 -k_q -1 -k_a 0.06

In all experiments, we selected training models that obtain the best MAP scores on the development set for testing.
You should be able to reproduce some scores close to the numbers in the experiment table of our paper.
If you want to reproduce the same score, you need to use the following command:

THEANO_FLAGS="dnn.conv.algo_bwd_filter=deterministic,dnn.conv.algo_bwd_data=deterministic" python

which makes the cuDNN's backward pass is deterministic. This is a reproduce problem for Theano, not our trick.

All Rights Reserved.