27182812 / qanet_paddle Goto Github PK

English | 简体中文

• QANET_paddle
  • 1、Introduction
  • 2、Accuracy+Align
    • step1: Model Structure Align
    • step2: Dataloader Align
    • step3: Metric Align
    • step4: Loss Function Align
    • step5: Backword Align
  • 3、Datasets
  • 4、Environment
  • 5、Quick start
    • step1: Clone
    • step2: Data Prep
    • step3: Train
  • 6、Code Structure
    • 6.1 Structure
    • 6.2 Parameter Description
    • 6.3 Training Process
  • 7、Model Information

This project is based on the paddlepaddle_v2.2.0rc0 framework to reproduce QANET(ICLR 2018)链接 。

The current end-to-end machine reading and question answering models are mainly based on recurrent neural networks that include attention. Apart from the advantages, the main disadvantage of these models is that they are less efficient in training and inference. Therefore, the author proposed a question and answer architecture called QANet. This network does not need to use a recursive network. Its encoder is completely composed of convolution and self-attention. The convolutional network processes local information, and self-attention processes the global scope. information.

paper:

• [1] Yu, A. W. , D Dohan, Luong, M. T. , Zhao, R. , Chen, K. , & Norouzi, M. , et al. (2018). Qanet: combining local convolution with global self-attention for reading comprehension.

Reference project：

• BangLiu/QANet-PyTorch: Re-implement "QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension" (github.com)

Project AI Studio address：

• notebook：https://aistudio.baidu.com/aistudio/projectdetail/2558567

The model is trained + verified on the SQuAD1.1 dataset.

! ! ! Indicators and Results

Please refer to the specific steps of align: https://github.com/PaddlePaddle/models/blob/develop/docs/ThesisReproduction_CV.md

When aligning the model structure, there are generally 3 main steps:

• Network structure code conversion
• Weight conversion
• Model network correctness verification

Align result is as follows：

Align result is as follows：

Align result is as follows：

Original repo operation results（Tried several times, and the original author of the same configuration and parameters, can not run his effect）：

Running results of this code：

weight file: https://aistudio.baidu.com/aistudio/datasetdetail/114931

SQuAD1.1

• dataset size：
  • train：87.5K
  • dev：10.1K
• data format：TEXT，JSON

• Hardware：CPU、GPU（16G and above）
• Framework：
  • PaddlePaddle >= 2.2.0rc0
• package：
  • spacy
  • ujson

# clone this repo
git clone [email protected]:27182812/QANet_paddle.git
cd QANet_paddle

Installation dependency

pip install -r requestments.txt

1. Create the "datasets/original" folder in the top-level directory, put glove.840B.300d.txt into the "Glove" folder under the secondary file, and place the SQuAD dataset into the "SQuAD" folder. As shown below:

   

   

2. First run, add --processed_data to preprocess the data, and put the processed data under the "datasets/original/processed" folder. It can be loaded directly in the next run.

1. Put pre-training weights in the top-level directory（I Initialized pytorch weight and converted to pdparams format)，here：https://aistudio.baidu.com/aistudio/datasetdetail/114636 。
2. run QANet_main.py

python QANet_main.py --batch_size 32 --epochs 60 --with_cuda --use_ema

├─data_loader                     # data load
├─datasets                        # dataset
├─imgs                            # illustrative picture
├─log_reprod                      # align files
├─model                           # model
├─trainer                         # train
├─util                            # utility function
│  README.md                      # English readme
│  README_CN.md                   # Chinese readme
│  requirements.txt               # rely
│  QANet_main.py                  # run file

Parameters related to training and evaluation can be set in QANet_main.py, as follows:

See 5、Quick Start

For other information about the model, please refer to the following table: