Giter Club home page Giter Club logo

adaptive-decision-boundary's Introduction

Deep Open Intent Classification with Adaptive Decision Boundary

A method to automatically learn the adaptive decision boundary (ADB) for open world classification.

The proposed method together with baselines are also integrated into the open intent detection module in our another scalable framework TEXTOIR, enjoy it!

Introduction

This repository provides the official PyTorch implementation of the research paper Deep Open Intent Classification with Adaptive Decision Boundary (Accepted by AAAI2021).

Related works can refer to the reading list.

Dependencies

We use anaconda to create python environment:

conda create --name python=3.6

Install all required libraries:

pip install -r requirements.txt

Model Preparation

Download the pre-trained bert model (bert-base-uncased) from the following link:
Baidu Cloud Drive with code: v8tk

Set the path of the uncased-bert model (parameter "bert_model" in init_parameter.py).

Usage

Run the experiments by:

sh scripts/run.sh

You can change the parameters in the script. The selected parameters are as follows:

dataset: clinc | banking | oos (default)
known_class_ratio: 0.25 | 0.5 | 0.75 (default)
labeled_ratio: 0.2 | 0.4 | 0.6 | 0.8 | 1.0 (default) 

Model

The model architecture of ADB: Model

Results

The detailed results can be seen in results.md.

Overall Performance

BANKING OOS StackOverflow
KIR* Methods Accuracy F1-score Accuracy F1-score Accuracy F1-score
25% MSP 43.67 50.09 47.02 47.62 28.67 37.85
DOC 56.99 58.03 74.97 66.37 42.74 47.73
OpenMax 49.94 54.14 68.50 61.99 40.28 45.98
DeepUnk 64.21 61.36 81.43 71.16 47.84 52.05
ADB 78.85 71.62 87.59 77.19 86.72 80.83
50% MSP 59.73 71.18 62.96 70.41 52.42 63.01
DOC 64.81 73.12 77.16 78.26 52.53 62.84
OpenMax 65.31 74.24 80.11 80.56 60.35 68.18
DeepUnk 72.73 77.53 83.35 82.16 58.98 68.01
ADB 78.86 80.90 86.54 85.05 86.40 85.83
75% MSP 75.89 83.60 74.07 82.38 72.17 77.95
DOC 76.77 83.34 78.73 83.59 68.91 75.06
OpenMax 77.45 84.07 76.80 73.16 74.42 79.78
DeepUnk 78.52 84.31 83.71 86.23 72.33 78.28
ADB 81.08 85.96 86.32 88.53 82.78 85.99

*KIR means "Known Intent Ratio".

Fine-grained Performance

BANKING OOS StackOverflow
KIR Methods Open Known Open Known Open Known
25% MSP 41.43 50.55 50.88 47.53 13.03 42.82
DOC 61.42 57.85 81.98 65.96 41.25 49.02
OpenMax 51.32 54.28 75.76 61.62 36.41 47.89
DeepUnk 70.44 60.88 87.33 70.73 49.29 52.60
ADB 84.56 70.94 91.84 76.80 90.88 78.82
50% MSP 41.19 71.97 57.62 70.58 23.99 66.91
DOC 55.14 73.59 79.00 78.25 25.44 66.58
OpenMax 54.33 74.76 81.89 80.54 45.00 70.49
DeepUnk 69.53 77.74 85.85 82.11 43.01 70.51
ADB 78.44 80.96 88.65 85.00 87.34 85.68
75% MSP 39.23 84.36 59.08 82.59 33.96 80.88
DOC 50.60 83.91 72.87 83.69 16.76 78.95
OpenMax 50.85 84.64 76.35 73.13 44.87 82.11
DeepUnk 58.54 84.75 81.15 86.27 37.59 81.00
ADB 66.47 86.29 83.92 88.58 73.86 86.80

“Open” and “Known” denote the macro f1-score over open class and known classes respectively.

If you are insterested in this work, and want to use the codes or results in this repository, please star this repository and cite by:

@article{Zhang_Xu_Lin_2021, 
      title={Deep Open Intent Classification with Adaptive Decision Boundary}, 
      volume={35}, 
      number={16}, 
      journal={Proceedings of the AAAI Conference on Artificial Intelligence}, 
      author={Zhang, Hanlei and Xu, Hua and Lin, Ting-En}, 
      year={2021}, 
      month={May}, 
      pages={14374-14382} 
}

Acknowledgments

This paper is founded by seed fund of Tsinghua University (Department of Computer Science and Technology)- Siemens Ltd., China Joint Research Center for Industrial Intelligence and Internet of Things.

adaptive-decision-boundary's People

Contributors

hanleizhang avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.