koanatakiyo / ner Goto Github PK

This software implements several experiments by using the following toolkits

• Stanford Linear Classifier
• Stanford NER
• SVMLight, tree-kernels
• Label propagation algorithm JUNTO

In addition it access external ressources, namely a database with the wikipedia pages until Feb 2014

We also implemented a weakly supervised algorithm that is first initialized with the weights given by the stanford linear classifier trained on little data.

This software is distributed under the CeCILL-C license.

etc/ner.properties , this file has the variables necesary to configure the different classifiers

• src/linearclassifier AnalyzeClassifier.java :

  • interfaces with Stanford Linear Classifier

  • implements a weakly supervised algorithm based on risk minimisation. It can use the closed form for the risk estimation or a numerical approximation to the risk.

  • configuration: you can give to the classifier the name of classifier, for the moment the following types are supported

    • "pers": binary classifier for detecting whether or not one word is a person
    • "org": binary classifier for detecting whether or not one word is a organization
    • "prod": binary classifier for detecting whether or not one word is a product
    • "loc": binary classifier for detecting whether or not one word is a localization
      • "pn": if there is one general classifier that detects a proper name
    • "all": multiclass classifier detecting the categories: person,organization,produt and localization. All these constants are setted in the class tools.CNConstans.
    • input: set the static variables LISTTRAINFILES and LISTTESTFILES, which are files containing the list of files to process, see as examples esterTrain.xmll and esterTest. YOU MUST SET THE LIST OF FILES TO TRAIN AND TEST IN THE PROPERTIES FILE: ner.properties You can set a flag for using wikipedia as an extrafeature. If entity found in wikipedia as person, place,organization or product. (it can take up to 2h) Margin.java, store stanford linear classifier weights, features, and instances.

    NumericalIntegration.java, implements the numerical approximation to the risk

• src/gmm : All classes for the gmm-training

• src/svm AnalyzeSVMClassifier.java, interfaces to SVMLight, prepares the input and evaluates the output. There are several input files, it generates dependency trees for using tree kernel, polynomial kerner or linear kernel, it can even use the same features as the Stanford linear classifier. YOU MUST SET THE LIST OF FILES TO TRAIN AND TEST IN THE PROPERTIES FILE: ner.properties

• src/lex , necessary classes for storing utterances, words, lexical unix, postags and dependency trees

• src/resources WikipediaAPI.java, access to wikipedia pages in French all stored in a mysql database, up to feb 2104 For the database configuration, in a mysql database, create an user "contonmina/contnomina" in localhost, Create the wikipedia database by executing the script in wikipedia/db/dbWikibackupMar32014.sql (11G)
  YOU MUST SET THE DATABASE SETTING IN THE PROPERTIES FILE:ner.properties and in the hibernate configuration file: src/hibernate.cfg.xml

• src/labelpropagation LabelPropagation.java, prepares the input for the JUNTO label propagation toolit and evaluates its output file

• src/reco ASROut.java Alignment of the ASR output, calls the Linera Classifier and CRF Capitalization.java CRF for automic capitalizing the output of the ASR YOU MUST SET THE DATABASE SETTING IN THE PROPERTIES FILE:ner.properties and in the hibernate configuration file: src/hibernate.cfg.xml