The SIPU Lab / University of Eastern Finland

Institute for Infocomm Research / A*Star / Singapore

Author: Ivan Kukanov, Email, Homepage, GitHub

Supervisor: Ville Hautamäki, Email, Homepage

Collaborator: Sabato Marco Siniscalchi, Homepage

Collaborator: Kong Aik Lee, Homepage

This python framework represents the implementation of the maximal figure-of-merit (MFoM) approach for approximation of discrete performance measures: F1-score (micro/macro), EER, DCF (detection cost function).
In particular, it is applied for the speech articulatory attributes detection, such as manner and place of articulation: fricative, glide, nasal, stop, vowel, voiced, coronal, dental, glottal, high, labial, low, middle, palatal and velar. Those articulatory attributes can be extracted from a speech signal and considered as a universal phonetic speech features. The application of these features is diverse: foreign accent detection, automatic speech recognition, spoken language recognition, e.t.c.

Note this project is based on the stories subset of the OGI Multi-language Telephone Speech corpus. We are not allowed to publish it here, because of the licence reason. You may check another project with the MFoM framework on the open dataset: Multi-label MFoM framework for DCASE 2016: Task 4.

The proposed MFoM approaches are used in the series of works

• Maximal Figure-of-Merit Embedding for Multi-label Audio Classification
• Recurrent Neural Network and Maximal Figure of Merit for Acoustic Event Detection
• Deep learning with Maximal Figure-of-Merit Cost to Advance Multi-label Speech Attribute Detection
• Boosting Universal Speech Attributes Classification with Deep Neural Network for Foreign Accent Characterization
• Speech Attribute Detection Using Deep Learning

Another implementations

• Multi-label MFoM framework for DCASE 2016: Task 4
• MULAN-LRE project Used during the NIST LRE 2015 challenge, includes KALDI trained models.

back to the TOC

The system is developed for Python 2.7. Currently, the baseline systems are tested only with Linux operating systems.

You may install the python environment using Conda and the yml setting file:

$ conda env create -f envs/conda/ai.py2.yml

and activate the environment

$ source activate ai

Specifically the project is working with Keras 2.0.2, Tensorflow-GPU 1.4.1.

back to the TOC

The executable file of the project is: experiments/run_ogits.py The system has two pipeline operating modes: Development mode and Submission (or evaluation) mode (TBD). The usage parameters are shown by executing python run_ogits.py -h. The system parameters are defined in experiments/params/ogits.yaml.

The main code of the project is in the src folder, see the description of the packages.

In this mode the system is trained and tested with the development dataset. This is the default operating mode. In order to run the system in this mode:

python run_ogits.py -m dev -p params/ogits.yaml -a manner,

where the parameter -a can have the values manner, place or fusion.

back to the TOC

In this project we release bunch of MFoM approaches. These are MFoM-microF1, MFoM-macroF1, MFoM-EER, MFoM-Cprime, MFoM-embedding. These approaches allow to optimize the performance metrics directly versus indirect optimization approaches with MSE, cross-entropy, binary cross-entropy and other objective functions. The implementation of the MFoM objective functions and layers see in src/model/objectives.py and src/model/mfom.py. Also, you may want check a simple example with the MFoM in tests/model/test_mfom_2d.py

back to the TOC

The parameters of the system are in experiments/params/ogits.yaml. It contains the next blocks.

Controlling the system pipeline flow

The pipeline of the system can be controlled through the configuration file.

pipeline:
    init_dataset: true
    extract_features: true
    search_hyperparams: false
    train_system: true
    test_system: true

General parameters

Dataset and experiment general settings

experiment:
    name: ogits
    development_dataset: <path to the OGI-TS development dataset>
    submission_dataset: <path to the submission dataset>
    lists_dir: <path to the original meta data of the dataset>
    attribute_class: place # or manner, fusion

System paths

This section contains the storage paths of the trained systems. Note by default all the system files (features, trained models) are not overwritten in consecutive running, you need either manually delete files or apply parameter --overwrite.

 path:
    base: system/
    meta: meta_data/
    logs: logs/
    features: features/
    models: models/
    hyper_search: hyper_search/
    train_result: train_results/
    submission: submissions/

These parameters defines the folder-structure to store acoustic features, trained acoustic models and store results.

Feature extraction

This section contains the feature extraction related parameters.

features:
  type: fbank    
  fbank:
    bands: 96
    fmax: 4000 # sample rate 8000Hz / 2
    fmin: 0
    hop_length_seconds: 0.02
    htk: false
    include_delta: false
    include_acceleration: false
    mono: true
    n_fft: 512
    window: hamming_asymmetric
    win_length_seconds: 0.04
    delta:
      width: 9
    acceleration:
      width: 9

We can define several types of features and specify the particular features in the parameter type. Currently we use log Mel-filter banks (type: fbank).

Model settings

This is the model settings for pre-training and fine-tune.
Note after changing any parameters the previous model will not be deleted, but new path with new hash will be generated.

model:
    type: sed_ogits        
    sed_ogits:
        do_pretrain: true
        do_finetune: true
        pretrain_set:
          metrics: [class_wise_eer, pooled_eer, micro_f1]
          activation: elu
          batch: 32
          batch_type: sed_sequence  # or sed_random_crop, see src/data_loader/ogits.py
          context_wnd: 128          # frame context
          dropout: 0.1
          feature_maps: 96
          loss: binary_crossentropy # or mfom_eer_normalized, mfom_microf1, pooled_mfom_eer, mfom_cprime, see src/model/objectives.py
          learn_rate: 0.0001
          n_epoch: 400
          optimizer: adam
          out_score: sigmoid
    
        finetune_set:
          activation: elu
          metrics: [class_wise_eer, pooled_eer, micro_f1]
          batch: 32
          batch_type: sed_sequence  # or sed_random_crop
          context_wnd: 128
          dropout: 0.1
          freeze_wt: false
          feature_maps: 96
          loss: mfom_eer_normalized # see src/model/objectives.py
          learn_rate: 0.0001
          n_epoch: 400
          optimizer: adam # or adadelta
          out_score: tanh

We can define several types of models and specify the particular model in the parameter type. Currently we use sound event detection (SED) model (type: sed_ogits). We can specify either we do model pre-training and tuning or only pre-training (do_pretrain: true and do_finetune: false). We can choose several metrics to calculate and monitor during training (metrics: [class_wise_eer, pooled_eer, micro_f1]), see src/train/ogits.py.

Trainer and callbacks settings

Set up parameters for metric of performance for monitoring, learning rate schedule, early stopping, tensorboard settings.

callback:
  monitor: class_wise_eer # micro_f1
  mode: min # or max for micro_f1
  chpt_save_best_only: true
  chpt_save_weights_only: true
  lr_factor: 0.5
  lr_patience: 20
  lr_min: 0.000001
  estop_patience: 25
  tensorboard_write_graph: true

back to the TOC

• First public release

back to the TOC

If you use the code or materials of this project, please cite as

@ARTICLE{8952610,  
  author = {I. {Kukanov} and T. N. {Trong} and V. {Hautamäki} and 
            S. M. {Siniscalchi} and V. M. {Salerno} and K. A. {Lee}},
  journal = {IEEE/ACM Transactions on Audio, Speech, and Language Processing},
  title = {Maximal Figure-of-Merit Framework to Detect Multi-Label Phonetic Features 
           for Spoken Language Recognition},   
  year={2020},  
  volume={28},  
  number={},  
  pages={682-695}
}

This software is released under the terms of the MIT License.