A set of tools and experimental scripts used to achieve multimodal learning with nonnegative matrix factorization (NMF).

This repository contains code to reproduce the experiments from the publications:

[Mangin2015]

O. Mangin, D. Filliat, L. ten Bosch, P.Y. Oudeyer, MCA-NMF: Multimodal concept acquisition with non-negative matrix factorization PLOS ONE (October 21, 2015) (bibtex)

[Mangin2013]

O. Mangin, P.Y. Oudeyer, Learning semantic components from sub symbolic multi modal perception Joint IEEE International Conference on Development and Learning and on Epigenetic Robotics (ICDL EpiRob), Osaka (Japan) (2013) (bibtex)

Please consider citing these papers when re-using the code in scientific publications.

This code is intended to work on its own. It however requires the dependencies mentioned below to be installed and available on the operating system. Please note the following elements.

• Locations: The file local.py is meant to hold local configuration of paths to data, features, etc. A template is provided in the repository. The paths defined in that file are only used when no path argument is provided to related functions (for examples the dataset loaders, or feature generators).
• Data: The experimental scripts require data (see below). This data must be accessible from the script and eventually pre-processed. For the motion data, pre-processing can be achieved through multimodal/db/scripts/build_choreo2_features. Alternatively, all features and metadata for the databases can be downloaded by running multimodal/db/scripts/download_dbs. This removes the need to pre-process the choreo2 dataset.
• Experiments: Just run the experimental script. Be sure to generate the required features. For the moment, the ACORNS Caregiver dataset is not available online, however feature files can be provided on request.

So for a quick hand on, just:

1. Install dependencies:

pip install numpy scipy matplotlib librosa

2. Download and install the sources:

git clone http://github.com/omangin/multimodal.git
cd multimodal
python setup.py develop --user

3. Download databases features and metadata (run from where you downloaded the repository):

./multimodal/db/scripts/download_dbs

4. Reproduce [Mangin2013]'s experiment:

./multimodal/experiments/icdl2013.py

• db: helpers to load and pre-process data
• features: helpers to build features
• lib: general purpose helpers
• experiments: experimental scripts
• test: a few unit tests

• experiment.py This file contains the main logic of the experimental setup. It defines classes to run multimodal experiments in a consistent and simplified manner. These include data preparation, training and evaluation.
• learner.py Contains a class that abstracts the process of learning from multiple modalities. Contains the learning mechanism based on the NMF algorithm (itself included in lib/nmf.py).
• evaluation.py Helpers to evaluate learning results.
• pairing.py Helpers to generate associations of samples from different modalities with same labels.
• plots.py Plot functions to generate specific figures.

• icdl2013.py Runs experiments and generate results corresponding to Mangin2013. Uses Acorns Caregiver database for speech sounds as well as the Choreo2 dataset.
• image_sound.py Runs experiments similar to Mangin2013 with speech sounds from the Acorns Caregiver database and images from the image database.
• launcher.py Script to run many experiments with variations of parameters. Requires expjobs and some time or a cluster.
• image_sound_eval_sliding_windows.py Script to run experiments with speech sound and images in which the learner is evaluated on its recognition sound from small time windows.
• plot_image_sound_eval_sliding_windows.py Interactive and static plots from the previous experiment.
• plot_info_matrix.py Script to generate plot of the mutual information between sample labels and internal coefficients. The script re-uses a trained dictionary obtained from launcher.py.
• two_modalities.py and three_modalities.py run an experiment from a configuration file. Mainly used by launcher.py.

• Acorns Caregiver is available online. Please refer to the article by Bergmann et al. for permission. However, feature and metadata files can be downloaded through the code (see the download_meta_and_features function in db/acorns.py or use multimodal/db/scripts/download_dbs). See also .
• Choreo1 can be found at flowers.inria.fr/choreo/doc.
• Choreo2 can be found at flowers.inria.fr/choreo2. See also for features.
• Object images (not publicly available yet). Pictures acquired by Natalia Lyubova and David Filliat as frames from interaction with an iCub robot, through an RGBD sensor (red, green, and blue camera coupled with a depth sensor). Feature and meta files can be downloaded (see the download_meta_and_features function in db/objects.py or use multimodal/db/scripts/download_dbs). See also .

This code is distributed under the new BSD license. Please see LICENSE for more details.

• Uses transformation.py from ROS tf package.
• Uses a few functions imported from scikit-learn. These are grouped in file sklearn_utils.py.
• Uses functions adapted from prettyplotlib.

• python >2.7 or >3
• numpy
• scipy
• librosa (for sound feature computation)