We could get away with rounding float results fairly heavily (say to 3dp == 1000th pixel) to keep cached experiments small. This is surprisingly tricky:
http://stackoverflow.com/questions/1447287/format-floats-with-standard-json-module

menpobench currently goes as far as building and testing methods, gets the fitting results, and then...does nothing with them.

We need to persist the results from a given method. I think a .pkl of all the fitting results should be stored in the menpobench cache, that way at a later date we can recompute different graphs (i.e. handle different normalization formats).

The current design in mind is as follows:

For a given unique combination of: (training_dataset, testing_dataset, method)
or, in the case of untrainable_methods: (testing_dataset, untrainable_method)

if all components are pre-defined, cache the resulting fitting results to the cache/results/hash.pkl. On future invocations, check this cache, and return if it's available. We will also add a predefined cache that ships with menpobench - this will be updated from the continuous integration server, so for most common tests menpobench can complete immediately.

If any component is user defined, we don't use the cache at all. There is a reasonable expectation that these non-predefined files may be modified by the user, and we don't want to accidentally serve a cached result. We could definitely make this more advanced in the future, but for now, KISS.

We then write a generic output generation function that takes a set of fitting results and draws up the curves in various forms (e.g. different normalisation schemes). The user could have to provide an output dir which we populate with:

1. The pkl of the experiment (regardless of whether the test was predefined or not)
2. Graphs for commonly used normalisation
3. CSVs for the above graphs.

A flag could be added to return a mat file if the user wishes to easily import the fitting result data into matlab. With this, we would be heading towards something like:

menpobench ./mytest.yaml -o ./output --mat

With all this the API would look like:

./output
  ./methods
    ./aam
      ./fitting_results.mat
      ./fitting_results.pkl
      ./ced.pdf
      ./ced.csv
    ./sdm
      ...

The code to generate bounding box initialisations should probably live inside menpobench. The two initialisations that interest us from the start are:

1. Initialise from object detection (to start with from dlib)
2. Initialise with noise, using the approach of menpofit (post refactor)

For the object detection, we will want to:

1. Detect bounding boxes using the object detector
2. Define an measure of 'acceptable performance', based on overlap of gt/detected bounding boxes
3. If the detection failed, or is below threshold for acceptable performance, a new bounding box will be synthesised
4. Synthesised bounding boxes are placed by using a learned multivariate gaussian on the acceptable bounding boxes.

@patricksnape @nontas @jalabort Do you agree with this approach? Some help with the following questions would be great:

1. What do we feel is a good overlap amount? I was thinking that 80% of ground truth is covered by detected?
2. If there are multiple detections, we just keep one right?
3. Sorry @jalabort, can you point me once again to the new style noise initialisations in menpofit/refactor?

If we get this sorted I can add lfpw, afw, ibug, and helen datasets to menpobench with detections today. Any other datasets we want for testing?

At the moment, we plan on natively supporting Matlab in a thorough manner. Therefore, we need to be able to reliably call out to Matlab. We could just say that users have to sort this themselves, but that is very unfriendly on OSX and Windows where matlab is not on the path by default. We should probably have some sort of logic that tries to work out which Matlab is installed, and if it fails it should fallback to something like the caching mechanism and store it in the .menpobench config file.

We should probably add Matlab existence to YAML validation as well.

We should validate the YAML to make sure the user has provided a validate schema. Also, it would be nice to have custom types using something like kwalify where we could even validate the Metadata. This would be particularly useful for something like Matlab support and finding the correct Matlab binary path!

Can be tackled after #1

One idea is to have another key in the YAML that specifies which error measure gets calculated:

error_function:
  - ibug_68_48_points_face_size

But this may be overly complicated. Perhaps the number of landmarks expected should be handled by another key? Or perhaps normalisation should be separate?

The biggest problem is something like Intraface which only returns 49 points - what if we trained on 68?

Metadata should come in two forms:

• Optional metadata - probably in the form of a dictionary
  • description
  • publications
  • any other notes
• Mandatory metadata
  • 'Pretty' name
  • Supported platforms - {win32, win64, linux32, linux64, osx64, all}

These should be used in conjunction with the yaml for validating an experiment.

http://www.humansensing.cs.cmu.edu/intraface/

As of 1st of Aug, this still says 'INTRAFACE WILL BE RELEASED ON JULY 15TH 2015'. We could wait and wrap the new one, or just go with the old one in the interim. Maybe work emailing to find out how long until release?

We have some kind of scaling issue that is happening, at least with yzt_iccv_2013.

Right now the x range of the CED is hardcoded to what is sensible for facial images - this should be dynamically set based on the error values.

Cheaper and faster for users this way.

The order that assets are loaded from each dataset should be randomised. Furthermore, when drawing from multiple assets, we should randomise which dataset we load from, rather than just chaining them.

Currently we aren't too careful about the naming of everything in the output directory. If a user runs an experiment with lots of custom modules (e.g. custom dataset, custom lm_processing, custom methods, custom normalisation results) we currently will just hit strange errors in the output. I had a little go at ensuring these problems would be mitigated at least for the error_name here:

        name = '{}_{}'.format(i, Path(error_name).name.replace('.', '_'))

but clearly this requires deeper thought to work for all modules.

You should be able to call something like:

menpobench config cache_dir /my/cache/dir

To update the config values. Particularly useful for things like the Matlab path, in case the automatically detected Matlab path is incorrect.

We currently don't use any of the metadata in making the output results more readable. As a simple example, the display_name_short metadata on methods would be an ideal candidate for use in CED legends and eventually on the website. Problems arise in using this in two areas:

1. Ensuring uniqueness. This is related to #10. As we move away from predefined keys with guaranteed uniqueness to custom paths and arbitrary strings in metadata, we can run into situations where two methods for instance could have the same short name.
2. Reloading after failure. Currently CED plotting only depends on the output dir - the actual modules don't have to be present and loaded. This is nice and self contained. Currently no metadata is persisted to the output dir, so we would have to source and reload all original modules to plot output after the fact. Currently we allow relative paths in experiments to other modules, so although we have the original experiment we don't have the modules there to load.