chrstn-hntschl / deepvcd Goto Github PK

EarlyStopping in Keras only supports epoch based patience (if metric does not improve after n epochs by a given value, learning is stopped). This may interfere with ReduceLROnPlateau: either we might stop learning before reduction of learning rate has stopped fluctuating or we have a rather high patience for early stopping. An alternative would be to have early stopping only active when a given learning rate has been reached.

Currently, training prints out a progress bar for each epoch. Setting verbosity to 2 decreases this for better logging.

Currently, top5 error of all given weights is computed when running models.alexnet.AlexNet.pyand printed. A summary should provide the best results of all give weights file as well as the average result.

Currently, when loading a dataset using DirectoryLoader and a concepts.jsonfile is present, the concepts listed in that file supersede the concepts in the directory structure. It would be better, to provide a file as parameter on ctor:
https://github.com/chrstn-hntschl/deepvcd/blob/fe2a1718c58047480f396f50d3bf0068af95ce2b/deepvcd/dataset/descriptor.py#L261C9-L261C59

Currently, training is based on hard-coded accuracy metric (val_accuracy or accuracy of no valset is given). Option to select multiple metrics on command line that are evaluated and one for selecting best model weights is preferred.

If weights file given to models.alexnet.AlexNet.pydoes not exist, an error is thrown only during prediction. Check if file exists already at beginning or at least catch error during loading of weights file to give a proper error message.

Currently, a DatasetDescriptor can only be loaded from a yaml file (via YAMLoader.read()). This requires parsing a yaml file which can be slow and files become large for larger datasets. When loading the entire ImageNet-21k dataset, this becomes infeasible.

Instead, loading the descriptor using the directory structure is an option. A directory containing the following structure is required:

`dataset_name

• train
  • class_1
    • image_1.jpg
    • ...
    • image_n.jpg
  • class_2
    • image_n+1.jpg
    • ...
  • ...
• val
  • class_1
    • image_1.jpg
    • ...
  • class_2
    • ...
• test
  • class_1
    • image_1.jpg
    • ...
      ´
      The existence of a ǜaland test`directory can be optional.

lines 89 and 92 of trainAlexNet.py:
if dataset_descriptor is neither file nor directory, deepvcd_ds is None.

ReduceLROnPlateau and EarlyStopping rely on a monitor mode to be set (i.e. wether the monitored metric is expected to increase or decrease). Currently, auto mode is used, which tries to deduce the monitor op from the name (when name contains acc the metric is expected to increase otherwise it should decrease). With the ability to chose the metric via cli, this does not work anymore (we could have arbitrary metric names. Two possible options:

• deduce mode from metric class (have a dict that maps class name to mode)
• have a mode command line parameter

Currently, only one weights file is written, once the training is ended. In case of very long training times it is advantageous to store intermediary results in order to be able to abort training after any epoch and without loosing the achieved results.

currently, trainAlexNet.py holds a hard-coded mean value. Should be loaded from model definition.

when running deepvcd.models.alexnet.AlexNet for prediction with default weights parameter (using imagenet weights loaded from URL), no option to explicitly set the parameter to None is provided. This makes it difficult to properly initialize the value from an env variable (such as e.g. in Makefile).

Instead use:

weights="imagenet" as default