philkr / voc-classification Goto Github PK

Train and evaluate VOC 2007 / 2012 image classification models. We used this code to evaluate several feature learning algorithms.

License: BSD 2-Clause "Simplified" License

Python 96.53% Shell 3.47%

voc-classification's Introduction

VOC 2007 / 2012 classification

This code trains and evaluates a 20-way binary classifier on the PASCAL VOC dataset. It uses the stardard sigmoid cross entropy loss and evaluates mAP.

Setup

Clone this repo and also get my caffe future branch https://github.com/philkr/caffe/tree/future (I use a few custom layers, mainly HDF5 related) and compile it.

Download the VOC 2007 dataset from http://host.robots.ox.ac.uk/pascal/VOC/voc2007/.

Setup your own user_config.py, see user_config.example.py for an example. You'll need to setup CAFFE_DIR to point to the caffe future branch, and VOC_DIR to point to the Pascal VOC directory.

Now everything should be ready.

Running the evaluation

The basic evaluation is run through

python3 src/train_cls.py path/to/your.prototxt path/to/your.caffemodel

If you want to automatically train all variations fc8 only, fc6-fc8 and conv1-fc8 for your model you can use the make_matrix.sh script. It relies on the wait_for_gpu script, which tries to wait for a free gpu before running a job (this is a very crude solution and you probably want to use something like slurm in production).

bash make_matrix.sh experiments/ALL_MODELS experiments/ALL_PARAMS output/directory

This script will run every model described in experiments/ALL_MODELS with every parameter setting described in experiments/ALL_PARAMS. You can then plot the results using the make_table.py script.

python3 make_table.py output/directory

NOTE: make_table.py currently doesn't to a perfect job in figuring out what parameters and models were used. If somebody want to fix this be my guest.

Overfitting and 'tuning' the classifier

I have found the default settings to work in all the evaluations and comparisons I have performed. I'm sure there are certain learning rate and weight decay settings that will work better for some models. I however do not have the time to find them. The current parameters are tuned such that alexnet trained on imagenet works ok mAP of 80%. If you want to tune the parameters for your own model, be my guest, but please do it also for all baseline models too!

Results

I'll publish a up to date list of the top performing methods here, as soon as I get a chance.

voc-classification's People

Contributors

Stargazers

Watchers

voc-classification's Issues

Is there a python2.x version of this code?

Because Caffe hasn't officially support python3, I met lots of troubles trying to use it.

dimension mismatch in test evaluation

The code to evaluate the test mAP fails due to a factor of 10 mismatch in the ground truth and model scores

Evaluating on test set:

File "src/train_cls.py", line 169, in <module>
    ap = metrics.average_precision_score(gts[i][gts[i]<=1], scr[i][gts[i]<=1]-1e-5*gts[i][gts[i]<=1])
IndexError: boolean index did not match indexed array along dimension 0; dimension is 49520 but corresponding boolean dimension is 4952

Evaluating on val set:

  File "src/train_cls.py", line 169, in <module>
    ap = metrics.average_precision_score(gts[i][gts[i]<=1], scr[i][gts[i]<=1]-1e-5*gts[i][gts[i]<=1])
IndexError: boolean index did not match indexed array along dimension 0; dimension is 25100 but corresponding boolean dimension is 2510

Here the ground truths are shape (20, X) but the scores are (20, X*10). I'm running on python 2.
My test_resize_layer.cpp failed in the 'make test' compiling so I removed it. Perhaps this is causing those shape mismatches now? Thanks for any tips.

VOC Classification Code is not working for my model that has the alexnet structure

Hi,

I have a pretrained model in torch and I have transfered the weights and biases to the caffe, and I checked if the model is transfered correctly. The problem is that when I want to evaluate my model for pascal dataset with philkr VOC classification code (https://github.com/philkr/voc-classification), it gives me 7% and it seems that it cannot finetune. I want to use the code with clipping from drop6 instead of drop7 which is the default in the code and I changed the code accordingly, but I still get the same result.I was wondering how I can use the code correctly?I really appreciate any help

Thank you so much,

Size mismatch in SigmoidCrossEntropyLoss

I met this bug and have no idea why it happenes

Traceback (most recent call last):
File "src/train_cls.py", line 121, in
s.run(args.nit)
File "/home/jason/egomotion/experiment/voc2007_cls/voc-classification/src/solver.py", line 90, in run
s.step(1)
File "/home/jason/egomotion/experiment/voc2007_cls/voc-classification/src/python_layers.py", line 54, in forward
lZ = np.log(1+np.exp(-np.abs(f))) * mask
ValueError: operands could not be broadcast together with shapes (10,20) (16,20)