lisaanne / eccv2016 Goto Github PK

Code to replicate "Generating Visual Explanations"

License: Other

Shell 5.04% Python 94.96%

eccv2016's Introduction

Generating Visual Explanations

This repository contains code for the following paper:

Hendricks, L.A., Akata, Z., Rohrbach, M., Donahue, J., Schiele, B. and Darrell, T., 2016. Generating Visual Explanations. ECCV 2016.

@article{hendricks2016generating,
  title={Generating Visual Explanations},
  author={Hendricks, Lisa Anne and Akata, Zeynep and Rohrbach, Marcus and Donahue, Jeff and Schiele, Bernt and Darrell, Trevor},
  journal={Proceedings of the European Conference on Computer Vision (ECCV)},
  year={2016}
}

This code has been edited extensively (you can see old code on deprecated branch). Hopefully it is easier to use, but please bug me if you run into issues.

Additionally, if you are interested in a pytorch version of this code, you can look at Stephan Alaniz's implementation here. Thanks Stephan!

Getting Started

Please clone my git repo. You will need to use my version of caffe, specifically the "bilinear" branch.
~~Download data using the "download_data.sh" script. This will also preprocess the CUB sentences. All my ECCV 2016 models will be put in "gve_models"~~
My website was deleted after I graduated, so you will have to download data from here.

Building the models

All the models are generated using NetSpec. Please build them by running "build_nets.sh". "build_nets.sh" will also generate bash scripts you can use to train models.

Training the models

If you would like to retrain my models, please use the following instructions. Note that all my trained models are in "gve_models". All the training scripts will be built using "build_nets.sh"

First train the description model ("./train_description.sh"). The learned hidden units of the description model are used to build a representation for the 200 CUB classes.
Run "make_class_embedding.sh" to build the class embeddings
Train definition and explanation_label models ("./train_definition.sh", "./train_explanation_label.sh").
Train sentence classification model ("./train_caption_classifer.sh"). This is needed for the reinforce loss. I found that using an embedding and LSTM hidden dim of 1000 and dropout of 0.75 worked best.
Train the explanation_dis and explanation models ("./train_explanation_dis.sh", "./train_explanation.sh"). These models are fine-tuned from the description and explanation_label model respecitvely. The weighting between the relevance and discriminative loss can impact perforance substantially. I found that loss weights of 80/20 on the relevance/discriminative losses worked best for the explanation-dis model and that loss weights of 110/20 on the relevance/discriminative losses worked best for the explanation model.

Evaluate

Please use the bash scripts eval_*.sh to compute image relevance metrics. To compute class relevance metrics, run "analyze_cider_scores.py". This relies on precomputed CIDEr scores between each generated sentence and reference sentences from each class. You can recompute these using "class_meteor_similarity_metric.py" but this will take > 10 hours.

Please note that I retrained the models since the initial arXiv version of the paper was released so the numbers are slightly different, though the main trends remain the same.

eccv2016's People

Contributors

Stargazers

Watchers

eccv2016's Issues

Compute class relevance scores

Hello Lisa,

Thank you for sharing your code!
I was interested in understanding how you compute the class relevance scores (Similarity and Rank).
At the end of the README, you mention that in order to compute these scores one should run the "analyze_cider_scores.py" script.
However, I do not seem to find it in the repo...could you please point me to it?
(I have also looked for the "class_meteor_similarity_metric.py" and can't find it either.)

Thank you,
Isabel

./data/cub/descriptions_bird.train.fg.json not available!

using https://github.com/LisaAnne/ECCV2016/blob/master/utils/preprocess_captions.py
I can generate

descriptions_bird.test.fg.json using test.txt
descriptions_bird.val.fg.json using val.txt
descriptions_bird.train_noCub.fg.json using train_noCub.txt
test.txt, val.txt train_noCub.txt are provided at https://drive.google.com/drive/folders/1nU9ATTLHAM6_jz-K6hoVlDzNrFcOXtyH (data.zip)

however, there is not any train.txt? what is train_noCub.txt?
could you please help me in this regard?

Help the make_class_embedding error!!!

When I run the make_class_embedding.sh, the got a './make_class_embedding.sh: 行 22: 1950' error! Is it my dataset error?

Regarding Computation of Class Embeddings

Hi Lisa,

Thanx a lot for your code. I'm running into a problem while using your code. For computing the class embeddings, one needs to run the description model. The steps in your README direct one to use the make_class_embeddings.sh file, which runs the following code :

python utils/extract_train_val.py --model_name $model
--LM_net $word_net
--dataset_name $dataset_name
--split_name $split_name
--vocab $vocab
--image_net $image_net
--size_input_feature $size_input_feature
--prev_word_restriction
--precomputed_h5 data/CUB_feature_010517.p

There is no such utils/extract_train_val.py file in the repo. Moreover, this file isn't generated by the build_models.sh code either. So I'm confused, and am unable to use the code. Would be glad if you can clear this doubt.

Regards,
Saurabh

Requesting Prototxt files for models?

Hi,

Thank you very much for your code. I would like to generate explanations using your models without retraining. May I please get required prototxt files for that? Or May I please know how can I generate those files?

Thank you in advance!

CUB data access

Hello and thank you for your work!
I am trying to reproduce the results from GVE but the URL:
https://people.eecs.berkeley.edu/~lisa_anne/generating_visual_explanations/data/ is now password protected.
Is there any other way to access the data files of CUB?
Thank you in advance

Result of eval_explanation_label.sh

Hi Lisa,

We ran the script eval_explanation_label.sh and got the json file created in below path.
generated_sentences/birds_fg_test/gve_models/explanation-label_1006_pred/beam1/generation_result.json

We wanted to know if this is final output for explanation labels for the birds data set you have considered. And we need to extract the labels from this json to display them in a more readable format. Or is there any additional step which we are missing to generate explanation labels? Please advice.

Rgds,
Chandra.

AttributeError: Assignment not allowed (no field "layer" in protocol message object).

We have downloaded your ECCV2016 code from your git repository and were able to successfully install the code. We were able to run the demo of coco-caption-eval in data/coco/coco-caption-eval successfully. Post that we executed the below steps from examples/ECCV2016 folder and were successful.

Getting Started

Please clone my git repo. My code for generating visual explanations is in /examples/ECCV2016. You will need to use my version of caffe.
Download data using the "download_data.sh" script. This will also preprocess the CUB sentences. All my ECCV 2016 models will be put in "gve_models"

But we got stuck on the building the models using below instructions.

Building the models

All the models are generated using NetSpec. Please build them by running "build_nets.sh". "build_nets.sh" will also generate bash scripts you can use to train models.

We are getting errors, details of which are given below. We are debugging from our end by understanding your implementation. But due to time constraint we wanted to check if you can through some light on these errors which can help us to proceed further.

./build_nets.sh
.........EXECUTING CHILD...........
['/sbin/ldconfig', '-p']
.........EXECUTING CHILD...........
['/sbin/ldconfig', '-p']
In main and parsing arguments
Traceback (most recent call last):
File "build_nets.py", line 218, in
float(args.lstm_drop))
File "build_nets.py", line 130, in caption_classifier
model_train.caption_classifier(save_file_train, accuracy=False, loss=True, embed_drop=embed_drop, lstm_drop=lstm_drop)
File "/home/chandra/LisaAnne/ECCV2016-ECCV2016/examples/ECCV2016/build_net/lrcn.py", line 160, in caption_classifier
self.write_net(save_file)
File "/home/chandra/LisaAnne/ECCV2016-ECCV2016/examples/ECCV2016/build_net/caffe_net.py", line 32, in write_net
write_proto = self.n.to_proto()
File "../../python/caffe/net_spec.py", line 178, in to_proto
top._to_proto(layers, names, autonames)
File "../../python/caffe/net_spec.py", line 92, in _to_proto
return self.fn._to_proto(layers, names, autonames)
File "../../python/caffe/net_spec.py", line 153, in _to_proto
assign_proto(layer, k, v)
File "../../python/caffe/net_spec.py", line 74, in assign_proto
setattr(proto, name, val)
AttributeError: Assignment not allowed (no field "layer" in protocol message object).
.........EXECUTING CHILD...........
['/sbin/ldconfig', '-p']
.........EXECUTING CHILD...........
['/sbin/ldconfig', '-p']
In main and parsing arguments
Traceback (most recent call last):
File "build_nets.py", line 203, in
build_sentence_generation_model('definition', True, False)
File "build_nets.py", line 102, in build_sentence_generation_model
model_train.make_sentence_generation_net(save_file_train, accuracy=False, loss=True)
File "/home/chandra/LisaAnne/ECCV2016-ECCV2016/examples/ECCV2016/build_net/lrcn.py", line 116, in make_sentence_generation_net
self.write_net(save_file)
File "/home/chandra/LisaAnne/ECCV2016-ECCV2016/examples/ECCV2016/build_net/caffe_net.py", line 32, in write_net
write_proto = self.n.to_proto()
File "../../python/caffe/net_spec.py", line 178, in to_proto
top._to_proto(layers, names, autonames)
File "../../python/caffe/net_spec.py", line 92, in _to_proto
return self.fn._to_proto(layers, names, autonames)
File "../../python/caffe/net_spec.py", line 153, in _to_proto
assign_proto(layer, k, v)
File "../../python/caffe/net_spec.py", line 74, in assign_proto
setattr(proto, name, val)
AttributeError: Assignment not allowed (no field "layer" in protocol message object).
.........EXECUTING CHILD...........
['/sbin/ldconfig', '-p']
.........EXECUTING CHILD...........
['/sbin/ldconfig', '-p']
In main and parsing arguments
Traceback (most recent call last):
File "build_nets.py", line 205, in
build_sentence_generation_model('description', False, True)
File "build_nets.py", line 102, in build_sentence_generation_model
model_train.make_sentence_generation_net(save_file_train, accuracy=False, loss=True)
File "/home/chandra/LisaAnne/ECCV2016-ECCV2016/examples/ECCV2016/build_net/lrcn.py", line 116, in make_sentence_generation_net
self.write_net(save_file)
File "/home/chandra/LisaAnne/ECCV2016-ECCV2016/examples/ECCV2016/build_net/caffe_net.py", line 32, in write_net
write_proto = self.n.to_proto()
File "../../python/caffe/net_spec.py", line 178, in to_proto
top._to_proto(layers, names, autonames)
File "../../python/caffe/net_spec.py", line 92, in _to_proto
return self.fn._to_proto(layers, names, autonames)
File "../../python/caffe/net_spec.py", line 153, in _to_proto
assign_proto(layer, k, v)
File "../../python/caffe/net_spec.py", line 74, in assign_proto
setattr(proto, name, val)
AttributeError: Assignment not allowed (no field "layer" in protocol message object).
.........EXECUTING CHILD...........
['/sbin/ldconfig', '-p']
.........EXECUTING CHILD...........
['/sbin/ldconfig', '-p']
In main and parsing arguments
Traceback (most recent call last):
File "build_nets.py", line 207, in
build_sentence_generation_model('explanation-label', True, True)
File "build_nets.py", line 102, in build_sentence_generation_model
model_train.make_sentence_generation_net(save_file_train, accuracy=False, loss=True)
File "/home/chandra/LisaAnne/ECCV2016-ECCV2016/examples/ECCV2016/build_net/lrcn.py", line 116, in make_sentence_generation_net
self.write_net(save_file)
File "/home/chandra/LisaAnne/ECCV2016-ECCV2016/examples/ECCV2016/build_net/caffe_net.py", line 32, in write_net
write_proto = self.n.to_proto()
File "../../python/caffe/net_spec.py", line 178, in to_proto
top._to_proto(layers, names, autonames)
File "../../python/caffe/net_spec.py", line 92, in _to_proto
return self.fn._to_proto(layers, names, autonames)
File "../../python/caffe/net_spec.py", line 153, in _to_proto
assign_proto(layer, k, v)
File "../../python/caffe/net_spec.py", line 74, in assign_proto
setattr(proto, name, val)
AttributeError: Assignment not allowed (no field "layer" in protocol message object).
.........EXECUTING CHILD...........
['/sbin/ldconfig', '-p']
.........EXECUTING CHILD...........
['/sbin/ldconfig', '-p']
In main and parsing arguments
Traceback (most recent call last):
File "build_nets.py", line 211, in
sentence_generation_reinforce('explanation-dis', orig_proto='prototxt/description_train.prototxt', classify_model=args.classify_model, classify_weights=args.classify_weights, weights=args.weights, class_conditional=False, lw=80) #Loss weight parameter chosen by parameter search
File "build_nets.py", line 169, in sentence_generation_reinforce
model_train.lrcn_reinforce(save_name=save_file_train, RL_loss=RL_loss, lw=lw)
File "/home/chandra/LisaAnne/ECCV2016-ECCV2016/examples/ECCV2016/build_net/reinforce.py", line 218, in lrcn_reinforce
self.write_net(save_name)
File "/home/chandra/LisaAnne/ECCV2016-ECCV2016/examples/ECCV2016/build_net/caffe_net.py", line 32, in write_net
write_proto = self.n.to_proto()
File "../../python/caffe/net_spec.py", line 178, in to_proto
top._to_proto(layers, names, autonames)
File "../../python/caffe/net_spec.py", line 92, in _to_proto
return self.fn._to_proto(layers, names, autonames)
File "../../python/caffe/net_spec.py", line 153, in _to_proto
assign_proto(layer, k, v)
File "../../python/caffe/net_spec.py", line 74, in assign_proto
setattr(proto, name, val)
AttributeError: Assignment not allowed (no field "layer" in protocol message object).
.........EXECUTING CHILD...........
['/sbin/ldconfig', '-p']
.........EXECUTING CHILD...........
['/sbin/ldconfig', '-p']
In main and parsing arguments
Traceback (most recent call last):
File "build_nets.py", line 213, in
sentence_generation_reinforce('explanation', orig_proto='prototxt/explanation-label_train.prototxt', classify_model=args.classify_model, classify_weights=args.classify_weights, weights=args.weights, class_conditional=True, lw=110) #Loss weight parameter chocen by parameter search
File "build_nets.py", line 169, in sentence_generation_reinforce
model_train.lrcn_reinforce(save_name=save_file_train, RL_loss=RL_loss, lw=lw)
File "/home/chandra/LisaAnne/ECCV2016-ECCV2016/examples/ECCV2016/build_net/reinforce.py", line 218, in lrcn_reinforce
self.write_net(save_name)
File "/home/chandra/LisaAnne/ECCV2016-ECCV2016/examples/ECCV2016/build_net/caffe_net.py", line 32, in write_net
write_proto = self.n.to_proto()
File "../../python/caffe/net_spec.py", line 178, in to_proto
top._to_proto(layers, names, autonames)
File "../../python/caffe/net_spec.py", line 92, in _to_proto
return self.fn._to_proto(layers, names, autonames)
File "../../python/caffe/net_spec.py", line 153, in _to_proto
assign_proto(layer, k, v)
File "../../python/caffe/net_spec.py", line 74, in assign_proto
setattr(proto, name, val)
AttributeError: Assignment not allowed (no field "layer" in protocol message object).
.........EXECUTING CHILD...........
['/sbin/ldconfig', '-p']
.........EXECUTING CHILD...........
['/sbin/ldconfig', '-p']
In main and parsing arguments
Traceback (most recent call last):
File "build_nets.py", line 209, in
build_sentence_generation_deploy()
File "build_nets.py", line 72, in build_sentence_generation_deploy
model_train.make_sentence_generation_deploy()
File "/home/chandra/LisaAnne/ECCV2016-ECCV2016/examples/ECCV2016/build_net/lrcn.py", line 59, in make_sentence_generation_deploy
self.write_net('prototxt/deploy.prototxt')
File "/home/chandra/LisaAnne/ECCV2016-ECCV2016/examples/ECCV2016/build_net/caffe_net.py", line 32, in write_net
write_proto = self.n.to_proto()
File "../../python/caffe/net_spec.py", line 178, in to_proto
top._to_proto(layers, names, autonames)
File "../../python/caffe/net_spec.py", line 92, in _to_proto
return self.fn._to_proto(layers, names, autonames)
File "../../python/caffe/net_spec.py", line 153, in _to_proto
assign_proto(layer, k, v)
File "../../python/caffe/net_spec.py", line 64, in assign_proto
proto_item = getattr(proto, name).add()
AttributeError: 'LayerParameter' object has no attribute 'shape'

`make all` fails for .build_release/tools/convert_imageset.bin

Hi there,

for me compilation fails with the following error message:

CXX/LD -o .build_release/tools/convert_imageset.bin
.build_release/lib/libcaffe.so: undefined reference tocaffe::SampleLayer::Forward_gpu(std::vector<caffe::Blob, std::allocator<caffe::Blob> > const&, std::vector<caffe::Blob, std::allocator<caffe::Blob> > const&)
.b uild_release/lib/libcaffe.so: undefined reference to caffe::SampleLayer::Forward_gpu(std::vector<caffe::Blob, std::allocator<caffe::Blob> > const&, std::vector<caffe::Blob, std::allocator<caffe::Blob> > const&)
collect2: error: ld returned 1 exit status
Makefile:564: recipe for target '.build_release/tools/convert_imageset.bin' failed
make: *** [.build_release/tools/convert_imageset.bin] Error 1

I was able to circumvent the error by declaring the mentioned function in ´sample_layer.cpp´, but that does not seem to be a real solution. Is this a known issue? I am just using the CPU for now, but would eventually like to run the model on a GPU aswell.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.

lisaanne / eccv2016 Goto Github PK

eccv2016's Introduction

Generating Visual Explanations

Getting Started

Building the models

Training the models

Evaluate

eccv2016's People

Contributors

Stargazers

Watchers

Forkers

eccv2016's Issues

Recommend Projects

Recommend Topics

Recommend Org