eval.py hanging about imagenetv2 HOT 16 CLOSED

Sorry for the issue with the S3 permissions! We accidentally forgot to make a few metadata files public and will do so as soon as possible.

Regarding the selection frequencies for the original ImageNet validation set: as described in our paper (see Section 4), we embedded parts of the original validation set into our MTurk tasks. Hence we also obtained selection frequencies for a subset of the original validation set.

In principle, the selection frequencies are accessible through the MTurkData class in code/mturk_data.py. But right now it has the same permission issues.

I'll update this issue when we have fixed the S3 permissions.

from imagenetv2.

We updated some of the S3 permissions. Can you give the script another try?

Thank you for your help with this!

from imagenetv2.

Thanks for the fix. The script downloading from the S3 bucket is still not working for me but I manually download some of the meta file with wget. But the eval.py is still stucking here:

    There were 85907 duplicate occurences.
    Ignored 0 candidate entries because they are on the blacklist (blacklist size: 31337).
Dataset download:   0%|                                                                                     | 0/10000 [00:00<?, ?it/s]

Key imagenet2candidates_mturk/f77002154a6fe681c5530c3c4434978bc8476640.jpg generated an exception: download backoff failed  imagenet2candidates_mturk/f77002154a6fe681c5530c3c4434978bc8476640.jpg 2.8284271247461907

And I'm not able to download it with wget. Could you also look into this please?

from imagenetv2.

May I ask if there are any updates for the downloading permission yet?
Thanks!!

from imagenetv2.

Yes, we did fix more permissions. Can you give it a try again? You also must have AWS credentials configured to run the script (to avoid an ""Unable to locate credentials." error).

from imagenetv2.

I still got the same error as stated earlier.
Alternatively, can you point me to the location of the files on S3 which contain the frequency information of human labeling for each pictures? (including the original imagenet and the new test sets) I can directly wget those files if I have the permission.

from imagenetv2.

I have the same problem. I'm using aws and have set the AWS credentials configured. But still I'm not be able to made it.

from imagenetv2.

Can you report the exact stack trace again? For the file keroro824 listed above, we can run wget https://s3-us-west-2.amazonaws.com/imagenet2datav2/imagenet2candidates_mturk/f77002154a6fe681c5530c3c4434978bc8476640.jpg, which doesn't know about aws permissions, so this shouldn't still be the problem.

from imagenetv2.

Yes I can download that jpg as well, but when running eval.py with datasets= imagenetv2-b-33, the log is

root@403173:/workspace# cd /understand/new_code/code/
root@403173:/understand/new_code/code# python eval.py --dataset imagenetv2-b-33 --models resnet50
Reading dataset from ../data/datasets/imagenetv2-b-33.json ...
stuck here 10000 ../data/datasets/imagenetv2-b-33.json
Reading from local file /understand/new_code/data/cache/metadata/imagenet_metadata_2018-09-14_01-26-58_UTC.pickle ... done
Loaded 208145 unique candidates from 230 search result JSON file(s).
    /understand/new_code/data/search_results/...
        2018-07-31_flickr_search_result_vaishaal_class_1_153.json
        2018-08-20-16-10-18_becca.json
        2018-08-25-11-43-09_becca.json
        2018-08-27-22-53-45_becca.json
        2018-08-30-02-40-26_becca.json
        2018-08-30-18-46-35_becca.json
        2018-08-30-19-31-10_becca.json
        2018-09-04-17-03-01_becca.json
        2018-09-04-17-36-03_becca.json
        2018-09-05-16-16-14_becca.json
        ...
    There were 85907 duplicate occurences.
    Ignored 0 candidate entries because they are on the blacklist (blacklist size: 31337).
or here
Dataset download:   0%|                                                                                      | 0/10000 [00:00<?, ?it/s]

Key imagenet2candidates_mturk/1d46513c782329bf5778467b0747f97963d94d46.jpg generated an exception: download backoff failed  imagenet2candidates_mturk/1d46513c782329bf5778467b0747f97963d94d46.jpg 2.8284271247461907

I tried wget for 1d46513c782329bf5778467b0747f97963d94d46.jpg one and it does work. But maybe the script has some connecting issue?

from imagenetv2.

Can you run aws configure list and check that you have an access_key and secret_key set?

from imagenetv2.

[cheluo@ip-172-31-16-191 ~]$ aws configure list
Name Value Type Location
---- ----- ---- --------
profile None None
access_key ****************PWRO iam-role
secret_key ****************+uet iam-role
region None None

Does this means I have the access_key and secret_key set correctly?

from imagenetv2.

Some diagnosis:

Sync does not work:

aws s3 sync s3://imagenet2datav2/imagenet2candidates_mturk/f77002154a6fe681c5530c3c4434978bc8476640.jpg .                                                                                                                              
fatal error: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied

cp works:

aws s3 cp s3://imagenet2datav2/imagenet2candidates_mturk/f77002154a6fe681c5530c3c4434978bc8476640.jpg ../data/cache/imagenet2candidates_mturk/
download: s3://imagenet2datav2/imagenet2candidates_mturk/f77002154a6fe681c5530c3c4434978bc8476640.jpg to ../data/cache/imagenet2candidates_mturk/f77002154a6fe681c5530c3c4434978bc8476640.jpg

wildcard and --recursive do not work:

aws s3 cp s3://imagenet2datav2/imagenet2candidates_mturk/ ../data/cache/imagenet2candidates_mturk/ --recursive
fatal error: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied

from imagenetv2.

change L260 in utils.py

 client.download_file(bucket, cur_key, local_filename)

to

s3 = boto3.resource('s3')
s3.Bucket(bucket).download_file(cur_key, local_filename)

works for me if anybody has the same issue.

One last issue here:

wget https://s3-us-west-2.amazonaws.com/imagenet2datav2/imagenet_validation_flat/ILSVRC2012_val_00030740.JPEG .  
--2019-03-18 06:26:39--  https://s3-us-west-2.amazonaws.com/imagenet2datav2/imagenet_validation_flat/ILSVRC2012_val_00030740.JPEG
Resolving s3-us-west-2.amazonaws.com (s3-us-west-2.amazonaws.com)... 52.218.245.8
Connecting to s3-us-west-2.amazonaws.com (s3-us-west-2.amazonaws.com)|52.218.245.8|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2019-03-18 06:26:39 ERROR 403: Forbidden.

Could you give the pictures in imagenet_validation_flat the same access as imagenet2candidates_mturk please?

Thank you all for your help!

from imagenetv2.

I've started the script that changes the permissions. They should be public in approximately 30 minutes.

Meanwhile in L260. If you do

 client = boto3.client('s3')
 client.download_file(bucket, cur_key, local_filename)

does it work?

from imagenetv2.

Thanks! I got the imagenet_validation_flat also downloaded.

I'm not sure about that but I just used the same code I mentioned earlier.

from imagenetv2.

Great to hear that you could also download the imagenet_validation_flat images. Let us know in case you run into more issues.

If you have time, could you also try the code snippet Vaishaal posted? We would like to understand what works differently on your side so that we can fix the bug on our side. Thank you!

from imagenetv2.