Reported results and reproduced results about hyperbolic-image-embeddings HOT 12 CLOSED

@ZhiGaomcislab @RongKaiWeskerMA Hi, thanks for raising this issue. My guess is that the problem is caused by --dim argument which is 64 by default and we used 1600 in the paper. We apologize if we haven't mentioned it in the paper, this is the default value used by many few-shot learning papers. Concretely, we are able to reproduce the results on the 1-shot 5-way MiniImageNet task with the following command:
python train_protonet.py --dataset MiniImageNet --way 30 --shot 1 --lr 0.005 --step 80 --gamma 0.5 --c 0.01 --model convnet --hyperbolic --not-riemannian --dim 1600
where the parameters are as reported in the paper (--not-riemannian is a dummy argument to be removed soon) and note that for training --way is set to 30 as commonly done.
Please tell me if this helps!

We will add the training scripts for complete reproducibility to the repo asap.

from hyperbolic-image-embeddings.

Do you use exactly the same hyperparameters in their paper?

from hyperbolic-image-embeddings.

I've tried all the hyperparameters reported in the paper, however, got 40%

from hyperbolic-image-embeddings.

I've tried all the hyperparameters reported in the paper, however, got 40%

Yes, I use the mentioned hyperparameters in the paper, and the final results are about 40%.

from hyperbolic-image-embeddings.

Thanks @KhrulkovV a lot, the performance improved a lot for 5 way 1 shot testing.
Can I ask a question that for 5 way testing, why would we use 30 way in training phase instead?
In terms of the result for 5 way 1 shot MiniImageNet task, the final test accuracy is around 51% which is slightly off with 54% with 4 layers ConvNet, how can I resolve this issue?

from hyperbolic-image-embeddings.

@ZhiGaomcislab @RongKaiWeskerMA Hi, thanks for raising this issue. My guess is that the problem is caused by --dim argument which is 64 by default and we used 1600 in the paper. We apologize if we haven't mentioned it in the paper, this is the default value used by many few-shot learning papers. Concretely, we are able to reproduce the results on the 1-shot 5-way MiniImageNet task with the following command:
python train_protonet.py --dataset MiniImageNet --way 30 --shot 1 --lr 0.005 --step 80 --gamma 0.5 --c 0.01 --model convnet --hyperbolic --not-riemannian --dim 1600
where the parameters are as reported in the paper (--not-riemannian is a dummy argument to be removed soon) and note that for training --way is set to 30 as commonly done.
Please tell me if this helps!

We will add the training scripts for complete reproducibility to the repo asap.

What are the values of dim and way for ResNet-18?

from hyperbolic-image-embeddings.

MINI-R18-1SHOT-5WAY.log
I have tried to run the same for ResNet-18 on the MiniImageNet dataset. The results I obtained is around 29%. I have tried for 30 Way too .. but the results are same.

Can you please help regarding this?

from hyperbolic-image-embeddings.

@sumo8291 Hey, sorry for the late response, I will look into it tomorrow.

from hyperbolic-image-embeddings.

MINI-R18-1SHOT-5WAY.log
I have tried to run the same for ResNet-18 on the MiniImageNet dataset. The results I obtained is around 29%. I have tried for 30 Way too .. but the results are same.

Can you please help regarding this?

I have used the Pretrained models https://drive.google.com/drive/folders/19TdjthkqMKLKSVHrbT5pVEmKvu-6-6iM and have achieved 56.2% accuracy on Resnet18 for 1-SHOT and 30WAY.

from hyperbolic-image-embeddings.

Hello!
The following script should be able to obtain the result from the paper for resnet18 miniImageNet 1-shot task (using the reported params from the paper).

python train_protonet.py --dataset MiniImageNet --way 30 --shot 1 --lr 0.001 --step 80 --gamma 0.5 --c 0.01 --model resnet18 --hyperbolic --not-riemannian --dim 512

from hyperbolic-image-embeddings.

@sumo8291 @RongKaiWeskerMA @ZhiGaomcislab hey, we've uploaded the scripts to reproduce all the results to https://github.com/leymir/hyperbolic-image-embeddings/tree/master/examples/fewshot/configs

Closing now.

from hyperbolic-image-embeddings.

Hello, sorry to follow up on this closed issue, but I'm having problems reproducing the results, even with the new scripts. I have not modified anything -- I literally just ran the scripts.

Here are is what I ran and what I got for CUB 1s5w (conv4):

# CUB 1s5w (conv4) 0.6402 +- 0.002
python train_protonet.py \
  --dataset CUB \
  --shot 1 \
  --lr 0.001 \
  --step 50 \
  --gamma 0.8 \
  --c 0.05 \
  --model convnet \
  --hyperbolic \
  --not-riemannian \
  --dim 1600

### OUTPUT ###
batch 9998: 62.48(49.33)
batch 9999: 62.47(54.67)
batch 10000: 62.47(54.67)
Val Best Acc 0.6809, Test Acc 0.6247
Test Acc 0.6247 + 0.0024

Here are is what I ran and what I got for MiniImageNet 1s5w (conv4):

# MiniImageNet 1s5w (conv4) 0.5443 +- 0.002
python train_protonet.py \
  --dataset MiniImageNet \
  --way 30 \
  --shot 1 \
  --lr 0.005 \
  --step 80 \
  --gamma 0.5 \
  --c 0.01 \
  --model convnet \
  --hyperbolic \
  --not-riemannian \
  --dim 1600

### OUTPUT ###
batch 9998: 52.15(44.00)
batch 9999: 52.15(60.00)
batch 10000: 52.15(42.67)
Val Best Acc 0.5362, Test Acc 0.5215
Test Acc 0.5215 + 0.0020

from hyperbolic-image-embeddings.