kohpangwei / group_dro Goto Github PK

Hi,

I cannot open the link that you provide (https://nlp.stanford.edu/data/dro/waterbird_complete95_forest2water2.tar.gz) for waterbird dataset. Could you create a new link to access the dataset? Thanks a lot!

With the command in repo for MNLI, I am not able to reproduce your results.
Could you send me the command which reproduce your best results on CUB and CeleBA?
Especially for that with Large Weight Decay and Early stop
Thanks a lot.

The link to the Codalab worksheet is broken. The page shows "Not found: '/worksheets/0x621811fe446b49bb818293bae2ef88c0'." Could you please update it? Thank you!

Hi
Could you provide the command to re-generate your cached_mnli_files. Also, Is this possible to have the code working from the raw text data of MNLI. thanks.

Hi,

Could you provide these files?

• list_attr_celeba.csv
• list_eval_partition.csv

It would be easier to just use what you already have, instead of converting them on my own.

Edit:
For now, I will be using these:
https://raw.githubusercontent.com/togheppi/cDCGAN/master/list_attr_celeba.csv
https://raw.githubusercontent.com/Golbstein/keras-face-recognition/master/list_eval_partition.csv

You could just close this issue.

First, thanks for the nice work!
In your paper you show the following:

I am having difficulty trying to find the part in your code corresponding to randomly picking a group: g~Uniform(1,...m)
Please could you tell me where can I find it?

It seems that there is no l2-penalities implementation in this code. Should I implement it myself to reproduce the results in the paper?

Hi @kohpangwei and @ssagawa ,

Your paper mentioned that "All benchmark models were evaluated at the best early stopping epoch (as measured by robust validation accuracy)." However, your code

Did you use the model selection rule (i) for ERM in your paper's experiments (e.g., Table 3)? I'm trying to reproduce your results, but I'm not sure if your results on ERM are from (i) or not.

Following up on previous question #7
Please can you clarify whether we need to sample only one group at each iteration or it is OK to have multiple groups in a batch? In your algorithm, it seems to say that we need to sample only one group at each iteration, but this doesn't seem to be the case in the code.

Additionally, please can you comment on the following remark from this paper https://arxiv.org/abs/2010.12230 ?

Is this remark justified?

Is there other places where I can download it?
Thanks!

Dear authors, Thank you for sharing a well polished codebase!

For the results in table 1, I have noticed that DRO method is always run with "reweight_groups" flag set to "True", whereas the same flag is "False" for the ERM algorithm [1]. As per the code, the "reweight_groups" flag performs a weighted random sampling guaranteeing an equal count of each group in any given batch. On the other hand, the ERM algorithm receives a smaller count of the minority sample as there is no weighted random sampling. Such a difference in implementation between ERM and GroupDRO suggests for an unfair comparison between the two methods.

Surely, as pointed out in the comment [2], the loss function could be considered unaffected by the "reweight_groups" flag as the DRO method uses the mean of per-group losses. However, the empirical estimate of these means in a given batch would be highly noisy when the sample count of the minority group is very small. This makes me wonder (and I hope it's okay for me to ask), that the gains reported in the paper are attributed solely to the use of weighted random sampling procedure rather than DRO update rule? Please clarify

Do you have any comparisons of the DRO algorithm with "reweight_groups" flag set to "False"? How does ERM with "reweight_flag=True" compare to ERM with "reweight_flag=False"?

Thank you

[1] https://worksheets.codalab.org/worksheets/0x621811fe446b49bb818293bae2ef88c0
[2]

Hi
With the command in repo for MNLI, I am not able to reproduce your results.
Could you send me the command which reproduce your best results on MNLI?
Basically seems some argument are missing from that command. thanks.

Thanks for the good work,
Can you please tell me how to obtain three datasets' ERM baseline, e.g., the commands?