@apsyx
Dear author, we have tried to reproduce the result of your method on Clothing1M.
We use the file in #4 to generate the clean_train, clean_val, noisy_train, clean_test folders.
But the result we obtained is still lower than the result in the paper.

This work propose to approximate the current meta-parameter gradient with information from the previous k step as follows:

I guess the second term is the gradient of alpha for just one step, but I haven't seen why the first term can approximate the previous K.
In the code, author seems to calculate the above equ 4 every iteration, and accumulate the gradients and update meta parameter every K step.

# accumulate discounted iterative gradient
for i, param in enumerate(meta_net.parameters()):
    if param.grad is not None:
        param.grad.add_(gamma * args.dw_prev[i])
        args.dw_prev[i] = param.grad.clone()
if (args.steps+1) % (args.gradient_steps)==0: # T steps proceeded by main_net
    meta_opt.step()
    args.dw_prev = [0 for param in meta_net.parameters()] # 0 to reset 

Could you please help further explain the gamma and how the equ 4 can approximate the K step gradients? Thanks @apsyx

Hi, I have tried to prove equation 4 several times, but I cannot figure out why there is a subtraction in equation 4. I have proved the approximated gradient of meta model from the Meta-Weight-Net. I think the method to get the implicit gradient of validation loss w.r.t. meta model is similar. Can you please share the detailed proof regarding how to get the equation 4-6? thanks.

Hi, dear author.

Thank you for your excellent work!

But when I try to run your code in clothing1M dataset (I follow the instruction in readme), but directory "clean_train" does not exist. Could you provide the related code about this directory?

Thank you very much.

There are important files that Microsoft projects should all have that are not present in this repository. A pull request has been opened to add the missing file(s). When the pr is merged this issue will be closed automatically.

Microsoft teams can learn more about this effort and share feedback within the open source guidance available internally.

Merge this pull request

Hello,
Thanks for the amazing work!
I am planning to run MLC on my custom dataset. I have around 1000 samples in the gold dataset and 10 times more on the noise dataset. The gold dataset is highly imbalanced. I would like to know the how to set the learning rate hyper parameters inorder to save the time. Does the Meta Learning rate needs to be lower than the Main learning rate ? Any insights would be helpful

Hi,
Thanks for sharing your code!
When applied to text classification tasks, convergence is slow and it is difficult to achieve good results.
Can the code related to text classification in the experiment be open sourced?

There are important files that Microsoft projects should all have that are not present in this repository. A pull request has been opened to add the missing file(s). When the pr is merged this issue will be closed automatically.

Microsoft teams can learn more about this effort and share feedback within the open source guidance available internally.

Merge this pull request

If we run the code for the first time. It gives error while creating

• Log file as 'log' directory is not present
• Similarly 'model' directory error while saving the model

To resolve:

1. Either check if exists and is directory dont present then create it.

Steps :

1. Clone repo
2. python3 main.py --dataset cifar10 --optimizer sgd --bs 100 --corruption_type unif --corruption_level 0.4 --gold_fraction 0.02 --epochs 120 --main_lr 0.1 --meta_lr 3e-4 --runid cifar10_run --cls_dim 128

Have you ever implemented the update_params function for Adam optimizer?
If so, how is the performance compared to the SGD-based methods?

Require the list of package that is required to train the model

The default value of gradient_steps is set to 1 (main.py), which seems that the first term of the eqn 6 in your paper is never used.

if (args.steps + 1) % (args.gradient_steps) == 0: # T steps proceeded by main_net
meta_opt.step()
args.dw_prev = [0 for param in meta_net.parameters()] # 0 to reset

The condition in the if expression is always True.

Hi,
thanks for sharing your implementation. I have two questions about it:

1. Does it also work on tabular data?
2. Is it possible to identify the noisy instances (return the noisy IDs or the clean set)?

Thanks!

Hi Guoqing,

Thanks for sharing the code! I am trying to run your code and find that in the cifar dataset file, the "utils" file is missed. And I found it in the clothing dataset file.

In addition, it seems like the "wideresnet" imported in "load_corrupted_data_mlg" is also missed, I didn't find it in other files.
Could you please tell me where to find it or did I miss anything?

Thx

Hi,

Thanks for your great work! When I tried to run the code on CIFAR-10 with the command provided, I found the module 'load_corrupted_data_mlg' is missing. Did you miss this file or should I install other libraries?