Unable to obttain the results written in the paper about glue-baselines HOT 10 OPEN

The reference to jiant above was to v1.3: https://github.com/nyu-mll/jiant-v1-legacy

The new v2.0 is mostly a wrapper around Transformers, so it drops LSTM support. Start with v1.3.

from glue-baselines.

You'll have a much easier time with jiant than with this repo, but if you need an exact reproduction for some reason, ping w4ngatang again.

from glue-baselines.

Hi!

The results in the paper are test set results (as it says in the caption), and several datasets have non-trivial differences between the dev and test data, so it's possible that you've already reproduced our results exactly.

In any case, though, I'd urge you to use the newer jiant codebase. It's much better documented, and gets strictly better results than the baselines here. We don't have public dev set numbers from that codebase yet, but if you post an issue there, we should be able to assemble some.

https://github.com/jsalt18-sentence-repl/jiant

If you do need to use this codebase, reply here and @W4ngatang should be able to share the exact hyperparameters we used.

from glue-baselines.

Thanks for your enthusiastic reply!

I've submitted it to the GLUE platform, but there are still some gaps in CoLA, QNLI and WNLI.

  | average | CoLA | SST-2 | MRPC | STS-B | QQP | MNLI | QNLI | RTE | WNLI
BiLSTM baseline | 63.5 | 24 | 85.8 | 82.1/71.9 | 68.8/67.0 | 59.1/80.2 | 65.8/66 | 71.1 | 46.8 | 63.7
my results | 60.4 | 13.9 | 84.6 | 81.6/73.0 | 68.8/66.7 | 57.2/79.7 | 61.3/61.8 | 63 | 54.2 | 52.7

So it will be very nice of you to offer me the hyperparameters which produces the baselines on this codebase.

Besides, I'm willing to transfer to jiant, but I'm not sure whether I can produce the GLUE baselines with it. Can I obtain the results on the leaderboard using the final_glue_runs.sh script without modification?

Thanks again!

from glue-baselines.

@W4ngatang - Could you take this one?

If you need to exactly match our baselines, jiant won't do that. This paper publishes numbers from the final_glue_runs script, though: https://openreview.net/pdf?id=Bkl87h09FX

Sam

from glue-baselines.

Hey @xxxxxyt , I've added the exact scripts that I'm running here. Could you try running those?

from glue-baselines.

Hey, after fixing lots of issues, I tried running the code. However, I still get the following error:

Traceback (most recent call last):
File "src/main.py", line 280, in
sys.exit(main(sys.argv[1:]))
File "src/main.py", line 186, in main
trainer = MultiTaskTrainer.from_params(model, args.run_dir + '/%s/' % task.name,
NameError: name 'MultiTaskTrainer' is not defined

I find that the 'MultiTaskTrainer' is not defined in the repository. I sincerely asking for the script for 'MultiTaskTrainer'. My great gratitude! @thxyutong @sleepinyourhat

from glue-baselines.

@Bogerchen hey bro, have you fix the problem?

from glue-baselines.

Running into the same MultiTaskTrainer issue. Did someone find a fix? Also @sleepinyourhat concerning jiant I tried using it but found no options for running non-transformers architectures (I want to rerun the LSTM as described in the GloVe paper). Maybe I missed something? Would appreciate you pointing the right way to do it :)

from glue-baselines.

Can someone share the MultiTaskTrainer script? I really need this script to reproduce exactly the original GLUE benckmark. Thanks.

from glue-baselines.