osainz59 / ask2transformers Goto Github PK

This seems like it would be really cool to use but unfortunately the example code in the readme is no longer valid, and I've been struggling for a long time to get this thing to work. It seems "NLITopicClassifier" now requires an additional argument to specify which pretrained model to use, but not all of them work.

Provide further explanation or documentation?

Excuse me,I want to reproduce the results in your team's paper: Textual Entailment for Event Argument Extraction: Zero and Few-Shot with Multi-Source Learning.However , readme.txt hadn't provide the full procedures to reproduce the Event Argument Extraction results.So I want to know how to reproduce the result.

Hi,
Thank you for sharing this wonderful work! Could you detail the hyperparameters of fine-tuning TACRED?
I run the experiments with

python run_glue.py \ --model_name_or_path roberta-large-mnli \ --train_file /data/tacred/train.mnli.json \ --validation_file /data/tacred/dev.mnli.json \ --do_train \ --do_eval \ --max_seq_length 128 \ --per_device_train_batch_size 32 \ --learning_rate 4e-6 \ --num_train_epochs 2 \ --overwrite_output_dir \ --fp16 True \ --gradient_accumulation 1 \ --output_dir ./results \ --save_steps 5000 \ --seed 0 \ --warmup_steps 1000

and then evaluate the final results with
python evaluation.py --config ../resources/predefined_configs/tacred.relation.config.json (modify the "nli_models" with tuned model path)

the test resutls are
"test": {"optimal_threshold": 0.5, "positive_accuracy": 0.9004511278195488, "precision": 0.43177373251090717, "recall": 0.8631578947368421, "f1-score": 0.5756117127958283 }

Could u kindly correct me how to tune and evaluate? Thank you!

I am still working on the documentation, so if you miss something, please consider posting it here!

Thank you!

Hi, I am just wondering if it is possible for you to release fewshot checkpoints for TACRED. Thx!

Thanks for sharing the code you used in your research, it's really useful!

Before coming across your research, I've seen some other papers using NSP for topic classification, and accuracy of NSP models were almost on par with NLI models (Ma et al. 2021; Sun et al. 2021). So I was surprised to see NSP perform as bad as a random model.

At first I thought this might have happened because of the data you used. However, I saw that you defined default positive output for NSP as 1 in this line. HuggingFace documentation for NSP gives this example:

outputs = model(**encoding, labels=torch.LongTensor([1]))
logits = outputs.logits
assert logits[0, 0] < logits[0, 1]  # next sentence was random

I may be wrong, but I think last line of this example says that the output with index 0 is the positive output (isNext). I am not certain if this is the problem, but I think we should look into that.

Thank you.

Hi,
Thanks for the framework.
By when tutorial or examples will be available so that we can have a way to utilise the same.
If you have any related to Text Classification & Relation Extraction.. please share.
Regards,
Gaurav

Hi Sainz,
I am trying to reproduce results on your paper "Textual Entailment for Event Argument Extraction: Zero- and Few-Shot with Multi-Source Learning", but cannot find the method of verbalization either in the paper or code. In Label verbalization section, you mentioned that "A verbalization is generated using templates that have been manually written based on the task guidelines of each dataset.", may I know how a sentence is generated after giving the model labels, template and original sentence? For example, when filling the template of " bought something" in Figure 1, how does the model know to choose "John D. Idol" but not "hired"? Please kindly reply when you have time.

Hi!
Do you have any plans for releasing the code for fine-tuning the relation extraction model?

Hi,
Thank you for this amazing repository, it is exactly what I was looking for! And congratulations for this work :)

I have two doubts about the fine-tuning process for Few-Shot RC:

1. I successfully used the provided script to convert TACRED data to MNLI, however, the run_glue.py script takes too long to train (I'm using Colab and it's taking like 23 hours per epoch, with a huge number of examples and optimization steps even with tiny splits of the dataset ). Am I missing something? These are the parameters I used:

python Ask2Transformers/a2t/relation_classification/run_glue.py --task_name mnli --train_file train.mnli.json --validation_file dev.mnli.json --test_file test.mnli.json --model_name_or_path roberta-large-mnli --output_dir output --cache_dir cache --do_train --do_eval --do_predict --seed 6 --per_device_train_batch_size 16 --overwrite_output_dir

2. After running the run_glue.py script, is the fine-tuned model supposed to be found in the output (--output_dir) directory? (I'm using hugging face transformers). Sorry if this question seems dumb, I'm just not sure how to proceed and use the fine-tuned model after training.

I would appreciate if you could give me some guidance about this process.
Thank you!

In the templates in "tacred.relation.config.json", for the relation "per:date_of_birth", there is the following template.
"{subj}\u00e2\u0080\u0099s birthday is on {obj}."

Shouldn't it be as below?
"{subj}'s birthday is on {obj}."

It looks like the apostrophe character (') is replaced with \u00e2\u0080\u0099. Is that a typo or is it intentional?

There is also the same issue in the following template as well.
"{obj} is the cause of {subj}\u00e2\u0080\u0099s death."

For your zero-shot Tacred relation classification results with 1% dev (see the above picture), the paper says you did 100 runs. Did you use the following set as the 1% dev set for all 100 runs? If not, how did you create 100 1% dev sets?

Ask2Transformers/resources/tacred_splits/dev/dev.0.01.split.txt