declare-lab / cicero Goto Github PK

A file which lists the requirements needs to be created, out of the box there are some dependency related errors.

Hello, I am an NLP researcher in South Korea and have a question about the ceiling performance of CICERO multiple choice task.
As you already know, many previous works regarding commonsense benchmarks set a ceiling performance by measuring human performance. (e.g. SocialIQA, CosmosQA, etc.)
You did conduct human evaluation on gold annotations for CICERO generation task, but I could not find human performance on CICERO multiple choice task.
Could you explain why the experiment for that particular task does not contain human performance?
I am looking at the paper accessed from here.
https://aclanthology.org/2022.acl-long.344/

I am impressed with your outstanding work CICERO and will be glad to receive your response soon.
Thank you.

After I finished pretraining, I countiuned to run scrpits/run_finetuning.sh and I got this error:
FileNotFoundError: Unable to find '/data/wlr/CICERO/v2/data/cicero_v2/preprocessed/train_multi.json' at /data/wlr/CICERO/v2

I can't find any way by following your instructions to build "./data/cicero_v2/preprocessed/" dir and there is no way to save these preprocessed data from pretraining code.

In other words, MCQ seq2seq model generates the answers. How do you convert it to accuracy?