drsy / dgen Goto Github PK

Hi! @DRSY

I am researching your paper Knowledge-Driven Distractor Generation for Cloze-Style Multiple Choice
Questions for my graduation topic. I already trained a model, and I want to use your dataset to evaluate my model.

Could you tell me what dataset are you used for evaluation in paper?
Is it the file in DGen/Layer1/dataset/total_new_cleaned_test.json ?

thanks!

Can you please give instructions in which order we can implement it and test it?

In the code, there is a mention about "cosine similarity from BERT-based embeddings but observed longer inference time and
similar performance", and this version of code is using SBRT similarity, which is missing "from Layer2.Fine_tuned_BERT import get_similarity_from_SBERT" file.

So in order to implement the paper settings, we need to replace this sbert similarity function to LDA similarity function.
(

Hey can you please elaborate how did you find recall@3. The cosine similarity between true distractors and the predicted distractors will lie between 0 and 1. Please elaborate how did you convert this fraction into a number where your recall@3=12.98
Please let me know if i have understood wrongly.
Let me elaborate my doubt:
For example original distractors where = ['red', 'black', 'blue']
And Predicted distractor is =['red','yellow','green']
Then the cosine similarity would be (returned values from word2vec similarity function): [1,0.8,0.5]
Similarly for n generated questions you get n such lists of length=3
That is: [
[1,0.8,0.5],
[0.7,0,0.04],
[0.3,0.8,0.2],
*
*
*
[0.2,0.4,0.6]
]
Now how did you calculate recall@3 or precision@3 ??
@DRSY