MRR calculation about kgt5 HOT 4 OPEN

Hi, thanks for your interest.

As mentioned in #8, please try the following:

1. Switch to branch apoorv-dump instead of main. This contains the latest code but is not cleaned, hence the large number of files.
2. In this branch, use eval_accelerate.py to evaluate (use sample size > 200). This saves all the generated predictions and scores to a scores file.
3. Use notebook filtered_eval_from_scores.ipynb which uses these saved scores to evaluate MRR etc. I would suggest you go through the notebook to see the exact way these are calculated (if you see any issues please do contact me)

The notebook contains the logic for MRR calculation from raw scores. Predictions are sorted by score, then 'seen' entities are removed from the prediction list (filtering), and then the rank of the correct entity is used to calculate MRR.

from kgt5.

I meant num_predictions=200 with beam_size=1 . Rest were all left default

from kgt5.

In this branch, use eval_accelerate.py to evaluate (use sample size > 200). This saves all the generated predictions and scores to a scores file.

@apoorvumang, I am sorry, did you mean beam_size or num_predictions instead of sample size? I didn't manage to find such parameter in your code and would like to make sure I understood you correctly. Also, what were other parameters equal to in your settings (like length_penalty, max_output_length, length_normalization, and of course # predictions and beam size)?

from kgt5.

hi, @apoorvumang
As you said, I set (num_reduction) to 10 in (eval_accelerate. py), which has caused my CUDA memory to exceed (30G). How do I set 200?
Thank you for your support！

from kgt5.