Unable to reproduce result about hitter HOT 4 CLOSED

Could you check those filtered_with_test metrics? I noticed that there is a big gap between your dev and test results, which could be for this reason.

from hitter.

Thanks for pointing it out. Using filtered_with_test metrics, I get
Dev set
MR=156.61, MRR= 0.375, Hits1 = 0.2815, Hits10= 0.5628 which is better than reported.

BTW, do I understand it correctly that filtered_with_test metric considers the test set for filtering in addition to dev and test set? Also, I=is there any equivalent of filtered_with_test metric for the test set evaluation?

from hitter.

Yes, the reported results are averages.

Yes, evaluation in each set filters out the examples in train/dev and itself by default, so evaluating in the test set is also filtered_with_test just not reported under this name.

You can check eval.filter_splits and its explanation here.

from hitter.

Cool, thanks for the clarification!

from hitter.