Using testing data in place of evaluation for selecting best model? about attentionexplanation HOT 2 CLOSED

Hi Sarah,
Thanks for pointing this out ! You are correct that we used test data to select our model. It was a deliberate choice on our part. Essentially, in this paper, we are not presenting new predictive models and as such we wanted to present our analysis on best possible version of our model for the test data (we believe this biases the model in favor of the attention rather than against it).

Second, we ran all our models using dev dataset too and we found that didn't actually made a difference in our results. I will push the code and results to this repository for the same by either tomorrow or Thursday.

Again, thanks for bringing this to our attention and we point to this choice in the main README for the project.

from attentionexplanation.

Hi Sarthak,
Thanks for the response and clarification.

from attentionexplanation.