dst evaluation about simpletod HOT 7 OPEN

sorry, I can't reproduce that, I got 50.46 joint acc.

from simpletod.

The results of the latest arXiv paper and NIPS official paper are 55.72, 55.76 and 57.47. Can you reproduce this result?

from simpletod.

sorry, I can't reproduce that, I got 50.46 joint acc.

I saw the similar results in #5 (comment)
After ignoring both none and dontcare, the results are close to those in the paper, but it is unfair!

from simpletod.

I agree. Thank you !

from simpletod.

Hello, I am also trying to reproduce the result, I notice there are many checkpoints saved, which checkpoint do you use? How do you figure out which one is the best?

from simpletod.

Based on this commit
917f66a

They were originally removing both none and dontcare during evaluation (which is probably the paper version)
and this schema is unfair compared to the TRADE setting which counts dontcare.

The current version is fixed as the author mentioned.

from simpletod.

Can someone tell me how to reproduce the dst results?(joint acc around 50 is also fine).

I see no belief states generated when I run generate_dialogue.py as in README.
dialogue_aggregated_pred_belief is empty.

from simpletod.