jiacheng-ye / zerogen Goto Github PK

Hi. From my understanding, there could be a label leakage in the prompt when doing zero-shot evaluation with Classification and NLI datasets?

I am seeing in the code, the following prompts are used (using imdb-zero-shot.json as an example):

• Positive (label=0)
  The movie review in positive sentiment is: "<X>"

• Negative (label=1)
  The movie review in negative sentiment is: "<X>"
  where <X> is replaced with the movie review, before feeding it to GPT-2.

The words "positive" and "negative" are not replaced. This is seems like label leakage. I don't see any code where this is removed in the zero-shot baseline

So it seems to me like the Prompting baseline in ZeroGen is incorrect and overestimates the true value.

Would love to hear your thoughts.

Hi Jiacheng, could you please provide these files? It is needed to run the repo.