Warning : Empty candidate sentence; Setting recall to be 0. about bert_score HOT 4 CLOSED

Hi @Colanim,

I'm sorry there is a typo here. The warning should be:

Warning : Empty reference sentence; Setting recall to be 0.

The reason for getting this warning is that some of the reference sentences (ground truth translations) are empty sentences.
Can you provide the scripts of how you get this message?
It might be either you have a "empty" reference sentence or something wrong with the API.

Best,
Felix

from bert_score.

hi @Colanim thank you for providing a detailed notebook to replicate the problem.

I took a look and the issue is that the nlp package expects a list of list of strings for the references.

if you modify the code to the following, it will work correctly.

rouge.add(lp.strip(), [lg.strip()])

We will communicate with the nlp package to see how best to improve on this input format issue.

from bert_score.

Thanks for your answer @felixgwu

Here is a Notebook reproducing the issue (using nlp package). None of the candidate / reference are empty, but I'm still having the warning : https://colab.research.google.com/drive/1uiS89fnHMG7HV_cYxp3r-_LqJQvNNKs9?usp=sharing

from bert_score.

Hi,

Could someone help me understand why is the BERTScore recall is non zero here. Isn't zero the right score for precision as well as recall with empty candidate sentence?

score([''], ['hello how are you'], lang='en')

This is what the model returns now:

Warning: Empty candidate sentence detected; setting precision to be 0.
(tensor([0.]), tensor([0.7738]), tensor([0.]))

best,
Anupama

from bert_score.