Support for multiple references per candidate about bert_score HOT 5 CLOSED

This issue will be addressed in the next version of this repo, which is likely going to happen by the end of February.

from bert_score.

Hi @TheShadow29,
Thank you for pointing this out. Your usage is correct. The bug has been fixed in the newest commit.

from bert_score.

hi @feralvam, internally bertscore caches the computed representations of encountered sentences. This means that if you have duplicated sentences, our package will reuse the cached vectors to compute bertscore. For multi-reference setting, you can simply repeat the candidate sentences for a couple times. This should be efficient.

That being said, I look forward to having a convenient interface to handle multiple references. Feel free to open a PR if you want to contribute.

from bert_score.

@Tiiiger thanks for the awesome repository.

Could you clarify the usage of multiple references? From the demo notebook (cell 15 https://github.com/Tiiiger/bert_score/blob/master/example/Demo.ipynb), I believe the usage should be:

single_candidates = [cand_for_sent1, cand_for_sent2 ....]
multi_references = [[ref1_for_sent1, ref2_for_sent1 ...,], [ref1_for_sent2, ref2_for_sent2, ...], ...]

I tried a naive example with one reference passed as multi reference.

single_cands = ['a woman is seen sitting in a chair holding a accordion and speaking to the camera',
 'the woman then begins playing the accordion while looking back to the camera']
multi_refs = [['a woman is seen speaking to the camera while holding an accordion and and moving her hands around'],
 ['she demonstrates how to play the instrument while still speaking to the camera and moving all around']]
P_mul, R_mul, F_mul = score(single_cands, multi_refs, lang="en", rescale_with_baseline=True)

However, this gives a RunTime error:

lib/python3.7/site-packages/bert_score/scorer.py in score(self, cands, refs, verbose, batch_size, return_hash)
    191             max_preds = []
    192             for start, end in ref_group_boundaries:
--> 193                 max_preds.append(all_preds[start:end].max(dim=0)[0])
    194             all_preds = torch.stack(max_preds, dim=0)
    195 

RuntimeError: cannot perform reduction function max on tensor with no elements because the operation does not have an identity

It seems both start, end have the same value. Any guesses why this is happening?

Thank you for your patience.

from bert_score.

Thanks a lot @felixgwu for the prompt response.
It works correctly now.

from bert_score.