tokens that exceeds max_len raise error about bert_score HOT 4 CLOSED

I see. Feel free to create a pull request.

from bert_score.

Hi Jennifer,

To our best knowledge, increasing the max_len can't be solved the problem without finetuning the BERT model. The reason is that BERT was trained with at most 512 tokens during the pre-training stage. We got the question before #17. We will update the README to remind the users. Please correct me if I'm wrong.

Best,
Felix

from bert_score.

Hi @felixgwu,

Thanks for the comments.

By the way, the point was not about increasing max_len but cutting off at the max_len.

IMHO, the following code

    arr = [tokenizer.encode(a, add_special_tokens=True, max_length=tokenizer.max_len) for a in arr]

from bert_score.

Hi @inmoonlight,
Would you like to make a PR? Or I can edit it.

from bert_score.