Hi, thank you for the article!

Here at dataset.py#L200, you are adding extra CLS token to the sentence, but no extra True token to the inverse_token_mask.

This creates an alignment issue which can be seen in your example:

masked_sentence    [[CLS], one, of, the, other, [MASK], has, ment...
masked_indices     [0, 5, 6, 7, 8, 2, 10, 11, 4825, 13, 2, 15, 16...
sentence           [[CLS], one, of, the, other, reviewers, has, m...
indices            [0, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16,...
token_mask         [True, True, True, True, False, True, True, Fa...
is_next                                                            1

So [MASK] token is actually at the index of 5 but in the token_mask row it is marked at the index of 4 because of that. Isn't this wrong? I've checked other parts of the code, but couldn't find any code which handles this alignment issue, which becomes even more a problem when two sentences are concatenated i think.

Thanks!

Dear Ivan,

All of the work that you did is great. But while using your code of IMDBDataset i found some strange things! Array of indicies and array of inverse token mask values do not match each other from the beggining of the sequence because of CLS token (see the reference below). And it was also strange for me, while i found out CLS token in the second sentence. So, i may be wrong, but original BERT uses only one CLS token in the begining. And the last one, calculating length of vocab each time could be very expensive. So, i hope, that my notes will help you to make your code better and more clear. I would be proud if you gave me a posibility to take part in this and help you to fix it.

I am looking forward to your reply,
Nesemenpolkov.

Reference: