def create_sentinel_ids(self, mask_indices):
# From https://github.com/huggingface/transformers/blob/main/examples/flax/language-modeling/run_t5_mlm_flax.py
start_indices = mask_indices - np.roll(mask_indices, 1,
axis=-1) * mask_indices
start_indices[:, 0] = mask_indices[:, 0]
sentinel_ids = np.where(start_indices != 0,
np.cumsum(start_indices, axis=-1),
start_indices)
sentinel_ids = np.where(sentinel_ids != 0,
(len(self.tokenizer) - sentinel_ids), 0)
sentinel_ids -= mask_indices - start_indices
return sentinel_ids
In the code, you replace mask with sentinel_ids, the position of which is at last of tokenizer。But before doing this,you had add motion token to the last of tokenzie,Was this done on purpose?