Hello, is it convenient for you to provide the dataset link ? thank you!
and can you upload the code for preprocessing the dataset?

total_loss = torch.exp(-self.log_vars[0]) * mlm_loss + torch.clamp(self.log_vars[0], min=0) + \ torch.exp(-self.log_vars[1]) * tec_loss + torch.clamp(self.log_vars[1], min=0) + \ torch.exp(-self.log_vars[2]) * sec_loss + torch.clamp(self.log_vars[2], min=0)
Hello, I wonder why use torch.clamp here. torch.clamp can guarantee the value >= 0, but I think the value here (self.log_vars) are always positive. because we initialize self.log_vars to be zero and it will monotone increase during training, which found after I derive gradients manually.
Is there something wrong with me? :)

When I running your code with sh wordreplace.sh and using latest spacy . Errors occur like the following:

so i wonder which version of spacy are you using?

Hi! I found something a bit confusing that you might be able to clarify. For the contrastive loss implemented in "LecbertForPreTraining" class, you use a sigmoid as the activation function before applying the BCE loss but in the paper you mention a normal InfoNCE loss (with softmax activations) was used. Am I misunderstanding something perhaps?

I see that this code can generate negative cases in English, but I don't think it can be used to generate negative cases in Chinese, can you answer? Thank you very much!

For example, python, torch, transformers, spacey, lemminfect, etc