Training loss decreases, but model doesn't learn about gector HOT 1 CLOSED

Hi, @sabetAI.
Sorry for the slow reply.

We had not faced such a problem in our experiments. (after some number of updates model start to produce other tags as well)
I think that the following could be helpful.

1. Exclude true negatives from the data (tn_prob=0) during the pretraining stage.
2. Use bigger batch_size (at least 128; better 256).
3. Use more data if possible.
4. Freeze encoder weights during the first couple of epochs (cold_step_count in [2,4])

Additionally, you could modify the mask in order to make weights for KEEP operation lower.
Something like this:

keep_bias = -0.5
weights = (labels == self.keep_index).long() * (keep_bias) + mask
loss_labels = sequence_cross_entropy_with_logits(logits_labels, labels, weights, label_smoothing=self.label_smoothing)

I hope that this will be useful to you.

from gector.