Do we support bert finetune classifier?我们支持 bert 的finetune 标签分类模型吗 about keras-bert HOT 10 CLOSED

These may be helpful to you, depending on what you want:
finetune BERT with custom dataset #20
single sentence classifier #21

Do you want to finetune with your non-labeled dataset, or labeled dataset ?
If your dataset is labeled, is it pair-sentence, or single-sentence, or something else like part of speech tagging ?

Suppose your labeled dataset is pair sentences and label is binary, and what you want is to finetune some specific layers, then simply feed it into the model and make sure layer.trainable = True

from keras-bert.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

from keras-bert.

These may be helpful to you, depending on what you want:
finetune BERT with custom dataset #20
single sentence classifier #21

Do you want to finetune with your non-labeled dataset, or labeled dataset ?
If your dataset is labeled, is it pair-sentence, or single-sentence, or something else like part of speech tagging ?

Suppose your labeled dataset is pair sentences and label is binary, and what you want is to finetune some specific layers, then simply feed it into the model and make sure layer.trainable = True

AFAIK, finetune means you should load a trained model from a checkpoint and then train it again with your own dataset.

I've checked the lasted codes in master branch, and find that there are two params control the behavior of the core class method, one is trainable and the other is training. What's more, the method related to load trained model from checkpoint would set these two params coherent, the default values of which are False.

So here is my question, to finetune the model, does it mean that I could simply put these two params to True? What is the consequent influence and behavior of this change? Say how would the new model be saved or etc?

@CyberZHG would you please kindly make some response to above? Since we need assurance here.

Thanks in advance.

from keras-bert.

@alexwwang

By setting trainable to True, you may get better results, with the cost of extra memories and time since you need to compute and remember the intermediate results and gradients of the whole model. While the model with setting trainable to False works just like a fixed dynamic embedding.

from keras-bert.

@alexwwang

The saving is the same as how you save a Keras model. You don't need to care about the checkpoints because they are used only for initializing the weights.

from keras-bert.

@CyberZHG Thanks for reply immediately. That's great.

One more thing I want to reassure is, could I take this as a procedure of finetune when I set trainable to True? Am I right?

from keras-bert.

@alexwwang

And don't forget there is a classification example: https://colab.research.google.com/github/CyberZHG/keras-bert/blob/master/demo/tune/keras_bert_classification_tpu.ipynb

from keras-bert.

@CyberZHG I will, great appreciated!

from keras-bert.

@CyberZHG Thanks for reply immediately. That's great.

One more thing I want to reassure is, could I take this as a procedure of finetune when I set trainable to True? Am I right?

Actually, you can train the model with trainable=False at the beginning, then set trainable=True with a small lr for further tuning.

from keras-bert.

@CyberZHG Thanks for reply immediately. That's great.
One more thing I want to reassure is, could I take this as a procedure of finetune when I set trainable to True? Am I right?

Actually, you can train the model with trainable=False at the beginning, then set trainable=True with a small lr for further tuning.

So if I have to train a model from sketch, I should set trainable=False;
while I have a trained model in hand, I could set trainable=True to finetune it for specific tasks.

from keras-bert.