shushanxingzhe / transformers_ner Goto Github PK
View Code? Open in Web Editor NEWAdd CRF or LSTM+CRF for huggingface transformers bert to perform better on NER task. It is very simple to use and very convenient to customize
License: MIT License
Add CRF or LSTM+CRF for huggingface transformers bert to perform better on NER task. It is very simple to use and very convenient to customize
License: MIT License
Thanks for providing this code. I'd love to use it, but am getting the following error when running the trainer.
(py38_test) [richier@reslnapollo02 transformers_ner]$ python bert_crf_trainer.py
Downloading builder script: 9.52kB [00:00, 8.24MB/s]
Downloading metadata: 3.79kB [00:00, 3.99MB/s]
Reusing dataset conll2003 (/home/richier/.cache/huggingface/datasets/conll2003/conll2003/1.0.0/63f4ebd1bcb7148b1644497336fd74643d4ce70123334431a3c053b7ee4e96ee)
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 495.17it/s]
Dataset({
features: ['id', 'tokens', 'pos_tags', 'chunk_tags', 'ner_tags'],
num_rows: 14042
}) Dataset({
features: ['id', 'tokens', 'pos_tags', 'chunk_tags', 'ner_tags'],
num_rows: 3454
})
Some weights of the model checkpoint at bert-base-cased were not used when initializing BertCRF: ['cls.predictions.bias', 'cls.seq_relationship.weight', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.seq_relationship.bias', 'cls.predictions.decoder.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.LayerNorm.weight']
- This IS expected if you are initializing BertCRF from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertCRF from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of BertCRF were not initialized from the model checkpoint at bert-base-cased and are newly initialized: ['classifier.weight', 'crf.transitions', 'crf.end_transitions', 'classifier.bias', 'crf.start_transitions']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
0%| | 0/1 [00:04<?, ?ba/s]
Traceback (most recent call last):
File "bert_crf_trainer.py", line 59, in <module>
train_dataset = train_dataset.map(tokenize, batched=True, batch_size=len(train_dataset))
File "/home/richier/anaconda3/envs/py38_test/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1955, in map
return self._map_single(
File "/home/richier/anaconda3/envs/py38_test/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 520, in wrapper
out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
File "/home/richier/anaconda3/envs/py38_test/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 487, in wrapper
out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
File "/home/richier/anaconda3/envs/py38_test/lib/python3.8/site-packages/datasets/fingerprint.py", line 458, in wrapper
out = func(self, *args, **kwargs)
File "/home/richier/anaconda3/envs/py38_test/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 2339, in _map_single
batch = apply_function_on_filtered_inputs(
File "/home/richier/anaconda3/envs/py38_test/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 2220, in apply_function_on_filtered_inputs
processed_inputs = function(*fn_args, *additional_args, **fn_kwargs)
File "/home/richier/anaconda3/envs/py38_test/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1915, in decorated
result = f(decorated_item, *args, **kwargs)
File "bert_crf_trainer.py", line 24, in tokenize
tokenids = tokenizer(tokens, add_special_tokens=False)
File "/home/richier/anaconda3/envs/py38_test/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2288, in __call__
return self.batch_encode_plus(
File "/home/richier/anaconda3/envs/py38_test/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2473, in batch_encode_plus
return self._batch_encode_plus(
File "/home/richier/anaconda3/envs/py38_test/lib/python3.8/site-packages/transformers/tokenization_utils_fast.py", line 418, in _batch_encode_plus
for key in tokens_and_encodings[0][0].keys():
IndexError: list index out of range
Any idea what's going on?
Unrelatedly, the datasets package must be updated relative to what is in the requirements.txt. See this issue: huggingface/datasets#3582.
In this implementation, it seems that the active loss supported by the original bert model is lost when adding crf layer
How can i train on my own dataset? Please provide dataset structure, parameters and how to train the model, Thks!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.