shannonai / chinesebert Goto Github PK
View Code? Open in Web Editor NEWCode for ACL 2021 paper "ChineseBERT: Chinese Pretraining Enhanced by Glyph and Pinyin Information"
License: MIT License
Code for ACL 2021 paper "ChineseBERT: Chinese Pretraining Enhanced by Glyph and Pinyin Information"
License: MIT License
I have got dev and test set by https://github.com/facebookresearch/XNLI . But I found that
.
P.S. I found the path with xnli_xxx.tsv
in your code
ChineseBert/tasks/XNLI/XNLI_trainer.py
Line 134 in f6b4cd9
xnli.xxx.tsv
论文中提到你们从CommonCrawl中筛选了约10%的数据,请问可否提供这份数据,或相应的清洗脚本呢~
我想请问一下,在运行tasks中的LCQMC示例的时候,如果没有GPU该如何调整
python LCQMC_trainer.py --bert_path ../ChineseBERT-base/ --data_dir E:/PycharmProjects/ChineseBert/lcqmc/ --save_path ../../cus_output/ --max_epoch=7 --lr=2e-5 --batch_size=16 --gpus=0最后一个参数可以不写吗?需要如何调整参数和修改代码
Hello,I have a question about textClassifier task,it is no error in thucnews data,but there are many errors in my own data (all codes are the same to your codes), the answer of this question in internet is that the lable_map should begin 0,eg : {"体育": 0, "娱乐": 1, "家居": 2}, the same error is still existence after I follow this answer, can you solve my question?t hank you!
请问下在NER任务中,pinyin_map.json, id2pinyin.json, pinyin2tensor.json这些文件是怎么生成的?
您好!请问您有模型预训练的代码吗?尝试使用run_mlm.py[https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling/run_mlm.py]进行进一步预训练,但代码中调用的tokenizer和您的模型中的tokenizer(BertMaskDataset)不同,替换后遇到了许多问题,希望您可以提供帮助~谢谢!
请问你们是在什么设备上训练的,进行了多长时间
下接crf为啥loss一直降不下去
您好,vocab.txt文件中344,21129行都是空行,有什么区别吗?
Some weights of the model checkpoint at E:\CODE\pythonProject\ChineseBert-main\CHINESEBERT_PATH were not used when initializing GlyceBertForSequenceClassification: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.decoder.weight', 'cls.predictions.decoder.bias']
请问这是怎么回事呢? 没有查到解决办法
AttributeError: 'function' object has no attribute 'endswith',请问这个错误要怎么解决呀
Hello Authors
What is the version of the transformers are you using? I was not able to execute your code. Moreover, if there are anymore dependencies please let us know.
文中提到因为预训练没有使用NSP任务而忽略了segment embedding,为什么源码fusion_embedding.py
里还是出现了token_type_embeddings
,虽然是用0初始化的
ChineseBert/models/fusion_embedding.py
Line 34 in 53d9c25
ChineseBert/models/fusion_embedding.py
Lines 60 to 61 in 53d9c25
ChineseBert/models/fusion_embedding.py
Line 75 in 53d9c25
ChineseBert/models/fusion_embedding.py
Line 77 in 53d9c25
您好,许多论文在MSRA的数据集上的评估效果都是在95-96左右,而在您的论文中即使是BERT也达到了99,想请问一下您是否对该数据集做了预处理/或者说您论文中所使用的数据集和公开的有一些不同呢?
如果是对数据集做了处理能方便公开一下该数据集么
写了--gpus=0,但是识别不了,出现报错
比如,我设置save topk=1 ,那么保存了一个checkpoint之后 尽管还在一直训练但不会再保存更新checkpoint
在拜读大神的文章和代码,对于tokenizer这里有些不解。
常规的tokenizer可以得到input_ids,
您的代码里的tokenizer可以同时得到input_ids和pinyin_ids,感觉很是神奇,
可否开源您训练tokenizer的代码呀
rt
Hello,could you tell me how to get the valid set and test set about XNLI? In the link, I only found train set.
想应用这个模型在领域内在微调一下,请问是否提供tf的支持呢
你好,我记得论文中有做全词掩码策略,为什么在代码中没看到??
谢谢!!
(yl) D:\ChineseBert-main\tasks\THUCNew>python THUCNews_trainer.py --bert_path ./111/ --data_dir ./cnews/ --save_path ./222/ --max_epoch=5 --lr=2e-5 --batch_size=8 --gpus=0
Some weights of the model checkpoint at ./111/ were not used when initializing GlyceBertForSequenceClassification: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.decoder.weight', 'cls.predictions.decoder.bias']
在运行ChnSetiCorp.py时,报错: Failed to interpret file 'ChineseBERT-base\config\._STFANGSO.TTF24.npy' as a pickle
请问这种问题该如何解决,在网上也没有找到答案.
parser.add_argument("--hidden_dropout_prob", default=0.1, type=float, help="dropout probability")
代码中设置了 hidden_dropout_prob,但是却通篇找不到使用了参数的地方
比如
self.bert = GlyceBertModel(config)
self.dropout = nn.Dropout(config.hidden_dropout_prob)
self.classifier = nn.Linear(config.hidden_size, config.num_labels)
modeling_glycebert.py中253行,直接使用了hidden_dropout_prob
请问模型如果在pycharm里面可以直接运行吗?
请问是否有开源预训练的代码呢
如上所述
请问能开源怎么做mlm的预训练吗
刚开始用pytorch的新手,就是readme里面的那个示例“我喜欢猫”,多次调用 chinese_bert.forward输出的结果由于模型里面有dropout不是应该不同吗,为什么我多次调用输出的结果总是一样的呢?
您好,首先恭喜这份工作被ACL2021录用,融入字形和拼音的预训练必然会对中文nlp任务带来一定的提升。
同样,我也希望能在除了论文中提及的其他任务中使用ChineseBert,请问有没有集成类似于BERT的API可以调用,
如:
tokenizer = Tokenizer.from_pretrain([ChineseBert])
config = Config.from_pretrain([ChineseBert])
model = Bert.from_pretrain([ChineseBert])
或者,有没有instruction说明一下调用方式
作者您好,感谢您的工作!请问我在依照quick tour中的实例调用接口时产生报错:ValueError: cannot reshape array of size 3555312 into shape (23236,24,24),问题来自np.load(np_file).astype(np.float32) for np_file in font_npy_files,可以帮忙看看是什么原因吗?(是字体npy文件下载的有问题吗,重新下载了几遍都未解决,各依赖包的版本也都是完全按照requirement.txt中配置的)再次感谢您的工作和帮助~
File "/root/ChineseBert/ChineseBert-main/models/fusion_embedding.py", line 72, in forward
inputs_embeds = self.map_fc(concat_embeddings)
File "/root/miniconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/root/miniconda3/lib/python3.8/site-packages/torch/nn/modules/linear.py", line 93, in forward
return F.linear(input, self.weight, self.bias)
File "/root/miniconda3/lib/python3.8/site-packages/torch/nn/functional.py", line 1692, in linear
output = input.matmul(weight.t())
RuntimeError: mat1 dim 1 must match mat2 dim 0
关于bert_dataset中
vocab_file = "/data/nfsdata2/sunzijun/glyce/glyce/bert_chinese_base_large_vocab/vocab.txt"
config_path = "/data/nfsdata2/sunzijun/glyce/glyce/config" 是改为CHINESEBERT_PATH中的路径吗
另外在bert_dataset中
tokenizer = BertMaskDataset(vocab_file, config_path) 的BertMaskDataset报红,
我试着在文件头加入from bert_mask_dataset import BertMaskDataset
提示ModuleNotFoundError: No module named 'bert_mask_dataset'
感谢您的查阅。
请问在你们预训练过程中,使用100GB的数据,是通过什么方式载入到dataset的。
据我所知,一般数据集不大的情况下是直接在dataset的__init__方法载入数据直接读进内存,但是数据量大的情况下采用什么方式?
我看到torch有提供一个iterabledataset,但是如果使用这个dataset的话,是不是就没有办法进行shuffle了呢?
Hello, I have met a version conflict and unable to manage it. (I am not quite familiar with the pytorch-lightning)
This expression here seems to have been deprecated in the newer version, this call back mechanism cannot be found in the User guide for pytorch lightning and the errrorpage also says that
so i changed it into the below way
and the afore-mentioned problem is gone, instead this thing appear
It seems tobe some what run out of num, how ever the dataset i custommed before was verified using the supplied method
There is also another possible thing that's probably going on, which is the relative path in use. But that is not likely to be the case
你好,我想问一下为什么字音嵌入的长度为8,而据我所知最长的拼音序列应该为6比如zhuang,加上音调长度也只为7,这样的话不就会一直空一个位置吗?
hi,
I meet this error. OSError: Failed to interpret file 'chineseBert20210929/datasets/ChineseBERT-base/config/._STXINGKA.TTF24.npy' as a pickle. Do you know how to solve it. thanks.
Validation sanity check: 0it [00:00, ?it/s]thread '<unnamed>' panicked at 'no entry found for key', D:\a\tokenizers\tokenizers\tokenizers\src\models\mod.rs:36:66
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Traceback (most recent call last):
File "ChnSetiCorp_trainer.py", line 227, in <module>
main()
File "ChnSetiCorp_trainer.py", line 216, in main
trainer.fit(model)
File "C:\Users\jingtanwang\Anaconda3\envs\chinesebert\lib\site-packages\pytorch_lightning\trainer\states.py", line 48, in wrapped_fn
result = fn(self, *args, **kwargs)
File "C:\Users\jingtanwang\Anaconda3\envs\chinesebert\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1084, in fit
results = self.accelerator_backend.train(model)
File "C:\Users\jingtanwang\Anaconda3\envs\chinesebert\lib\site-packages\pytorch_lightning\accelerators\cpu_backend.py", line 39, in train
results = self.trainer.run_pretrain_routine(model)
File "C:\Users\jingtanwang\Anaconda3\envs\chinesebert\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1224, in run_pretrain_routine
self._run_sanity_check(ref_model, model)
File "C:\Users\jingtanwang\Anaconda3\envs\chinesebert\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1257, in _run_sanity_check
eval_results = self._evaluate(model, self.val_dataloaders, max_batches, False)
File "C:\Users\jingtanwang\Anaconda3\envs\chinesebert\lib\site-packages\pytorch_lightning\trainer\evaluation_loop.py", line 305, in _evaluate
for batch_idx, batch in enumerate(dataloader):
File "C:\Users\jingtanwang\Anaconda3\envs\chinesebert\lib\site-packages\torch\utils\data\dataloader.py", line 279, in __iter__
return _MultiProcessingDataLoaderIter(self)
File "C:\Users\jingtanwang\Anaconda3\envs\chinesebert\lib\site-packages\torch\utils\data\dataloader.py", line 719, in __init__
w.start()
File "C:\Users\jingtanwang\Anaconda3\envs\chinesebert\lib\multiprocessing\process.py", line 105, in start
self._popen = self._Popen(self)
File "C:\Users\jingtanwang\Anaconda3\envs\chinesebert\lib\multiprocessing\context.py", line 223, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "C:\Users\jingtanwang\Anaconda3\envs\chinesebert\lib\multiprocessing\context.py", line 322, in _Popen
return Popen(process_obj)
File "C:\Users\jingtanwang\Anaconda3\envs\chinesebert\lib\multiprocessing\popen_spawn_win32.py", line 65, in __init__
reduction.dump(process_obj, to_child)
File "C:\Users\jingtanwang\Anaconda3\envs\chinesebert\lib\multiprocessing\reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
pyo3_runtime.PanicException: no entry found for key
Based on huggingface/tokenizers#260: it suggests that vocab.json miss one entry, but I try with tokenizer.encode(sentence) with all lines in ChnSentiCorp, it works
预训练的时候MASK,对应的拼音是全为0还是?
large模型训练ChnSetiCorp,batch size设置为1,两个GPU ,每个11G显存,仍然报显存不够
when i want to download the model on hugging face, I got a 404 on the page, and that's why?Thanks
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.