morizeyao / gpt2-chinese Goto Github PK

我用sudo pip3 install -r requirements.txt 安装package.
安装显示正常...
但import tokenization_bert 会找不到package....
请问这会是哪出错了？

(pytorch_p36) ubuntu@ip-172-31-38-29:$ sudo pip3 install -r requirements.txt
Requirement already satisfied: pytorch-transformers in /usr/local/lib/python3.5/dist-packages (from -r requirements.txt (line 1)) (1.0.0)
Requirement already satisfied: pytorch-pretrained-bert in /usr/local/lib/python3.5/dist-packages (from -r requirements.txt (line 2)) (0.6.2)
Requirement already satisfied: torch in /usr/local/lib/python3.5/dist-packages (from -r requirements.txt (line 3)) (1.2.0)
Requirement already satisfied: numpy in /usr/local/lib/python3.5/dist-packages (from -r requirements.txt (line 4)) (1.17.0)
Requirement already satisfied: tqdm in /usr/local/lib/python3.5/dist-packages (from -r requirements.txt (line 5)) (4.33.0)
Requirement already satisfied: sklearn in /usr/local/lib/python3.5/dist-packages (from -r requirements.txt (line 6)) (0.0)
Requirement already satisfied: keras in /usr/local/lib/python3.5/dist-packages (from -r requirements.txt (line 7)) (2.2.4)
Requirement already satisfied: tb-nightly in /usr/local/lib/python3.5/dist-packages (from -r requirements.txt (line 8)) (1.15.0a20190813)
Requirement already satisfied: boto3 in /usr/local/lib/python3.5/dist-packages (from pytorch-transformers->-r requirements.txt (line 1)) (1.9.72)
Requirement already satisfied: sentencepiece in /usr/local/lib/python3.5/dist-packages (from pytorch-transformers->-r requirements.txt (line 1)) (0.1.82)
Requirement already satisfied: regex in /usr/local/lib/python3.5/dist-packages (from pytorch-transformers->-r requirements.txt (line 1)) (2019.6.8)
Requirement already satisfied: requests in /usr/local/lib/python3.5/dist-packages (from pytorch-transformers->-r requirements.txt (line 1)) (2.22.0)
Requirement already satisfied: scikit-learn in /usr/local/lib/python3.5/dist-packages (from sklearn->-r requirements.txt (line 6)) (0.20.2)
Requirement already satisfied: scipy>=0.14 in /usr/lib/python3/dist-packages (from keras->-r requirements.txt (line 7)) (0.17.0)
Requirement already satisfied: pyyaml in /usr/lib/python3/dist-packages (from keras->-r requirements.txt (line 7)) (3.11)
Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.5/dist-packages (from keras->-r requirements.txt (line 7)) (1.12.0)
Requirement already satisfied: keras-applications>=1.0.6 in /usr/local/lib/python3.5/dist-packages (from keras->-r requirements.txt (line 7)) (1.0.8)
Requirement already satisfied: keras-preprocessing>=1.0.5 in /usr/local/lib/python3.5/dist-packages (from keras->-r requirements.txt (line 7)) (1.1.0)
Requirement already satisfied: h5py in /usr/lib/python3/dist-packages (from keras->-r requirements.txt (line 7)) (2.6.0)
Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.5/dist-packages (from tb-nightly->-r requirements.txt (line 8)) (0.15.5)
Requirement already satisfied: absl-py>=0.4 in /usr/local/lib/python3.5/dist-packages (from tb-nightly->-r requirements.txt (line 8)) (0.7.1)
Requirement already satisfied: protobuf>=3.6.0 in /usr/local/lib/python3.5/dist-packages (from tb-nightly->-r requirements.txt (line 8)) (3.9.0)
Requirement already satisfied: wheel>=0.26; python_version >= "3" in /usr/local/lib/python3.5/dist-packages (from tb-nightly->-r requirements.txt (line 8)) (0.33.4)
Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.5/dist-packages (from tb-nightly->-r requirements.txt (line 8)) (3.1.1)
Requirement already satisfied: grpcio>=1.6.3 in /usr/local/lib/python3.5/dist-packages (from tb-nightly->-r requirements.txt (line 8)) (1.22.0)
Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.5/dist-packages (from tb-nightly->-r requirements.txt (line 8)) (41.0.1)
Requirement already satisfied: botocore<1.13.0,>=1.12.72 in /usr/local/lib/python3.5/dist-packages (from boto3->pytorch-transformers->-r requirements.txt (line 1)) (1.12.206)
Requirement already satisfied: s3transfer<0.2.0,>=0.1.10 in /usr/local/lib/python3.5/dist-packages (from boto3->pytorch-transformers->-r requirements.txt (line 1)) (0.1.13)
Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python3.5/dist-packages (from boto3->pytorch-transformers->-r requirements.txt (line 1)) (0.9.4)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.5/dist-packages (from requests->pytorch-transformers->-r requirements.txt (line 1)) (2019.6.16)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.5/dist-packages (from requests->pytorch-transformers->-r requirements.txt (line 1)) (1.25.3)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.5/dist-packages (from requests->pytorch-transformers->-r requirements.txt (line 1)) (3.0.4)
Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python3.5/dist-packages (from requests->pytorch-transformers->-r requirements.txt (line 1)) (2.8)
Requirement already satisfied: docutils<0.15,>=0.10 in /usr/lib/python3/dist-packages (from botocore<1.13.0,>=1.12.72->boto3->pytorch-transformers->-r requirements.txt (line 1)) (0.12)
Requirement already satisfied: python-dateutil<3.0.0,>=2.1; python_version >= "2.7" in /usr/local/lib/python3.5/dist-packages (from botocore<1.13.0,>=1.12.72->boto3->pytorch-transformers->-r requirements.txt (line 1)) (2.8.0)
(pytorch_p36) ubuntu@ip-172-31-38-29:$ python3
Python 3.6.5 |Anaconda, Inc.| (default, Apr 29 2018, 16:14:56)
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

import pytorch_transformers
import torch
import tokenization_bert
Traceback (most recent call last):
File "", line 1, in
ModuleNotFoundError: No module named 'tokenization_bert'

词表打乱顺序或截断部分会影响训练么？

谢谢

Token indices sequence length is longer than the specified maximum sequence length for this model (1067 > 1024). Running this sequence through the model will result in indexing errors。
采用默认的配置，其最大长度为1024，而在train.py中，只对最小长度作了限制，但是没有对最大长度作限制，是否也应该根据配置文件的长度限制进行截断。

sublines = [full_tokenizer.tokenize(line) for line in sublines if
                    len(line) > min_length]  # 只考虑长度超过min_length的句子
# 在此转换时，会出现警告
sublines = [full_tokenizer.convert_tokens_to_ids(line) for line in sublines]

我有看到您新一版的训练成果有个体育新闻部分，想请教一下这训练的语料准备是多少的体育新闻数量？ 每则新闻大约有多少字？

感谢～

我按照作者写的instruction训练斗破苍穹（是txt文件），改了pre_process_data.py，具体改写如下：
def is_default_file_type():
return False

def load():
with open("./data/train.txt", 'r', encoding='utf-8') as f:
print('reading lines')
lines = f.readlines();
lines = [line.replace('\n', ' [SEP] ') for line in lines]
return lines

用make_vocab.py成功生成了一个50000的词表，
然后运行了train_single.py，运行命令为：

python ./train_single.py
--device=1
--model_config=config/model_config_small.json
--tokenizer_path=cache/vocab_user.txt
--raw_data_path=data/train.txt
--raw
--epochs=5
--batch_size=1
--lr=1.5e-4
--stride=1023
--output_dir=model/
--pretrained_model=''
--segment

模型配置文件model_config_small.json里面的配置为：
{
"initializer_range": 0.02,
"layer_norm_epsilon": 1e-05,
"n_ctx": 1024,
"n_embd": 768,
"n_head": 12,
"n_layer": 10,
"n_positions": 1024,
"vocab_size": 50000
}
这里vocab_size不太清楚应该是50000还是50005？因为加了特殊字符[cls][sep]等

然后成功开始运行，数据也成功处理了，但是运行了一会儿就报错了，具体错误信息为：

Traceback (most recent call last):
File "./train_single.py", line 235, in
main()
File "./train_single.py", line 177, in main
outputs = model.forward(input_ids=batch_inputs, labels=batch_labels)
File "/opt/conda/lib/python3.6/site-packages/pytorch_transformers/modeling_gpt2.py", line 606, in forward
past=past, head_mask=head_mask)
File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/opt/conda/lib/python3.6/site-packages/pytorch_transformers/modeling_gpt2.py", line 523, in forward
outputs = block(hidden_states, layer_past, head_mask[i])
File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/opt/conda/lib/python3.6/site-packages/pytorch_transformers/modeling_gpt2.py", line 345, in forward
m = self.mlp(self.ln_2(x))
File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/opt/conda/lib/python3.6/site-packages/pytorch_transformers/modeling_gpt2.py", line 326, in forward
h = self.act(self.c_fc(x))
File "/opt/conda/lib/python3.6/site-packages/pytorch_transformers/modeling_gpt2.py", line 102, in gelu
return 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3))))
RuntimeError: CUDA error: device-side assert triggered

不太明白问题出在哪里了？请给点建议，另外我的训练语料前需要加[cls] token吗？

谢谢

这个参数限定样本的长度。

但是，对于不同的篇章，长度是不一样的。而且多篇篇章通过[CLS][MASK]分隔后混在一起。

直接取长度为n_ctx的字符作为一个样本，感觉这样不太合理。

请问有什么考虑吗？

从代码的逻辑上看，重复输出样本，实在不知道有什么作用，还是作者想批量生成，最后没实现？如果作者有时间的话，也请重新审视一下代码吧，细节上有太多问题了，虽然我想改，但是有些东西，不是我实现的，避免误改，还是请作者自己修改吧。

你玩晒啦

@Morizeyao 感谢分享代码。请问下文本摘要生成，即seq2seq任务用gpt2该怎么修改？ 是直接标题和内容拼接输入吗？

now time: 9:22. Step 4 of piece 20 of epoch 1, loss 7.352459907531738
Traceback (most recent call last):
File "c:\Users\LeeDaga.vscode\extensions\ms-python.python-2019.8.30787\pythonFiles\ptvsd_launcher.py", line 43, in
main(ptvsdArgs)
File "c:\Users\LeeDaga.vscode\extensions\ms-python.python-2019.8.30787\pythonFiles\lib\python\ptvsd_main_.py", line 432, in main
run()
File "c:\Users\LeeDaga.vscode\extensions\ms-python.python-2019.8.30787\pythonFiles\lib\python\ptvsd_main_.py", line 316, in run_file
runpy.run_path(target, run_name='main')
File "C:\Users\LeeDaga\Anaconda3\lib\runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "C:\Users\LeeDaga\Anaconda3\lib\runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "C:\Users\LeeDaga\Anaconda3\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "d:\GPT2CN\train_single.py", line 224, in
main()
File "d:\GPT2CN\train_single.py", line 166, in main
outputs = model.forward(input_ids=batch_inputs, labels=batch_labels)
File "C:\Users\LeeDaga\Anaconda3\lib\site-packages\pytorch_transformers\modeling_gpt2.py", line 593, in forward
past=past, head_mask=head_mask)
File "C:\Users\LeeDaga\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 547, in call
result = self.forward(*input, **kwargs)
File "C:\Users\LeeDaga\Anaconda3\lib\site-packages\pytorch_transformers\modeling_gpt2.py", line 495, in forward
inputs_embeds = self.wte(input_ids)
File "C:\Users\LeeDaga\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 547, in call
result = self.forward(*input, **kwargs)
File "C:\Users\LeeDaga\Anaconda3\lib\site-packages\torch\nn\modules\sparse.py", line 114, in forward
self.norm_type, self.scale_grad_by_freq, self.sparse)
File "C:\Users\LeeDaga\Anaconda3\lib\site-packages\torch\nn\functional.py", line 1467, in embedding
return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: index out of range: Tried to access index 13411 out of table with 13316 rows. at C:\w\1\s\windows\pytorch\aten\src\TH/generic/THTensorEvenMoreMath.cpp:237

换了训练数据也不行

运行train.py 的时候报错 module 'tensorflow.io' has no attribute 'gfile'

想请问一下如果在有GPU的机器上训练好模型后，在产生内文时是否可以移至CPU上来运行？我是知道诸如LSTM跟RNN是都可以把tensor跟model移至CPU来执行，由于GPT-2有预训练过不知道是否也可以训练好移至CPU上运行？

我照配置使用small.json去配置，在4GPU每个GPU有8GB的环境去跑还是会出现既济体不够的问题

File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/pytorch_transformers-1.0.0-py3.6.egg/pytorch_transformers/modeling_gpt2.py", line 100, in gelu
return 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3))))
RuntimeError: CUDA out of memory. Tried to allocate 24.00 MiB (GPU 0; 7.44 GiB total capacity; 6.88 GiB already allocated; 21.50 MiB free; 128.30 MiB cached)

我执行NVIDIA-smi
(pytorch_p36) ubuntu@ip-172-31-38-29:~/GPT2$ nvidia-smi
Wed Aug 14 06:03:59 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.67 Driver Version: 418.67 CUDA Version: 10.1 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla M60 On | 00000000:00:1B.0 Off | 0 |
| N/A 34C P8 23W / 150W | 0MiB / 7618MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla M60 On | 00000000:00:1C.0 Off | 0 |
| N/A 39C P8 22W / 150W | 0MiB / 7618MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 2 Tesla M60 On | 00000000:00:1D.0 Off | 0 |
| N/A 36C P8 22W / 150W | 0MiB / 7618MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 3 Tesla M60 On | 00000000:00:1E.0 Off | 0 |
| N/A 39C P8 22W / 150W | 0MiB / 7618MiB | 0% Default |
+-------------------------------+----------------------+----------------------+

确定每台有8GB GRAM, 但是为何看起来只使用到8G做配置

执行命令如下...
python3 train.py --raw --device="0,1,2,3"

这会是哪方面问题？

我使用自己的代码，大致实现是和作者一样的，多gpu单精度训练和单gpu混合精度训练都没问题，但是如果多gpu混合精度训练就会报arguments are located on different GPUs的错误，请问作者使用多gpu混合精度训练出现过类似的bug吗？

generate.py 第80行应该放在79行前面哟。

Bug Report
在生成脚本 generate.py 中第 169 行，打开的文件指针 samples_file 在使用完后没有被关闭，可能会造成句柄泄露。

samples_file = open(args.save_samples_path + '/samples.txt', 'w', encoding='utf8')

更正建议
在退出循环前，显示关闭文件指针 samples_file.close()

我去下载了斗破苍穹的语料，但它是纯text档案而非json档案。
当我修改train.py中...
# doupo = json.load(f)
doupo = f.read()

却发生警示...
W0726 13:58:53.042458 4556223936 tokenization.py:126] Token indices sequence length is longer than the specified maximum sequence length for this BERT model (5340786 > 512). Running this sequence through BERT will result in indexing errors
请问这是何问题？

嗨，我使用 pip install -r requirements.txt 設定好環境，但是跑 train_single.py 時出現底下 import 錯誤，有人碰到一樣情況嗎？

Traceback (most recent call last):
  File "train_single.py", line 223, in <module>
    main()
  File "train_single.py", line 65, in main
    from tokenizations import tokenization_bert_without_wordpiece as tokenization_bert
  File "/content/GPT2-Chinese/tokenizations/tokenization_bert_without_wordpiece.py", line 25, in <module>
    from pytorch_transformers.tokenization_utils import PreTrainedTokenizer, clean_up_tokenization
ImportError: cannot import name 'clean_up_tokenization'

看官方實現有定義 clean_up_tokenization，不過一時想不到怎麼解決這個 import error.

作者在保存模型时没有保存optimizer的参数和scheduler的参数，是否会对中断后继续训练有影响？

如题！谢谢！

在说明文档中, 提到 “ 若使用BPE Tokenizer，需自己建立中文词表”
怎么理解?

谢谢

训练完的模型是否可以相互共享呢

大佬用的什么机器跑的

您好，loss在每读一个新的num_piece之后loss会增加再往下降低，这是正常的嘛，比如从2.多突然增到5.多，再继续往下降低，这是为什么呢？

作者能够提供一下在新闻预料上训练时最后的loss或者ppl？我试验过使用bert的tokenizer，也就是用字作为基本单位，loss下降的情况并没有subword好，文本生成的效果没有比较，不知道作者有没有进行过这方面的比较？

需要训练多少epoch是否有个经验值？是否和数据量有关，有没有什么参考意见？

你好，请问语料需要的token需要使用空格分隔吗？还有预训练完成之后怎么使用呢？

比如说这个标题：震惊，人工智能竟然让李白活过来了！或者是：起点作者的末日？人工智能竟然能编写小说！随着gpu性能的提升还有人工智能asic单元的出现，未来有可能会普及，这个东西对于世界的影响绝对比deepfake要大。（本人观点，不喜勿喷）

flake8 testing of https://github.com/Morizeyao/GPT2-Chinese on Python 3.7.1

$ flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics

./train_single.py:184:21: F821 undefined name 'running_loss'
                    running_loss += loss.item()
                    ^
2371     F821 undefined name 'running_loss'

你好！

当前的generator应该属于命题作文类，或者“续写”模式。

我的需求是这样的：给定一句话，生成另外一句话（限定最大长度）。

进而我想实现的需求是生成的这句话与给定的一句话，表达类似的意思。（非问答模式）。

感谢！

您好，想请教几个问题，1)训练数据的格式是["文章1","文章2","文章3"]这样吗，首尾必须带[]吗 ? 2）针对斗破苍穹的数据，是把每一章当作一篇文章吗 ，原数据是包含了全部章节，您是怎么做处理的 3)如果单篇文章很长，会做截断吗(比如bert会限制长度512)，非常感谢

使用 train_single.py 跑一整個文件時跳出這個 error。還沒找到方法解決，但看了幾篇網路上的文章貌似跟文本太長有關係，不知道大佬或是其他人有沒有遇過這個問題？

• Pytorch 版本：1.1.0
• pytorch-transformer: 1.1.0

Token indices sequence length is longer than the specified maximum sequence length for this model (12413 > 1024). Running this sequence through the model will result in indexing errors

...

RuntimeError: Creating MTGP constants failed. at /opt/conda/conda-bld/pytorch_1556653099582/work/aten/src/THC/THCTensorRandom.cu:33

完整 traceback:

Traceback (most recent call last):
  File "train_single.py", line 240, in <module>
    main()
  File "train_single.py", line 179, in main
    outputs = model.forward(input_ids=batch_inputs, labels=batch_labels)
  File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/pytorch_transformers/modeling_gpt2.py", line 593, in forward
    past=past, head_mask=head_mask)
  File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/pytorch_transformers/modeling_gpt2.py", line 503, in forward
    hidden_states = self.drop(hidden_states)
  File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/dropout.py", line 58, in forward
    return F.dropout(input, self.p, self.training, self.inplace)
  File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/functional.py", line 830, in dropout
    else _VF.dropout(input, p, training))
RuntimeError: Creating MTGP constants failed. at /opt/conda/conda-bld/pytorch_1556653099582/work/aten/src/THC/THCTensorRandom.cu:33

计划用多大规模的语料？使用什么GPU？

模型默认的warmup=2000，lr=1.5e-4，假设此时总的步数<2000，那么实际lr=[0,1.5e-4)之间，步数越小，lr越小，会出现难收敛的问题。
假设我一开始想让模型收敛得快一些，设lr=0.001,那么实际也得到2000步时，lr才到真正的0.001，之前都比这小，与lr参数的初衷相违背，所以warmup默认值，建议设为0。
最后，输出不一致问题。假设设置了gradient_accumulation=4，而log_step默认，那么每隔4次，输出一个loss，但是log_step默认为1，每次都输出loss=0，当然对结果没有影响，但是感觉...

您好，请问可以支持有监督的方式训练吗？

Great repo. However, the train_single script seems to be broken.

  File "train_single.py", line 223, in <module>
    main()
  File "train_single.py", line 74, in main
    full_tokenizer = tokenization_bert.BertTokenizer(vocab_file=args.tokenizer_path)
UnboundLocalError: local variable 'tokenization_bert' referenced before assignment

采用分片切割的方式，分得越多，丢失的数据越多
1 每个Piece所拥有的数据长度不一，在使用移动窗口分割数据时，会完全丢失数据，这部分数据（当然数据不多，小于seq_len）将永远不参与训练。分得越多，丢得越多。
2 在分batch时，不足以组成一个完整batch的数据将被丢失，当然这个可以通过多轮epoch来解决，但是通过shuffle，无法保证上一轮丢弃的数据一定会参与训练。此处丢失[0,seq_len*(batch_size-1)]，batch_size越大，丢失得越多。
3 stride的取值问题，会使得部分数据参与运算的次数高于其它数据，人为改变了数据原有的分布，请问怎么看待这个问题？
建议可通过基于步数分割成片，如每Piece包含多少步。而每步长=(seq_len-stride)+stride*batch_size，这样最多永久丢失一步数据[0,步长)，不想丢失，可人为补至完整一步。

由于我的硬体上无法训练BERT还无法能验证，由于我有一个项目要对短视频剧本生成AI写作的需求，之前我用LSTM 模型来实作，但是只能保有句子本生身通顺但是句子间或是多句文章时就看不出主题取向效果有点不能接受。看到大大的BERT模型燃起一丝希望，想请教您的过往经验如果用这BERT模型是否能解决LSTM的剧本生成的问题？

感谢赐教～

错误原因
train.py 脚本中第 57 行 parser 设置参数 gradient_accumulation 的类型为 str

parser.add_argument('--gradient_accumulation', default=1, type=str, required=False, help='梯度积累')

会导致第 139 进行除运算时抛出类型错误，不能对 str 做除法

total_steps = int(full_len / stride * epochs / batch_size / gradient_accumulation)

更正建议
在第 57 行设置参数 gradient_accumulation 的类型为 int

您好，想请问一下我如果想以词级别来训练语言模型，是需要用tokenization_bert_without_wordpiece.py这个数据预处理嘛，文本的格式是需要分好词的格式吗，比如“今天 天气 真好”

您好，我想请问一下我只想要从头训练一个语言模型来计算句子的ppl，应该如何使用您的代码呢，是不是只要使用train.py和预处理的代码呢，generate.py是不是用不上呢，谢谢您~

经由一晚的训练，但又些疑问...想提出来讨论一下...
由于想得到更好效果，我ePoch 做到75(15+60)，结果...
now time: 23:18. Step 4 of piece 99 of epoch 60, loss 0.003577812574803829
now time: 23:18. Step 5 of piece 99 of epoch 60, loss 0.0026147072203457355
now time: 23:18. Step 6 of piece 99 of epoch 60, loss 0.0035277323331683874
now time: 23:18. Step 7 of piece 99 of epoch 60, loss 0.003405306488275528
saving model for epoch 60
epoch 60 finished
time: 2019-08-14 23:18:30.795687
time for one epoch: 0:09:19.922468
training finished

Loss 降到0.003，这让我惊讶了，从来没看过这么低的Loss，这真的很不可思议！
但相对的问题来了，我在产出中有看到产出的文本会跟训练语料的文本完全相同(一字不差)，感觉好像是过度学习了...我想Loss 还是控制在0.9~0.1之间比较理想。
是不是在学习过程要加入判断Loss值低于多少就自动停止，这样会比较好...

输出结果感觉还算不错...文笔算是很自然，提供分享

• (环境：aws g3,16xlarge EC2 (M60 4 GPU , 32GiB GRAM), n_layers=7, 语料 16.7MB)

======

• [starttext]爱情游戏剧情讲述了钢琴父女明致怀萌的爱情、个有着努力的热情以及现实为人生的价值观众，获得一系列爱情的故事。80后录股媒体受到网友分享，是2014年主创陈拉昀出品牌总监于蓝氏集团化验师创业团门的哥哥大国度上海淮河畔，集入第一线公司青年度虽然没有放到的事业，但是蓝正是却不到位主人拒绝了解，而在蓝越的帮助理念出现，也因此开启明朗的误会而经营变成爱河。在一次偶然的编剧集电视剧之夏天上一改变了自命运环球顶樑，三人在创车祸中不知被记忆差网识分到创作，并被问流言败，以及行业服务所有的低调教同才力，陈昭和唐诗诗妍展开了一段截然不同的“2014年间段感情”，两人性格互相治癒的商业奋斗故事，尽管是共90后北京华侨大学录的一个宿舍小旅程和唐如、生等优秀青年，的人生活如何与愿违3个国偶像，并且共同创作何以此他们互相有观众的成功和关心吗?[endtext]

• [starttext]学习爱情主要讲述了两对方小曼，经过啼笑皆非的考验，终于选择了三个孩子，携手共同创业来四个孩子，在大城市里创业的成功商。两家内事业的加入了北京城市，经过了一次元城市融风雨故、差异后得到异的他们，最终收获了梦想的真正属于自己的爱情。赞助理想、电视剧、剧等主创业时代人物特点在北京举行开机仪式，该剧以当下海南三个新人青年轻人面人海南梅竹马的电视角，讲述了几个在北京、喜剧代人生活中增强非浪漫的年轻人，以独特的双时代年轻人从来到北京城市化**大城市走出发展以海南方的变迁在语种城市闯关于人生态的同时，以及他们渐渐的生活方式为自己方向上演了那么简单俗，是当代际拍摄的就如何在这个城市里都市里?那么平静的城市就是城市的风格特张嘉和支持工作打造，而这是一点就要打造出机场话剧组会。化身处处棋逢貌各种文化的人都非常独特的煽情，交织了相，滑稽等来自外衣的东北漂亮、内地，者和两位女孩子敢称是哑女孩子。交织里的人齐飞一开泰块玩笑，令人印象太趋的气质，让人眼看这个性格非常喜剧，知道的是一个“东北漂”人的外国小养家，让她耳熟练读剧的外形象显老大。之后齐飞、表示爱朗的齐飞、范儿、楚月子、白天杰。两代人的生活里友情似乎没有结合、精彩表态的开朗和丽丽丽。[endtext]

==========

非常感谢能分享这么好的项目。有几个问题想请教一下。
请问一开始模型参数是加载的预训练模型，还是只是简单的初始化？
如果是从预训练模型开始训练，是使用OpenAI开源的预训练模型吗？
如果是从头开始训练，只用几百M的语料就已经足够了吗，毕竟OpenAI用了40G的语料。从头开始训练在四卡2080Ti上大概需要多长时间呢？

您好，请问您在训练过程中有碰到训练到一定时候loss开始抖动嘛，训练了大概1000多步开始？

由于GPT-2有预训练，请问GPT-2有支援正体中文吗？如果提供正体中文的训练数据，是否可以产生正体中文的文章？

NVIDIA/apex#286 (comment)
Should not be hard I guess?
Just edit the train_with_ga.py file and PR xD.