truthless11 / hrl-re Goto Github PK

Train the model with
python main.py --lr 2e-5 --datapath ../data/NYT10/

Dev and test set F1 are 0， training accuracy is 0 on each training epoh while the loss is continuing decline
After training 2 epochs, all sentences' top_actions are[4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ..., 0, 0]

Environment:
Python 3.5.2
Pytorch 1.0.1(Have change the code to Pytorch1.0 and no error came out yet )
@truthless11

First to thank your project, I do really think HRL-RE is a gooooood idea.
However, when i use colab to train my model, I find it hard to use 4 process, so i change the process into 2, then the problem shows as below:

Starting training service, overall process number: 2
training epoch 0
Process 1 start service.
Process 0 start service.
Process Process-2:
Process Process-2:
Process Process-3:
Traceback (most recent call last):
File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/content/drive/My Drive/workspace/HRL-RE-master/code/TrainProcess.py", line 55, in worker
acc, cnt, tot, loss = workProcess(model, datas, sample_round, mode)
File "/content/drive/My Drive/workspace/HRL-RE-master/code/TrainProcess.py", line 25, in workProcess
preoptions, preactions)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/content/drive/My Drive/workspace/HRL-RE-master/code/Model.py", line 80, in forward
prehid, prec = self.preLSTML(wvs[x], (prehid, prec))
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/rnn.py", line 723, in forward
self.check_forward_input(input)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/rnn.py", line 563, in check_forward_input
if input.size(1) != self.input_size:
RuntimeError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

pytorch version is 1.0.1, so i don't know what the problem can be. By the way, i can not find log "Process Process-2:" in anywhere. Was the problem due to the version or pytorch multi-process instead? Thanks for any reply on this issue

Hello！
Can you share NYT10 original data because your released data don't have entity type?

Thank you!

Running:

• Pytorch 0.3.1
• Python 3.5.2

The RL training doesn't work for me for the NYT10 dataset (havent checked other yet).
I first ran pretraining for 10 epochs with:

python main.py --epochPre 10 --numprocess 8 --datapath ../data/NYT10/ --pretrain True

which gets roughly 58 F1 on the test set, and then afterwards try the RL training with:

python main.py --epochRL 10 --numprocess 8 --start checkpoints/model_HRL_10 --datapath ../data/NYT10/

I stopped RL training after 3 epochs because not only was dev and test set F1 at 0, even training accuracy is 0.
Loss started at around 30, then after only 60 batches decreases to about -20, then slowly increases again and ends up hovering around -0.00005
Checking the optimize() method all reward arrays contain either straight 0's or negative numbers.

the code is here ：Panda0406/Reinforcement-Learning-Distant-Supervision-RE，I try to run both of urs，but did not success until now，so I wonder if u have done some test ，thx @truthless11

Is there a preprocess scripts that can convert the origin NYT data to your data format ?

As i am doing an compare research, however none of your paper use the same data preprocessing pipeline? I want to prepare the data as I guess, but it troublesome because what i am doing might get worse performance..

So, i wonder if there is any scripts that can convert the origin NYT data to your data format directly? Thank you very much if you can help!

Traceback (most recent call last):
File "/Users/dubhe/anaconda2/envs/hrl-re/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
self.run()
File "/Users/dubhe/anaconda2/envs/hrl-re/lib/python3.5/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/Users/dubhe/Project/HRL-RE-master/code/TrainProcess.py", line 55, in worker
acc, cnt, tot, loss = workProcess(model, datas, sample_round, mode)
File "/Users/dubhe/Project/HRL-RE-master/code/TrainProcess.py", line 25, in workProcess
preoptions, preactions)
File "/Users/dubhe/anaconda2/envs/hrl-re/lib/python3.5/site-packages/torch/nn/modules/module.py", line 541, in call
result = self.forward(*input, **kwargs)
File "/Users/dubhe/Project/HRL-RE-master/code/Model.py", line 80, in forward
prehid, prec = self.preLSTML(wvs[x], (prehid, prec))
File "/Users/dubhe/anaconda2/envs/hrl-re/lib/python3.5/site-packages/torch/nn/modules/module.py", line 541, in call
result = self.forward(*input, **kwargs)
File "/Users/dubhe/anaconda2/envs/hrl-re/lib/python3.5/site-packages/torch/nn/modules/rnn.py", line 936, in forward
self.check_forward_input(input)
File "/Users/dubhe/anaconda2/envs/hrl-re/lib/python3.5/site-packages/torch/nn/modules/rnn.py", line 759, in check_forward_input
if input.size(1) != self.input_size:
IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

the train.json dont exist in the files download from the link
`
(base) u@pc:~/HRL-RE/data/NYT10$ ls
filtered-freebase-simple-topic-dump-3cols.tsv kb_manual README.md
heldout_relations manual-05-06.tgz relations.tar.gz
heldout_relations.tgz nyt-2005-2006.backup

`

Nice work on dealing with overlapped RE problem, but i still have some confuse. In your calc_acc function, u only consider begin of source/target entity(4 and 5), but in your paper P5 Evaluation Metrics part, "A triplet is regarded as correct if the relation type and the two corresponding entities are all correct." I wonder whether there's some detail i missed in the paper.Could u explain that for me?Thank u.

I meet the question:
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1524586445097/work/aten/src/THC/THCTensorRandom.cu line=25 error=2 : out of memory Traceback (most recent call last): File "<string>", line 1, in <module> File "/home/zhaoy/anaconda3/envs/tensorflow/lib/python3.6/multiprocessing/spawn.py", line 105, in spawn_main exitcode = _main(fd) File "/home/zhaoy/anaconda3/envs/tensorflow/lib/python3.6/multiprocessing/spawn.py", line 115, in _main self = reduction.pickle.load(from_parent) File "/home/zhaoy/anaconda3/envs/tensorflow/lib/python3.6/site-packages/torch/multiprocessing/reductions.py", line 95, in rebuild_storage_cuda torch.cuda._lazy_init() File "/home/zhaoy/anaconda3/envs/tensorflow/lib/python3.6/site-packages/torch/cuda/__init__.py", line 161, in _lazy_init torch._C._cuda_init() RuntimeError: cuda runtime error (2) : out of memory at /opt/conda/conda-bld/pytorch_1524586445097/work/aten/src/THC/THCTensorRandom.cu:25
what is your model's hardware requirements?

I split the NYT11 dataset by myself (90% for training, 10% for validation), and I want to construct the dictionary for future use. Here is what I did:

• I don't want the word appearing in the entities to be replaced by the <unk> token, so the words in the entities go all into the dictionary.
• I filter out the words with low frequency (appearing at least 3 times in the whole training set).

After filtering, 40,666 words are selected out of 83,399 words, which means half of the words will be replaced by <unk> token. I feel this is not a very acceptable result while I don't know the traditional way of solving this problem of 'low frequency' in the Joint Extraction of Entities and Relations task. Am I doing it wrong?

Then I found that there are some (654) of the words in the dictionary are not in the 'vector.txt' file, which contains 144,885 words. I'm wondering how you construct the dictionary for the NYT11 corpus.

Any advice is appreciated. Thanks.

Hi, I notice that you filtered the dataset by "removing 1) the relations in ......; 2)the sentence that ...." as stated in the paper. And it says "Such a preprocess is also in line with the setting in the literature (for instance, Tagging)" . However, I read through the original paper (Zheng et al. 2017) and I can't find any similar description about the preprocessing. Did you confirm it with Zheng, or am I missing something? Could you please explain a little further? Thank you very much !!!

Hi there @truthless11 , can I confirm with you two things regarding the relation indicators:

• A relation can be tagged to only one token at each time in the sequence, i.e. the relation indicator has no "inside", only one "beginning"
• There is no ground-truth label for the position of the relation, so for your accuracy/F1/top reward calculations, as long as the relation is inside the list of top_actions, regardless of position, you consider it a hit for the relation/top action

Thank you.

hi,

   thanks for  the work that you guys  have done.  I thinks it is useful  for me. By the way, Did you guys  think about replacing LSTM with Bert ?  I did that. Just for you know,  I think it's better than lstm  . 

original :

epoch 0: dev F1: 0.5004633920296572, test F1: 0.4455146364494807
2 epoch 0: dev F1: 0.5252709640616087, test F1: 0.4531946508172363
3 epoch 1: dev F1: 0.6126023657870792, test F1: 0.5121126245543468
4 epoch 2: dev F1: 0.65594573897823, test F1: 0.5601257532093266
5 epoch 3: dev F1: 0.7098411297440423, test F1: 0.6034816247582204
6 epoch 4: dev F1: 0.7194697225030896, test F1: 0.6027715690657129
7 epoch 5: dev F1: 0.7338809497018117, test F1: 0.6199315191926474
8 epoch 6: dev F1: 0.7422517750478981, test F1: 0.6272242796495349
9 epoch 7: dev F1: 0.7402568145978825, test F1: 0.621670592498641
10 epoch 8: dev F1: 0.7411869701026328, test F1: 0.6330103990756377
11 epoch 9: dev F1: 0.7489399687569738, test F1: 0.6232657417289221
12 epoch 10: dev F1: 0.7425932184296495, test F1: 0.6252486887321396
13 epoch 11: dev F1: 0.7450980392156862, test F1: 0.6251227568967057
14 epoch 12: dev F1: 0.7373334851417511, test F1: 0.6234788178241376

replace lstm with Bert :

epoch 0: dev F1: 0.6218990590248076, test F1: 0.6196589769307924
epoch 1: dev F1: 0.6512136453094249, test F1: 0.6552458185504308
epoch 2: dev F1: 0.7469775474956822, test F1: 0.7369901547116737
epoch 3: dev F1: 0.6816816816816816, test F1: 0.685594321703489
epoch 4: dev F1: 0.7550843790566855, test F1: 0.7470723651286157
epoch 5: dev F1: 0.7624784853700516, test F1: 0.7482357618526986
epoch 6: dev F1: 0.7304055473839042, test F1: 0.7266874756145143
epoch 7: dev F1: 0.7833407177669471, test F1: 0.7696565863659661
epoch 8: dev F1: 0.7689260563380282, test F1: 0.746217121966081
epoch 9: dev F1: 0.7817984832069339, test F1: 0.7576124218592458
epoch 10: dev F1: 0.7933634992458521, test F1: 0.770678950530743
epoch 11: dev F1: 0.7926086956521738, test F1: 0.7637494981934967
epoch 12: dev F1: 0.8042003937869174, test F1: 0.769355001514081

Process Process-2:
Traceback (most recent call last):
File "/home/guoweihu/anaconda3/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/home/guoweihu/anaconda3/lib/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/home/guoweihu/gwh/untitled/HRL-RE-master/HRL-RE-master/code/TrainProcess.py", line 64, in worker
acc, cnt, tot, loss = workProcess(model, datas, sample_round, mode)
File "/home/guoweihu/gwh/untitled/HRL-RE-master/HRL-RE-master/code/TrainProcess.py", line 32, in workProcess
top_action, top_actprob, bot_action, bot_actprob = model(mode, data['text'],preoptions, preactions)
File "/home/guoweihu/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/home/guoweihu/gwh/untitled/HRL-RE-master/HRL-RE-master/code/Model.py", line 80, in forward
prehid, prec = self.preLSTML(wvs[x], (prehid, prec))
File "/home/guoweihu/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/home/guoweihu/anaconda3/lib/python3.6/site-packages/torch/nn/modules/rnn.py", line 1058, in forward
self.bias_ih, self.bias_hh,
IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

Hi, I'm trying to train a model with my own data. So I want to know the meaning of "tags", the list of numbers in data.

For example, a json in train data:

{"sentext": "if we can figure out how to program thousands of cores on a chip , the future looks rosy , '' said david a. patterson , a university of california , berkeley computer scientist who is a co-author of one of the standard textbooks on microprocessor design . ''", "entities": ["california", "berkeley"], "relations": [{"rtext": "/location/location/contains", "em2": "berkeley", "em1": "california", "tags": [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 0, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}]}