Cannot reproduce the experimental result shown in Table 1 about pdtb3 HOT 3 CLOSED

Hi! Thanks for raising this issue. Comparing the parameters with what's in our log, the only major difference that jumps out to me is the random seed; we set seed={1, 2, 3, 4, 5} to get the results in Table 1. With seed=1, the mean across 12 validation folds was 50.29 (std=2.02; note that the mean and std in Table 1 are from mean of means across the 5 seeds).

We did notice that even with the --deterministic flag, there were slight performance fluctuations across different machines (we believe this is an inherent GPU issue), but I remember the difference to be not as large as the results you have gotten.

Here is the full set of parameters (with some dir names substituted) that we used for a single run of the experiment, namely cross-validation for PDTB2, Fold 2 with bert-base. For this particular run, the accuracy was 49.87.

Training/evaluation parameters Namespace(adam_epsilon=1e-08, cache_dir='', config_name='', cuda_no=0, data_dir='my/dir', 
device=device(type='cuda'), do_eval=True, do_lower_case=True, do_train=True, eval_all_checkpoints=False,
evaluate_during_training=True, fp16=False, fp16_opt_level='O1', gradient_accumulation_steps=1, intermediate_weights=None,
learning_rate=5e-06, local_rank=-1, logging_steps=500, max_grad_norm=1.0, max_seq_length=128, max_steps=-1,
model_name_or_path='bert-base-uncased', model_type='bert', n_gpu=1, no_cuda=False, num_train_epochs=10.0,
output_dir='data/xval_bert_base_seed1_fold_2', output_mode='classification', overwrite_cache=False,
overwrite_output_dir=False, patience=5, per_gpu_eval_batch_size=32, per_gpu_train_batch_size=8, save_steps=500, seed=1,
server_ip='', server_port='', task_name='pdtb2_level2', tb_logdir='', tokenizer_name='', validation_metric='acc',
warmup_steps=0, weight_decay=0.0)

Please let us know if you still have trouble getting similar numbers! Would also be happy to look at the full log, if you send it to [email protected] .

from pdtb3.

Also posting the model config, in case this helps:

Model config {
"architectures": [ "BertForMaskedLM" ], "attention_probs_dropout_prob": 0.1, "finetuning_task": "pdtb2_level2", "hidden_act": "gelu", "hidden_dropout_prob": 0.1, "hidden_size": 768, "initializer_range": 0.02, "intermediate_size": 3072, "layer_norm_eps": 1e-12, "max_position_embeddings": 512, "num_attention_heads": 12, "num_hidden_layers": 12, "num_labels": 11, "output_attentions": false, "output_hidden_states": false, "torchscript": false, "type_vocab_size": 2, "vocab_size": 30522 }

from pdtb3.

Hi Kiyomaru,

Since you've raised this issue we ran small replication experiments ourselves by setting up the repo from scratch and running the experiments on two different machines. And we got results similar in number to our originally reported result rather than the lower number you reported.

I wonder if the issue could lie in the data rather than pytorch-transformers. I've noticed that running the preprocessing code with Python version 3.6.9 throws a UnicodeDecodeError for me. If you've had to patch this error yourself to run the preprocessing code, maybe some unicode gunk happened that made the data subtly different. Perhaps running preprocessing again with Python > 3.7.3 and rerunning the experiment (making sure to delete cached data files) would help? This is just a speculation, but something worth checking out since you were using Python 3.6.

from pdtb3.