The exbert's discuss from yaserjaradeh

exbert's Issues

Link prediction results on the UMLS

Hello, Mr.Jaradeh. Sorry to disturb you. I have tried the example how to run UMLS with Sci-BERT on link prediction task, as you mentioned in the README file.

python3 exBERT.py --task htp \
     --do_train \
     --do_eval \
     --do_predict \
     --data_dir ./data/UMLS \
     --bert_model allenai/scibert_scivocab_uncased \
     --max_seq_length 15 \
     --train_batch_size 32 \
     --learning_rate 5e-5 \
     --num_train_epochs 8.0 \
     --output_dir ./output_UMLS/ \
     --gradient_accumulation_steps 1 \
     --eval_batch_size 135 \
     --fp16

And I got the following results：

test_loss:		0.8371883630752563
test_simple_accuracy:		0.7947677207483119
test_hits_left_@1:		0.12405446293494705
test_hits_right_@1:		0.12405446293494705
test_hits_@1:		0.12405446293494705
test_hits_left_@3:		0.41603630862329805
test_hits_right_@3:		0.41603630862329805
test_hits_@3:		0.41603630862329805
test_hits_left_@10:		0.9077155824508321
test_hits_right_@10:		0.9077155824508321
test_hits_@10:		0.9077155824508321
test_mean_rank_left:		5.193645990922844
test_mean_rank_right:		5.193645990922844
test_mean_rank:		5.193645990922844
test_mean_reciprocal_rank_left:		0.3445958540911836
test_mean_reciprocal_rank_right:		0.3445958540911836
test_mean_reciprocal_rank:		0.3445958540911836
test_runtime:		16.7289
test_samples_per_second:		1620.012
test_steps_per_second:		3.049

There is somewhat different from the results published in the paper. Is there something wrong with the process I'm running or someplace I forgot to modify?

Recommend Projects