hackiey / qanet-pytorch Goto Github PK

Hi, thanks for the implementation. Would you mind adding a license? Would love to try the model but right now it is not possible.

I tested the performance of this implementation. What I get is:
With ema (26 epochs): EM: 77.16 F1: 66.43

My repository is in https://github.com/BangLiu/QANet-PyTorch,
and https://github.com/BangLiu/QANet-PyTorch/blob/master/model/QANet_hackiey.py this model file is directly merged from your repository. I don't know why I cannot reach F1 79 and EM 70 even with EMA, so I am still trying to find potential bugs in my code. It will be great if you can take a look at my repository and check the potential problem that caused the performance difference.

I also tried to set num_head = 8 for both model_encoder_n_heads and embed_encoder_n_heads. But I get out of memory problem.

That is all what I have found currently.

when I run you code on 1080Ti 12G, about 20 steps,I meet out of memory
RuntimeError: cuda runtime error (2) : out of memory at /pytorch/aten/src/THC/generic/THCStorage.cu:58
do you meet this problem?

Hi @hackiey,

Thanks for the code, I've successfully reproduced an em 67.515 results (w/o EMA)! But when I read the codes carefully, I noticed something might be wrong in self_attention.py,

heads = torch.zeros(self.n_heads, batch_size, l, self.value_dim, device=device)  # L30, creation
x = heads.view(batch_size, l, -1)  # L47,  resize

https://github.com/hackiey/QAnet-pytorch/blob/master/qanet/self_attention.py#L47

which I think should be

x = heads.permute(1, 2, 0, 3).contiguous().view(batch_size, l, -1)

Does this make sense to you? I've also tested the code with the proposed change, but the result seems not changing. Any idea how it happens?

the constants.py file specifies cuda:0 as device when cuda is available, is there a way to use multiple GPUs by modifying the constants.py or other parts of the code?