chenrocks / uniter Goto Github PK

View Code? Open in Web Editor NEW

772.0 18.0 109.0 176 KB

Research code for ECCV 2020 paper "UNITER: UNiversal Image-TExt Representation Learning"

Home Page: https://arxiv.org/abs/1909.11740

Dockerfile 0.30% Python 96.69% Shell 3.02%

transformers pre-training vision-and-language pytorch

uniter's People

Contributors

Stargazers

Watchers

Forkers

brucew91 namisan jaeyun95 hexiang-hu yjxiao nooneust wanboyang panda0881 mikejqzhang zhanghanbo taaccoo-beta agam-kashyap intersun zmykevin david-mys tiendzung-le higuseonhye youngfly11 dianarolien freedom1810 dandelin weiyunfei shivgarg bruinxiong floschne rohanrajnair claw-lab zeta1999 cuiyuhao1996 penghu-cs hitluobin yipeng-sun fightingfighting ramaneswaran jayleicn bei21 qingervt eaidova xixiareone jaredfern chxw20 kris927b yichunli95 billyang98 xeransis liupeng0606 leeflora liaojh1998 kent0304 smiyawaki0820 mohbattharani wh-forker hwanheelee1993 carboncoo maxylee davidyang2008 adrielkuek radidd paperfactory zhangxuemiao daniel00008 qiao1025566574 deepaliverma calvinzhao0502 lesliechen233 abhisheksahoo242 zyy598 wintersurvival realtaki anonymousdestroyer zgj-gutou mitjanikolaus sunlizhuang hellothewholeworld cv-ip huang-xx longkukuhi zhuxuhan sanjayss34 nitcoh 2292384454 dmcinerney spyairsg jeozhao arealgoodname txhan yyxx1997 rainyday7 glamor-usc s-wheels fatemeh490 hanxuhu moyutianque woojeongjin shiyicong naoya-takagi garett0 hypercookie demo0617 bennokrojer

uniter's Issues

broadcast_tensors get stuck

Hi, I met the following problem recently:

W /tmp/pip-install-fzrlm1c4/horovod/horovod/common/stall_inspector.cc:105] One or more tensors were sub
mitted to be reduced, gathered or broadcasted by subset of ranks and are waiting for remainder of ranks for more than 60 seconds. This may indicate that different ranks
are trying to submit different tensors or that only subset of ranks is submitting tensors, which will cause deadlock

It happed when broadcast the model using train_vpa.
I run the code on two gpus. Do you know how to solve it? Thanks

Could you please provide the feature used in SNLI-VE task?

Just as the title, could you please also provide the processed data(image feature, location information) used in SNLI-VE task?

Have you meet this error?

[1,2]: File "train_itm.py", line 410, in
[1,2]: main(args)
[1,2]: File "train_itm.py", line 221, in main
[1,2]: LOGGER.info(f"image retrieval R1: "
[1,2]:KeyError: 'img_r1'
[1,1]:Traceback (most recent call last):
[1,1]: File "train_itm.py", line 410, in
[1,1]: main(args)
[1,1]: File "train_itm.py", line 221, in main
[1,1]: LOGGER.info(f"image retrieval R1: "
[1,1]:KeyError: 'img_r1'
[1,3]:Traceback (most recent call last):
[1,3]: File "train_itm.py", line 410, in
[1,3]: main(args)
[1,3]: File "train_itm.py", line 221, in main
[1,3]: LOGGER.info(f"image retrieval R1: "
[1,3]:KeyError: 'img_r1'
[1,0]:
[1,0]:10/24/2020 15:08:26 - INFO - main - evaluation finished in 565 seconds
[1,0]:10/24/2020 15:08:26 - INFO - main - image retrieval R1: 63.73,
[1,0]:image retrieval R5: 87.61,
[1,0]:image retrieval R10: 93.51
[1,0]:text retrieval R1: 74.95,
[1,0]:text retrieval R5: 92.70,
[1,0]:text retrieval R10: 97.14
[1,0]:10/24/2020 15:08:26 - INFO - main - ==================================================================
[1,0]:Traceback (most recent call last):
[1,0]: File "train_itm.py", line 410, in
[1,0]: main(args)
[1,0]: File "train_itm.py", line 176, in main
[1,0]: all_reduce_and_rescale_tensors(grads, float(1))
[1,0]: File "/root/paddlejob/workspace/env_run/utils/distributed.py", line 35, in all_reduce_and_rescale_tensors
[1,0]: hvd.allreduce_(buffer_t[:offset])
[1,0]: File "/opt/conda/envs/py36/lib/python3.6/site-packages/horovod/torch/mpi_ops.py", line 252, in allreduce_
[1,0]: handle = allreduce_async_(tensor, average, name, op)
[1,0]: File "/opt/conda/envs/py36/lib/python3.6/site-packages/horovod/torch/mpi_ops.py", line 228, in allreduce_async_
[1,0]: return _allreduce_async(tensor, tensor, name, op)
[1,0]: File "/opt/conda/envs/py36/lib/python3.6/site-packages/horovod/torch/mpi_ops.py", line 127, in _allreduce_async
[1,0]: name.encode() if name is not None else _NULL, true_op)
[1,0]:RuntimeError: Horovod has been shut down. This was caused by an exception on one of the ranks or an attempt to allreduce, allgather or broadcast a tensor after one of the ranks finished execution. If the shutdown was caused by an exception, you should see the exception in the log before the first shutdown message.
[1,0]:
10%|# | 2000/20000 [34:10<5:07:33, 1.03s/it]-------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.

mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was

Process name: [[10998,1],3]
Exit code: 1

RefCOCO training / evaluation details

Hello,
I have some questions regarding RefCOCO/+/g training / evaluation details.

Are you going to upload RefCOCO/+/g training/evaluation codes?
Which boxes did you finetune UNITER on?
Which boxes did you use to evaluate on val, test, val^d, and test^d evaluation respectively? Did you use Mask R-CNN boxes from MattNet?

Table from UNITER

It seems ViLBERT-MT authors finetuned their model on 100 BUTD boxes + Mask R-CNN boxes from MattNet-> code.
Then they used 100 BUTD boxes during evaluation -> code

I calculated oracle scores on RefCOCOg val split: "if there exists a candidate box with iou(candidate,target) > 0.5 => correct"

Mask R-CNN boxes from MAttNet -> 86.10%
MS COCO GT boxes -> 99.6%
VilBERT-MT's 100 BUTD boxes on RefCOCOg -> 96.53%

Since BUTD boxes have better coverage on Mask R-CNN boxes from MAttNet, I don't think this is fair comparison to MattNet. Also this is not consistent with the ViLBERT-MT paper.

Paragraph from ViLBERT-MT

ViLBERT-MT authors compared ViLBERT-MT and UNITER on test^d. I wonder which boxes you used for UNITER finetuning and evaluation.

Table from ViLBERT-MT

VCR Second Stage Pretraining

Would it be possible to release the code or references for performing the second stage pretraining for VCR?

Pretraining Code

Hi,

Can you share your pretraining code as well ?

Thanks.

About two stage pre-train on VCR task.

one stage pre-training approach is using all task(MLM+ITM+MRC-kl+MRFR+WRA).

two stage pre-training is using MLM, MRFR , MRC task. is that right?

Thank you for always kind reply :)

confusion about WRA

Thank you for the great method UNITER (both code and the paper), but I really got confused when I read the detials of WRA, I'm so thankful if you are convenient to release the code of the parts of WRA loss functions.

Best wishes and appreciations anyhow.

Model image in paper

I noticed that in the paper, the input of the transformer is illustrated through the model image. The image is on the left while text is on the right.

However, in the code

UNITER/model/model.py

Lines 326 to 327 in 4fae838

 embedding_output = torch.gather(torch.cat([txt_emb, img_emb], dim=1), 

 dim=1, index=gather_index)

it's the other way around.

Just curious, I guess the performance is not affected.

i got RuntimeError: Horovod has been shut down. error

during training on VCR i got this error.
have you seen this error?

thank you

root@4a938104a7f8:/src# horovodrun -np 2 python train_vcr.py --config config/train-vcr-base-4gpu.json --output_dir /src/output
[1,0]<stderr>:09/18/2020 05:52:45 - INFO - __main__ -   device: cuda:0 n_gpu: 2, rank: 0, 16-bits training: True
[1,1]<stderr>:09/18/2020 05:52:45 - INFO - __main__ -   device: cuda:1 n_gpu: 2, rank: 1, 16-bits training: True
[1,0]<stderr>:09/18/2020 05:52:45 - INFO - __main__ -   Loading Train Dataset ['/txt/vcr_train.db/'], ['/img/vcr_gt_train/;/img/vcr_train/']
[1,1]<stderr>:09/18/2020 05:52:45 - INFO - __main__ -   Loading Train Dataset ['/txt/vcr_train.db/'], ['/img/vcr_gt_train/;/img/vcr_train/']
[1,1]<stderr>:09/18/2020 05:52:48 - INFO - __main__ -   Loading Val Dataset /txt/vcr_val.db/, /img/vcr_gt_val/;/img/vcr_val/
[1,1]<stderr>:09/18/2020 05:52:48 - INFO - model.model -   Model config {
[1,1]<stderr>:  "attention_probs_dropout_prob": 0.1,
[1,1]<stderr>:  "hidden_act": "gelu",
[1,1]<stderr>:  "hidden_dropout_prob": 0.1,
[1,1]<stderr>:  "hidden_size": 768,
[1,1]<stderr>:  "initializer_range": 0.02,
[1,1]<stderr>:  "intermediate_size": 3072,
[1,1]<stderr>:  "max_position_embeddings": 512,
[1,1]<stderr>:  "num_attention_heads": 12,
[1,1]<stderr>:  "num_hidden_layers": 12,
[1,1]<stderr>:  "type_vocab_size": 2,
[1,1]<stderr>:  "vocab_size": 28996
[1,1]<stderr>:}
[1,1]<stderr>:
[1,0]<stderr>:09/18/2020 05:52:48 - INFO - __main__ -   Loading Val Dataset /txt/vcr_val.db/, /img/vcr_gt_val/;/img/vcr_val/
[1,0]<stderr>:09/18/2020 05:52:48 - INFO - model.model -   Model config {
[1,0]<stderr>:  "attention_probs_dropout_prob": 0.1,
[1,0]<stderr>:  "hidden_act": "gelu",
[1,0]<stderr>:  "hidden_dropout_prob": 0.1,
[1,0]<stderr>:  "hidden_size": 768,
[1,0]<stderr>:  "initializer_range": 0.02,
[1,0]<stderr>:  "intermediate_size": 3072,
[1,0]<stderr>:  "max_position_embeddings": 512,
[1,0]<stderr>:  "num_attention_heads": 12,
[1,0]<stderr>:  "num_hidden_layers": 12,
[1,0]<stderr>:  "type_vocab_size": 2,
[1,0]<stderr>:  "vocab_size": 28996
[1,0]<stderr>:}
[1,0]<stderr>:
[1,1]<stderr>:09/18/2020 05:52:49 - INFO - model.model -   Weights of UniterForVisualCommonsenseReasoning not initialized from pretrained model: ['uniter.embeddings.word_embeddings.weight', 'uniter.embeddings.position_embeddings.weight', 'uniter.embeddings.token_type_embeddings.weight', 'uniter.embeddings.LayerNorm.weight', 'uniter.embeddings.LayerNorm.bias', 'uniter.img_embeddings.img_linear.weight', 'uniter.img_embeddings.img_linear.bias', 'uniter.img_embeddings.img_layer_norm.weight', 'uniter.img_embeddings.img_layer_norm.bias', 'uniter.img_embeddings.pos_layer_norm.weight', 'uniter.img_embeddings.pos_layer_norm.bias', 'uniter.img_embeddings.pos_linear.weight', 'uniter.img_embeddings.pos_linear.bias', 'uniter.img_embeddings.mask_embedding.weight', 'uniter.img_embeddings.LayerNorm.weight', 'uniter.img_embeddings.LayerNorm.bias', 'uniter.encoder.layer.0.attention.self.query.weight', 'uniter.encoder.layer.0.attention.self.query.bias', 'uniter.encoder.layer.0.attention.self.key.weight', 'uniter.encoder.layer.0.attention.self.key.bias', 'uniter.encoder.layer.0.attention.self.value.weight', 'uniter.encoder.layer.0.attention.self.value.bias', 'uniter.encoder.layer.0.attention.output.dense.weight', 'uniter.encoder.layer.0.attention.output.dense.bias', 'uniter.encoder.layer.0.attention.output.LayerNorm.weight', 'uniter.encoder.layer.0.attention.output.LayerNorm.bias', 'uniter.encoder.layer.0.intermediate.dense.weight', 'uniter.encoder.layer.0.intermediate.dense.bias', 'uniter.encoder.layer.0.output.dense.weight', 'uniter.encoder.layer.0.output.dense.bias', 'uniter.encoder.layer.0.output.LayerNorm.weight', 'uniter.encoder.layer.0.output.LayerNorm.bias', 'uniter.encoder.layer.1.attention.self.query.weight', 'uniter.encoder.layer.1.attention.self.query.bias', 'uniter.encoder.layer.1.attention.self.key.weight', 'uniter.encoder.layer.1.attention.self.key.bias', 'uniter.encoder.layer.1.attention.self.value.weight', 'uniter.encoder.layer.1.attention.self.value.bias', 'uniter.encoder.layer.1.attention.output.dense.weight', 'uniter.encoder.layer.1.attention.output.dense.bias', 'uniter.encoder.layer.1.attention.output.LayerNorm.weight', 'uniter.encoder.layer.1.attention.output.LayerNorm.bias', 'uniter.encoder.layer.1.intermediate.dense.weight', 'uniter.encoder.layer.1.intermediate.dense.bias', 'uniter.encoder.layer.1.output.dense.weight', 'uniter.encoder.layer.1.output.dense.bias', 'uniter.encoder.layer.1.output.LayerNorm.weight', 'uniter.encoder.layer.1.output.LayerNorm.bias', 'uniter.encoder.layer.2.attention.self.query.weight', 'uniter.encoder.layer.2.attention.self.query.bias', 'uniter.encoder.layer.2.attention.self.key.weight', 'uniter.encoder.layer.2.attention.self.key.bias', 'uniter.encoder.layer.2.attention.self.value.weight', 'uniter.encoder.layer.2.attention.self.value.bias', 'uniter.encoder.layer.2.attention.output.dense.weight', 'uniter.encoder.layer.2.attention.output.dense.bias', 'uniter.encoder.layer.2.attention.output.LayerNorm.weight', 'uniter.encoder.layer.2.attention.output.LayerNorm.bias', 'uniter.encoder.layer.2.intermediate.dense.weight', 'uniter.encoder.layer.2.intermediate.dense.bias', 'uniter.encoder.layer.2.output.dense.weight', 'uniter.encoder.layer.2.output.dense.bias', 'uniter.encoder.layer.2.output.LayerNorm.weight', 'uniter.encoder.layer.2.output.LayerNorm.bias', 'uniter.encoder.layer.3.attention.self.query.weight', 'uniter.encoder.layer.3.attention.self.query.bias', 'uniter.encoder.layer.3.attention.self.key.weight', 'uniter.encoder.layer.3.attention.self.key.bias', 'uniter.encoder.layer.3.attention.self.value.weight', 'uniter.encoder.layer.3.attention.self.value.bias', 'uniter.encoder.layer.3.attention.output.dense.weight', 'uniter.encoder.layer.3.attention.output.dense.bias', 'uniter.encoder.layer.3.attention.output.LayerNorm.weight', 'uniter.encoder.layer.3.attention.output.LayerNorm.bias', 'uniter.encoder.layer.3.intermediate.dense.weight', 'uniter.encoder.layer.3.intermediate.dense.bias', 'uniter.encoder.layer.3.output.dense.weight', 'uniter.encoder.layer.3.output.dense.bias', 'uniter.encoder.layer.3.output.LayerN[1,1]<stderr>:orm.weight', 'uniter.encoder.layer.3.output.LayerNorm.bias', 'uniter.encoder.layer.4.attention.self.query.weight', 'uniter.encoder.layer.4.attention.self.query.bias', 'uniter.encoder.layer.4.attention.self.key.weight', 'uniter.encoder.layer.4.attention.self.key.bias', 'uniter.encoder.layer.4.attention.self.value.weight', 'uniter.encoder.layer.4.attention.self.value.bias', 'uniter.encoder.layer.4.attention.output.dense.weight', 'uniter.encoder.layer.4.attention.output.dense.bias', 'uniter.encoder.layer.4.attention.output.LayerNorm.weight', 'uniter.encoder.layer.4.attention.output.LayerNorm.bias', 'uniter.encoder.layer.4.intermediate.dense.weight', 'uniter.encoder.layer.4.intermediate.dense.bias', 'uniter.encoder.layer.4.output.dense.weight', 'uniter.encoder.layer.4.output.dense.bias', 'uniter.encoder.layer.4.output.LayerNorm.weight', 'uniter.encoder.layer.4.output.LayerNorm.bias', 'uniter.encoder.layer.5.attention.self.query.weight', 'uniter.encoder.layer.5.attention.self.query.bias', 'uniter.encoder.layer.5.attention.self.key.weight', 'uniter.encoder.layer.5.attention.self.key.bias', 'uniter.encoder.layer.5.attention.self.value.weight', 'uniter.encoder.layer.5.attention.self.value.bias', 'uniter.encoder.layer.5.attention.output.dense.weight', 'uniter.encoder.layer.5.attention.output.dense.bias', 'uniter.encoder.layer.5.attention.output.LayerNorm.weight', 'uniter.encoder.layer.5.attention.output.LayerNorm.bias', 'uniter.encoder.layer.5.intermediate.dense.weight', 'uniter.encoder.layer.5.intermediate.dense.bias', 'uniter.encoder.layer.5.output.dense.weight', 'uniter.encoder.layer.5.output.dense.bias', 'uniter.encoder.layer.5.output.LayerNorm.weight', 'uniter.encoder.layer.5.output.LayerNorm.bias', 'uniter.encoder.layer.6.attention.self.query.weight', 'uniter.encoder.layer.6.attention.self.query.bias', 'uniter.encoder.layer.6.attention.self.key.weight', 'uniter.encoder.layer.6.attention.self.key.bias', 'uniter.encoder.layer.6.attention.self.value.weight', 'uniter.encoder.layer.6.attention.self.value.bias', 'uniter.encoder.layer.6.attention.output.dense.weight', 'uniter.encoder.layer.6.attention.output.dense.bias', 'uniter.encoder.layer.6.attention.output.LayerNorm.weight', 'uniter.encoder.layer.6.attention.output.LayerNorm.bias', 'uniter.encoder.layer.6.intermediate.dense.weight', 'uniter.encoder.layer.6.intermediate.dense.bias', 'uniter.encoder.layer.6.output.dense.weight', 'uniter.encoder.layer.6.output.dense.bias', 'uniter.encoder.layer.6.output.LayerNorm.weight', 'uniter.encoder.layer.6.output.LayerNorm.bias', 'uniter.encoder.layer.7.attention.self.query.weight', 'uniter.encoder.layer.7.attention.self.query.bias', 'uniter.encoder.layer.7.attention.self.key.weight', 'uniter.encoder.layer.7.attention.self.key.bias', 'uniter.encoder.layer.7.attention.self.value.weight', 'uniter.encoder.layer.7.attention.self.value.bias', 'uniter.encoder.layer.7.attention.output.dense.weight', 'uniter.encoder.layer.7.attention.output.dense.bias', 'uniter.encoder.layer.7.attention.output.LayerNorm.weight', 'uniter.encoder.layer.7.attention.output.LayerNorm.bias', 'uniter.encoder.layer.7.intermediate.dense.weight', 'uniter.encoder.layer.7.intermediate.dense.bias', 'uniter.encoder.layer.7.output.dense.weight', 'uniter.encoder.layer.7.output.dense.bias', 'uniter.encoder.layer.7.output.LayerNorm.weight', 'uniter.encoder.layer.7.output.LayerNorm.bias', 'uniter.encoder.layer.8.attention.self.query.weight', 'uniter.encoder.layer.8.attention.self.query.bias', 'uniter.encoder.layer.8.attention.self.key.weight', 'uniter.encoder.layer.8.attention.self.key.bias', 'uniter.encoder.layer.8.attention.self.value.weight', 'uniter.encoder.layer.8.attention.self.value.bias', 'uniter.encoder.layer.8.attention.output.dense.weight', 'uniter.encoder.layer.8.attention.output.dense.bias', 'uniter.encoder.layer.8.attention.output.LayerNorm.weight', 'uniter.encoder.layer.8.attention.output.LayerNorm.bias', 'uniter.encoder.layer.8.intermediate.dense.weight', 'uniter.encoder.layer.8.intermediate.dense.bias', 'uniter.encoder.layer.8.output.dense.weight', 'uniter.encoder.l[1,1]<stderr>:ayer.8.output.dense.bias', 'uniter.encoder.layer.8.output.LayerNorm.weight', 'uniter.encoder.layer.8.output.LayerNorm.bias', 'uniter.encoder.layer.9.attention.self.query.weight', 'uniter.encoder.layer.9.attention.self.query.bias', 'uniter.encoder.layer.9.attention.self.key.weight', 'uniter.encoder.layer.9.attention.self.key.bias', 'uniter.encoder.layer.9.attention.self.value.weight', 'uniter.encoder.layer.9.attention.self.value.bias', 'uniter.encoder.layer.9.attention.output.dense.weight', 'uniter.encoder.layer.9.attention.output.dense.bias', 'uniter.encoder.layer.9.attention.output.LayerNorm.weight', 'uniter.encoder.layer.9.attention.output.LayerNorm.bias', 'uniter.encoder.layer.9.intermediate.dense.weight', 'uniter.encoder.layer.9.intermediate.dense.bias', 'uniter.encoder.layer.9.output.dense.weight', 'uniter.encoder.layer.9.output.dense.bias', 'uniter.encoder.layer.9.output.LayerNorm.weight', 'uniter.encoder.layer.9.output.LayerNorm.bias', 'uniter.encoder.layer.10.attention.self.query.weight', 'uniter.encoder.layer.10.attention.self.query.bias', 'uniter.encoder.layer.10.attention.self.key.weight', 'uniter.encoder.layer.10.attention.self.key.bias', 'uniter.encoder.layer.10.attention.self.value.weight', 'uniter.encoder.layer.10.attention.self.value.bias', 'uniter.encoder.layer.10.attention.output.dense.weight', 'uniter.encoder.layer.10.attention.output.dense.bias', 'uniter.encoder.layer.10.attention.output.LayerNorm.weight', 'uniter.encoder.layer.10.attention.output.LayerNorm.bias', 'uniter.encoder.layer.10.intermediate.dense.weight', 'uniter.encoder.layer.10.intermediate.dense.bias', 'uniter.encoder.layer.10.output.dense.weight', 'uniter.encoder.layer.10.output.dense.bias', 'uniter.encoder.layer.10.output.LayerNorm.weight', 'uniter.encoder.layer.10.output.LayerNorm.bias', 'uniter.encoder.layer.11.attention.self.query.weight', 'uniter.encoder.layer.11.attention.self.query.bias', 'uniter.encoder.layer.11.attention.self.key.weight', 'uniter.encoder.layer.11.attention.self.key.bias', 'uniter.encoder.layer.11.attention.self.value.weight', 'uniter.encoder.layer.11.attention.self.value.bias', 'uniter.encoder.layer.11.attention.output.dense.weight', 'uniter.encoder.layer.11.attention.output.dense.bias', 'uniter.encoder.layer.11.attention.output.LayerNorm.weight', 'uniter.encoder.layer.11.attention.output.LayerNorm.bias', 'uniter.encoder.layer.11.intermediate.dense.weight', 'uniter.encoder.layer.11.intermediate.dense.bias', 'uniter.encoder.layer.11.output.dense.weight', 'uniter.encoder.layer.11.output.dense.bias', 'uniter.encoder.layer.11.output.LayerNorm.weight', 'uniter.encoder.layer.11.output.LayerNorm.bias', 'uniter.pooler.dense.weight', 'uniter.pooler.dense.bias', 'vcr_output.0.weight', 'vcr_output.0.bias', 'vcr_output.2.weight', 'vcr_output.2.bias', 'vcr_output.3.weight', 'vcr_output.3.bias']
[1,0]<stderr>:09/18/2020 05:52:50 - INFO - model.model -   Weights of UniterForVisualCommonsenseReasoning not initialized from pretrained model: ['uniter.embeddings.word_embeddings.weight', 'uniter.embeddings.position_embeddings.weight', 'uniter.embeddings.token_type_embeddings.weight', 'uniter.embeddings.LayerNorm.weight', 'uniter.embeddings.LayerNorm.bias', 'uniter.img_embeddings.img_linear.weight', 'uniter.img_embeddings.img_linear.bias', 'uniter.img_embeddings.img_layer_norm.weight', 'uniter.img_embeddings.img_layer_norm.bias', 'uniter.img_embeddings.pos_layer_norm.weight', 'uniter.img_embeddings.pos_layer_norm.bias', 'uniter.img_embeddings.pos_linear.weight', 'uniter.img_embeddings.pos_linear.bias', 'uniter.img_embeddings.mask_embedding.weight', 'uniter.img_embeddings.LayerNorm.weight', 'uniter.img_embeddings.LayerNorm.bias', 'uniter.encoder.layer.0.attention.self.query.weight', 'uniter.encoder.layer.0.attention.self.query.bias', 'uniter.encoder.layer.0.attention.self.key.weight', 'uniter.encoder.layer.0.attention.self.key.bias', 'uniter.encoder.layer.0.attention.self.value.weight', 'uniter.encoder.layer.0.attention.self.value.bias', 'uniter.encoder.layer.0.attention.output.dense.weight', 'uniter.encoder.layer.0.attention.output.dense.bias', 'uniter.encoder.layer.0.attention.output.LayerNorm.weight', 'uniter.encoder.layer.0.attention.output.LayerNorm.bias', 'uniter.encoder.layer.0.intermediate.dense.weight', 'uniter.encoder.layer.0.intermediate.dense.bias', 'uniter.encoder.layer.0.output.dense.weight', 'uniter.encoder.layer.0.output.dense.bias', 'uniter.encoder.layer.0.output.LayerNorm.weight', 'uniter.encoder.layer.0.output.LayerNorm.bias', 'uniter.encoder.layer.1.attention.self.query.weight', 'uniter.encoder.layer.1.attention.self.query.bias', 'uniter.encoder.layer.1.attention.self.key.weight', 'uniter.encoder.layer.1.attention.self.key.bias', 'uniter.encoder.layer.1.attention.self.value.weight', 'uniter.encoder.layer.1.attention.self.value.bias', 'uniter.encoder.layer.1.attention.output.dense.weight', 'uniter.encoder.layer.1.attention.output.dense.bias', 'uniter.encoder.layer.1.attention.output.LayerNorm.weight', 'uniter.encoder.layer.1.attention.output.LayerNorm.bias', 'uniter.encoder.layer.1.intermediate.dense.weight', 'uniter.encoder.layer.1.intermediate.dense.bias', 'uniter.encoder.layer.1.output.dense.weight', 'uniter.encoder.layer.1.output.dense.bias', 'uniter.encoder.layer.1.output.LayerNorm.weight', 'uniter.encoder.layer.1.output.LayerNorm.bias', 'uniter.encoder.layer.2.attention.self.query.weight', 'uniter.encoder.layer.2.attention.self.query.bias', 'uniter.encoder.layer.2.attention.self.key.weight', 'uniter.encoder.layer.2.attention.self.key.bias', 'uniter.encoder.layer.2.attention.self.value.weight', 'uniter.encoder.layer.2.attention.self.value.bias', 'uniter.encoder.layer.2.attention.output.dense.weight', 'uniter.encoder.layer.2.attention.output.dense.bias', 'uniter.encoder.layer.2.attention.output.LayerNorm.weight', 'uniter.encoder.layer.2.attention.output.LayerNorm.bias', 'uniter.encoder.layer.2.intermediate.dense.weight', 'uniter.encoder.layer.2.intermediate.dense.bias', 'uniter.encoder.layer.2.output.dense.weight', 'uniter.encoder.layer.2.output.dense.bias', 'uniter.encoder.layer.2.output.LayerNorm.weight', 'uniter.encoder.layer.2.output.LayerNorm.bias', 'uniter.encoder.layer.3.attention.self.query.weight', 'uniter.encoder.layer.3.attention.self.query.bias', 'uniter.encoder.layer.3.attention.self.key.weight', 'uniter.encoder.layer.3.attention.self.key.bias', 'uniter.encoder.layer.3.attention.self.value.weight', 'uniter.encoder.layer.3.attention.self.value.bias', 'uniter.encoder.layer.3.attention.output.dense.weight', 'uniter.encoder.layer.3.attention.output.dense.bias', 'uniter.encoder.layer.3.attention.output.LayerNorm.weight', 'uniter.encoder.layer.3.attention.output.LayerNorm.bias', 'uniter.encoder.layer.3.intermediate.dense.weight', 'uniter.encoder.layer.3.intermediate.dense.bias', 'uniter.encoder.layer.3.output.dense.weight', 'uniter.encoder.layer.3.output.dense.bias', 'uniter.encoder.layer.3.output.LayerN[1,0]<stderr>:orm.weight', 'uniter.encoder.layer.3.output.LayerNorm.bias', 'uniter.encoder.layer.4.attention.self.query.weight', 'uniter.encoder.layer.4.attention.self.query.bias', 'uniter.encoder.layer.4.attention.self.key.weight', 'uniter.encoder.layer.4.attention.self.key.bias', 'uniter.encoder.layer.4.attention.self.value.weight', 'uniter.encoder.layer.4.attention.self.value.bias', 'uniter.encoder.layer.4.attention.output.dense.weight', 'uniter.encoder.layer.4.attention.output.dense.bias', 'uniter.encoder.layer.4.attention.output.LayerNorm.weight', 'uniter.encoder.layer.4.attention.output.LayerNorm.bias', 'uniter.encoder.layer.4.intermediate.dense.weight', 'uniter.encoder.layer.4.intermediate.dense.bias', 'uniter.encoder.layer.4.output.dense.weight', 'uniter.encoder.layer.4.output.dense.bias', 'uniter.encoder.layer.4.output.LayerNorm.weight', 'uniter.encoder.layer.4.output.LayerNorm.bias', 'uniter.encoder.layer.5.attention.self.query.weight', 'uniter.encoder.layer.5.attention.self.query.bias', 'uniter.encoder.layer.5.attention.self.key.weight', 'uniter.encoder.layer.5.attention.self.key.bias', 'uniter.encoder.layer.5.attention.self.value.weight', 'uniter.encoder.layer.5.attention.self.value.bias', 'uniter.encoder.layer.5.attention.output.dense.weight', 'uniter.encoder.layer.5.attention.output.dense.bias', 'uniter.encoder.layer.5.attention.output.LayerNorm.weight', 'uniter.encoder.layer.5.attention.output.LayerNorm.bias', 'uniter.encoder.layer.5.intermediate.dense.weight', 'uniter.encoder.layer.5.intermediate.dense.bias', 'uniter.encoder.layer.5.output.dense.weight', 'uniter.encoder.layer.5.output.dense.bias', 'uniter.encoder.layer.5.output.LayerNorm.weight', 'uniter.encoder.layer.5.output.LayerNorm.bias', 'uniter.encoder.layer.6.attention.self.query.weight', 'uniter.encoder.layer.6.attention.self.query.bias', 'uniter.encoder.layer.6.attention.self.key.weight', 'uniter.encoder.layer.6.attention.self.key.bias', 'uniter.encoder.layer.6.attention.self.value.weight', 'uniter.encoder.layer.6.attention.self.value.bias', 'uniter.encoder.layer.6.attention.output.dense.weight', 'uniter.encoder.layer.6.attention.output.dense.bias', 'uniter.encoder.layer.6.attention.output.LayerNorm.weight', 'uniter.encoder.layer.6.attention.output.LayerNorm.bias', 'uniter.encoder.layer.6.intermediate.dense.weight', 'uniter.encoder.layer.6.intermediate.dense.bias', 'uniter.encoder.layer.6.output.dense.weight', 'uniter.encoder.layer.6.output.dense.bias', 'uniter.encoder.layer.6.output.LayerNorm.weight', 'uniter.encoder.layer.6.output.LayerNorm.bias', 'uniter.encoder.layer.7.attention.self.query.weight', 'uniter.encoder.layer.7.attention.self.query.bias', 'uniter.encoder.layer.7.attention.self.key.weight', 'uniter.encoder.layer.7.attention.self.key.bias', 'uniter.encoder.layer.7.attention.self.value.weight', 'uniter.encoder.layer.7.attention.self.value.bias', 'uniter.encoder.layer.7.attention.output.dense.weight', 'uniter.encoder.layer.7.attention.output.dense.bias', 'uniter.encoder.layer.7.attention.output.LayerNorm.weight', 'uniter.encoder.layer.7.attention.output.LayerNorm.bias', 'uniter.encoder.layer.7.intermediate.dense.weight', 'uniter.encoder.layer.7.intermediate.dense.bias', 'uniter.encoder.layer.7.output.dense.weight', 'uniter.encoder.layer.7.output.dense.bias', 'uniter.encoder.layer.7.output.LayerNorm.weight', 'uniter.encoder.layer.7.output.LayerNorm.bias', 'uniter.encoder.layer.8.attention.self.query.weight', 'uniter.encoder.layer.8.attention.self.query.bias', 'uniter.encoder.layer.8.attention.self.key.weight', 'uniter.encoder.layer.8.attention.self.key.bias', 'uniter.encoder.layer.8.attention.self.value.weight', 'uniter.encoder.layer.8.attention.self.value.bias', 'uniter.encoder.layer.8.attention.output.dense.weight', 'uniter.encoder.layer.8.attention.output.dense.bias', 'uniter.encoder.layer.8.attention.output.LayerNorm.weight', 'uniter.encoder.layer.8.attention.output.LayerNorm.bias', 'uniter.encoder.layer.8.intermediate.dense.weight', 'uniter.encoder.layer.8.intermediate.dense.bias', 'uniter.encoder.layer.8.output.dense.weight', 'uniter.encoder.l[1,0]<stderr>:ayer.8.output.dense.bias', 'uniter.encoder.layer.8.output.LayerNorm.weight', 'uniter.encoder.layer.8.output.LayerNorm.bias', 'uniter.encoder.layer.9.attention.self.query.weight', 'uniter.encoder.layer.9.attention.self.query.bias', 'uniter.encoder.layer.9.attention.self.key.weight', 'uniter.encoder.layer.9.attention.self.key.bias', 'uniter.encoder.layer.9.attention.self.value.weight', 'uniter.encoder.layer.9.attention.self.value.bias', 'uniter.encoder.layer.9.attention.output.dense.weight', 'uniter.encoder.layer.9.attention.output.dense.bias', 'uniter.encoder.layer.9.attention.output.LayerNorm.weight', 'uniter.encoder.layer.9.attention.output.LayerNorm.bias', 'uniter.encoder.layer.9.intermediate.dense.weight', 'uniter.encoder.layer.9.intermediate.dense.bias', 'uniter.encoder.layer.9.output.dense.weight', 'uniter.encoder.layer.9.output.dense.bias', 'uniter.encoder.layer.9.output.LayerNorm.weight', 'uniter.encoder.layer.9.output.LayerNorm.bias', 'uniter.encoder.layer.10.attention.self.query.weight', 'uniter.encoder.layer.10.attention.self.query.bias', 'uniter.encoder.layer.10.attention.self.key.weight', 'uniter.encoder.layer.10.attention.self.key.bias', 'uniter.encoder.layer.10.attention.self.value.weight', 'uniter.encoder.layer.10.attention.self.value.bias', 'uniter.encoder.layer.10.attention.output.dense.weight', 'uniter.encoder.layer.10.attention.output.dense.bias', 'uniter.encoder.layer.10.attention.output.LayerNorm.weight', 'uniter.encoder.layer.10.attention.output.LayerNorm.bias', 'uniter.encoder.layer.10.intermediate.dense.weight', 'uniter.encoder.layer.10.intermediate.dense.bias', 'uniter.encoder.layer.10.output.dense.weight', 'uniter.encoder.layer.10.output.dense.bias', 'uniter.encoder.layer.10.output.LayerNorm.weight', 'uniter.encoder.layer.10.output.LayerNorm.bias', 'uniter.encoder.layer.11.attention.self.query.weight', 'uniter.encoder.layer.11.attention.self.query.bias', 'uniter.encoder.layer.11.attention.self.key.weight', 'uniter.encoder.layer.11.attention.self.key.bias', 'uniter.encoder.layer.11.attention.self.value.weight', 'uniter.encoder.layer.11.attention.self.value.bias', 'uniter.encoder.layer.11.attention.output.dense.weight', 'uniter.encoder.layer.11.attention.output.dense.bias', 'uniter.encoder.layer.11.attention.output.LayerNorm.weight', 'uniter.encoder.layer.11.attention.output.LayerNorm.bias', 'uniter.encoder.layer.11.intermediate.dense.weight', 'uniter.encoder.layer.11.intermediate.dense.bias', 'uniter.encoder.layer.11.output.dense.weight', 'uniter.encoder.layer.11.output.dense.bias', 'uniter.encoder.layer.11.output.LayerNorm.weight', 'uniter.encoder.layer.11.output.LayerNorm.bias', 'uniter.pooler.dense.weight', 'uniter.pooler.dense.bias', 'vcr_output.0.weight', 'vcr_output.0.bias', 'vcr_output.2.weight', 'vcr_output.2.bias', 'vcr_output.3.weight', 'vcr_output.3.bias'][1,0]<stderr>:
[1,1]<stdout>:Unexpected_keys: ['feat_regress.net.3.bias', 'cls.predictions.decoder.weight', 'cls.predictions.bias', 'feat_regress.net.2.weight', 'feat_regress.net.2.bias', 'cls.predictions.transform.dense.weight', 'feat_regress.net.0.weight', 'region_classifier.net.0.bias', 'region_classifier.net.3.weight', 'region_classifier.net.3.bias', 'cls.predictions.transform.LayerNorm.weight', 'region_classifier.net.0.weight', 'feat_regress.net.0.bias', 'itm_output.bias', 'region_classifier.net.2.weight', 'cls.predictions.transform.LayerNorm.bias', 'itm_output.weight', 'region_classifier.net.2.bias', 'feat_regress.net.3.weight', 'cls.predictions.transform.dense.bias']
[1,1]<stdout>:Missing_keys: ['uniter.img_embeddings.mask_embedding.weight', 'vcr_output.0.bias', 'vcr_output.2.bias', 'vcr_output.3.bias', 'vcr_output.2.weight', 'vcr_output.0.weight', 'vcr_output.3.weight']
[1,0]<stdout>:Unexpected_keys: ['region_classifier.net.2.weight', 'cls.predictions.decoder.weight', 'region_classifier.net.2.bias', 'feat_regress.net.0.bias', 'cls.predictions.transform.dense.weight', 'feat_regress.net.0.weight', 'feat_regress.net.3.bias', 'cls.predictions.bias', 'region_classifier.net.0.weight', 'region_classifier.net.3.weight', 'cls.predictions.transform.LayerNorm.weight', 'feat_regress.net.2.bias', 'feat_regress.net.2.weight', 'feat_regress.net.3.weight', 'cls.predictions.transform.LayerNorm.bias', 'itm_output.weight', 'itm_output.bias', 'region_classifier.net.0.bias', 'region_classifier.net.3.bias', 'cls.predictions.transform.dense.bias']
[1,0]<stdout>:Missing_keys: ['vcr_output.3.bias', 'vcr_output.0.weight', 'vcr_output.0.bias', 'vcr_output.2.weight', 'uniter.img_embeddings.mask_embedding.weight', 'vcr_output.3.weight', 'vcr_output.2.bias']
[1,1]<stdout>:Selected optimization level O2:  FP16 training with FP32 batchnorm and FP32 master weights.
[1,1]<stdout>:
[1,1]<stdout>:Defaults for this optimization level are:
[1,1]<stdout>:enabled                : True
[1,1]<stdout>:opt_level              : O2
[1,1]<stdout>:cast_model_type        : torch.float16
[1,1]<stdout>:patch_torch_functions  : False
[1,1]<stdout>:keep_batchnorm_fp32    : True
[1,1]<stdout>:master_weights         : True
[1,1]<stdout>:loss_scale             : dynamic
[1,1]<stdout>:Processing user overrides (additional kwargs that are not None)...
[1,1]<stdout>:After processing overrides, optimization options are:
[1,1]<stdout>:enabled                : True
[1,1]<stdout>:opt_level              : O2
[1,1]<stdout>:cast_model_type        : torch.float16
[1,1]<stdout>:patch_torch_functions  : False
[1,1]<stdout>:keep_batchnorm_fp32    : True
[1,1]<stdout>:master_weights         : True
[1,1]<stdout>:loss_scale             : dynamic
[1,0]<stdout>:Selected optimization level O2:  FP16 training with FP32 batchnorm and FP32 master weights.
[1,0]<stdout>:
[1,0]<stdout>:Defaults for this optimization level are:
[1,0]<stdout>:enabled                : True
[1,0]<stdout>:opt_level              : O2
[1,0]<stdout>:cast_model_type        : torch.float16
[1,0]<stdout>:patch_torch_functions  : False
[1,0]<stdout>:keep_batchnorm_fp32    : True
[1,0]<stdout>:master_weights         : True
[1,0]<stdout>:loss_scale             : dynamic
[1,0]<stdout>:Processing user overrides (additional kwargs that are not None)...
[1,0]<stdout>:After processing overrides, optimization options are:
[1,0]<stdout>:enabled                : True
[1,0]<stdout>:opt_level              : O2
[1,0]<stdout>:cast_model_type        : torch.float16
[1,0]<stdout>:patch_torch_functions  : False
[1,0]<stdout>:keep_batchnorm_fp32    : True
[1,0]<stdout>:master_weights         : True
[1,0]<stdout>:loss_scale             : dynamic
  0%|          | 0/8000 [00:00<?, ?it/s][1,0]<stderr>:09/18/2020 05:52:51 - INFO - __main__ -   ***** Running training with 2 GPUs *****
[1,0]<stderr>:09/18/2020 05:52:51 - INFO - __main__ -     Num examples = 425182
[1,0]<stderr>:09/18/2020 05:52:51 - INFO - __main__ -     Batch size = 4000
[1,0]<stderr>:09/18/2020 05:52:51 - INFO - __main__ -     Accumulate steps = 5
[1,0]<stderr>:09/18/2020 05:52:51 - INFO - __main__ -     Num steps = 8000
[1,0]<stdout>:4a938104a7f8:212:253 [0] NCCL INFO NET/Socket : Using [0]eth0:172.17.0.2<0>
[1,0]<stdout>:4a938104a7f8:212:253 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).
[1,0]<stdout>:4a938104a7f8:212:253 [0] NCCL INFO NET/IB : No device found.
[1,0]<stdout>:NCCL version 2.4.6+cuda10.1
[1,1]<stdout>:4a938104a7f8:213:252 [1] NCCL INFO NET/Socket : Using [0]eth0:172.17.0.2<0>
[1,1]<stdout>:4a938104a7f8:213:252 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).
[1,1]<stdout>:4a938104a7f8:213:252 [1] NCCL INFO NET/IB : No device found.
[1,0]<stdout>:4a938104a7f8:212:253 [0] NCCL INFO Setting affinity for GPU 0 to ffff
[1,1]<stdout>:4a938104a7f8:213:252 [1] NCCL INFO Setting affinity for GPU 1 to ffff
[1,0]<stdout>:4a938104a7f8:212:253 [0] NCCL INFO Channel 00 :    0   1
[1,1]<stdout>:4a938104a7f8:213:252 [1] NCCL INFO Ring 00 : 1[1] -> 0[0] via P2P/IPC
[1,0]<stdout>:4a938104a7f8:212:253 [0] NCCL INFO Ring 00 : 0[0] -> 1[1] via P2P/IPC
[1,0]<stdout>:4a938104a7f8:212:253 [0] NCCL INFO Using 256 threads, Min Comp Cap 6, Trees disabled
[1,1]<stdout>:4a938104a7f8:213:252 [1] NCCL INFO comm 0x7f0af13a4090 rank 1 nranks 2 cudaDev 1 nvmlDev 1 - Init COMPLETE
[1,0]<stdout>:4a938104a7f8:212:253 [0] NCCL INFO comm 0x7f6a653a4450 rank 0 nranks 2 cudaDev 0 nvmlDev 0 - Init COMPLETE
[1,0]<stdout>:4a938104a7f8:212:253 [0] NCCL INFO Launch mode Parallel
[1,1]<stdout>:Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 32768.0
[1,0]<stdout>:Warning: NaN or Inf found in input tensor.
[1,0]<stdout>:Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 32768.0
  0%|          | 1/8000 [00:03<7:33:35,  3.40s/it][1,1]<stdout>:Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 16384.0
[1,0]<stdout>:Warning: NaN or Inf found in input tensor.
[1,0]<stdout>:Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 16384.0
  1%|▏         | 100/8000 [04:43<6:32:05,  2.98s/it][1,0]<stderr>:09/18/2020 05:57:34 - INFO - __main__ -   ============Step 100=============
[1,0]<stderr>:09/18/2020 05:57:34 - INFO - __main__ -   32000 examples trained at 113 ex/s
[1,0]<stderr>:09/18/2020 05:57:34 - INFO - __main__ -   ===========================================
  2%|▏         | 178/8000 [08:35<6:15:30,  2.88s/it][1,0]<stderr>:Traceback (most recent call last):
[1,0]<stderr>:  File "train_vcr.py", line 488, in <module>
[1,0]<stderr>:    main(args)
[1,0]<stderr>:  File "train_vcr.py", line 243, in main
[1,0]<stderr>:    scaled_loss.backward()
[1,0]<stderr>:  File "/opt/conda/lib/python3.6/site-packages/torch/tensor.py", line 107, in backward
[1,0]<stderr>:    torch.autograd.backward(self, gradient, retain_graph, create_graph)
[1,0]<stderr>:  File "/opt/conda/lib/python3.6/site-packages/torch/autograd/__init__.py", line 93, in backward
[1,0]<stderr>:    allow_unreachable=True)  # allow_unreachable flag
[1,0]<stderr>:RuntimeError: CUDA out of memory. Tried to allocate 36.00 MiB (GPU 0; 10.92 GiB total capacity; 5.57 GiB already allocated; 11.00 MiB free; 238.12 MiB cached)
[1,1]<stderr>:Traceback (most recent call last):
[1,1]<stderr>:  File "train_vcr.py", line 488, in <module>
[1,1]<stderr>:    main(args)
[1,1]<stderr>:  File "train_vcr.py", line 250, in main
[1,1]<stderr>:    all_reduce_and_rescale_tensors(grads, float(1))
[1,1]<stderr>:  File "/src/utils/distributed.py", line 35, in all_reduce_and_rescale_tensors
[1,1]<stderr>:    hvd.allreduce_(buffer_t[:offset])
[1,1]<stderr>:  File "/opt/conda/lib/python3.6/site-packages/horovod/torch/mpi_ops.py", line 200, in allreduce_
[1,1]<stderr>:    return synchronize(handle)
[1,1]<stderr>:  File "/opt/conda/lib/python3.6/site-packages/horovod/torch/mpi_ops.py", line 436, in synchronize
[1,1]<stderr>:    mpi_lib.horovod_torch_wait_and_clear(handle)
[1,1]<stderr>:RuntimeError: Horovod has been shut down. This was caused by an exception on one of the ranks or an attempt to allreduce, allgather or broadcast a tensor after one of the ranks finished execution. If the shutdown was caused by an exception, you should see the exception in the log before the first shutdown message.
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[50371,1],0]
  Exit code:    1

Unable to locate tools/generate_npz.py

While recreating the preprocessing pipeline, I am not able to locate the file generate_npz.py for image feature extraction. This file is referenced in scripts/extract_imgfeat.sh for your reference.

extracted features of the indomain dataset VG

Hi, I find that you use dense captions with all the box features. Can I concate them?

Why use more LayerNorm in ImageEmbedding?

UNITER/model/model.py

Line 252 in feda6bb

self.img_layer_norm = FusedLayerNorm(config.hidden_size, eps=1e-12)

From the paper, I think there only one layernorm in the last step, but I notice there are more layernorms.

Could you provide the logfile of pretraining process?

Hi,
I want to get an understanding of what happened during the pretraining, especially the train and validation loss change.
Thank you.

Training with other dataset.

Hi, Thank you for sharing nice work.
It seems your work is really good at vision-language multi-grounding.
I wanna check the performance with other tasks.
What should i follow, if i wanna pre-train the model with other dataset?
Thank you.

Retrieval time

Thank you so much for providing the code for UNITER, great work!
The README mentions the following "NOTE: Image-Text Retrieval is computationally heavy, especially on COCO."

Do you have a rough estimation in mind of how many hours (or days) does it take to run the full text-to-image retrieval on COCO on 1 GPU (eg. Tesla V100).

Thanks for your help!

Could you provide the code for the VCR dataset?

i want to train vcr data!
Could you provide the code for the VCR dataset?

thank you!:)

RuntimeError: CUDA error: device-side assert triggered

Hi everyone,

I am creating a UNITER model for a classification task, but after a few steps of training it launches the error

RuntimeError: CUDA error: device-side assert triggered

TRAINING...
0%
0/15 [00:01<?, ?it/s]
1%
4/563 [00:11<27:16, 2.93s/it]
0
/usr/local/lib/python3.6/dist-packages/apex/amp/_initialize.py:25: UserWarning: An input tensor was not cuda.
  warnings.warn("An input tensor was not cuda.")
1
2
3
4

RuntimeError                              Traceback (most recent call last)
<ipython-input-52-743c7191f227> in <module>()
     19       b_labels = batch['targets']
     20 
---> 21       b_logits = model(batch)
     22 
     23       logits.extend(b_logits)

11 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    720             result = self._slow_forward(*input, **kwargs)
    721         else:
--> 722             result = self.forward(*input, **kwargs)
    723         for hook in itertools.chain(
    724                 _global_forward_hooks.values(),

/usr/local/lib/python3.6/dist-packages/apex/amp/_initialize.py in new_fwd(*args, **kwargs)
    195                 def new_fwd(*args, **kwargs):
    196                     output = old_fwd(*applier(args, input_caster),
--> 197                                      **applier(kwargs, input_caster))
    198                     return applier(output, output_caster)
    199                 return new_fwd

<ipython-input-33-52c750f75352> in forward(self, batch, compute_loss)
     34                                       img_feat, img_pos_feat,
     35                                       attn_masks, gather_index,
---> 36                                       output_all_encoded_layers=False)
     37         pooled_output = self.uniter.pooler(sequence_output)
     38         output = self.hateful_memes_output(pooled_output)

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    720             result = self._slow_forward(*input, **kwargs)
    721         else:
--> 722             result = self.forward(*input, **kwargs)
    723         for hook in itertools.chain(
    724                 _global_forward_hooks.values(),

<ipython-input-32-5e573da9f309> in forward(self, input_ids, position_ids, img_feat, img_pos_feat, attention_mask, gather_index, img_masks, output_all_encoded_layers, txt_type_ids, img_type_ids)
    354         encoded_layers = self.encoder(
    355             embedding_output, extended_attention_mask,
--> 356             output_all_encoded_layers=output_all_encoded_layers)
    357         if not output_all_encoded_layers:
    358             encoded_layers = encoded_layers[-1]

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    720             result = self._slow_forward(*input, **kwargs)
    721         else:
--> 722             result = self.forward(*input, **kwargs)
    723         for hook in itertools.chain(
    724                 _global_forward_hooks.values(),

<ipython-input-32-5e573da9f309> in forward(self, input_, attention_mask, output_all_encoded_layers)
    277         hidden_states = input_
    278         for layer_module in self.layer:
--> 279             hidden_states = layer_module(hidden_states, attention_mask)
    280             if output_all_encoded_layers:
    281                 all_encoder_layers.append(hidden_states)

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    720             result = self._slow_forward(*input, **kwargs)
    721         else:
--> 722             result = self.forward(*input, **kwargs)
    723         for hook in itertools.chain(
    724                 _global_forward_hooks.values(),

<ipython-input-31-529775d24505> in forward(self, hidden_states, attention_mask)
    147     def forward(self, hidden_states, attention_mask):
    148         attention_output = self.attention(hidden_states, attention_mask)
--> 149         intermediate_output = self.intermediate(attention_output)
    150         layer_output = self.output(intermediate_output, attention_output)
    151         return layer_output

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    720             result = self._slow_forward(*input, **kwargs)
    721         else:
--> 722             result = self.forward(*input, **kwargs)
    723         for hook in itertools.chain(
    724                 _global_forward_hooks.values(),

<ipython-input-31-529775d24505> in forward(self, hidden_states)
    120     def forward(self, hidden_states):
    121         hidden_states = self.dense(hidden_states)
--> 122         hidden_states = self.intermediate_act_fn(hidden_states)
    123         return hidden_states
    124 

<ipython-input-31-529775d24505> in gelu(x)
     16         Also see https://arxiv.org/abs/1606.08415
     17     """
---> 18     return x * 0.5 * (1.0 + torch.erf(x / math.sqrt(2.0)))
     19 
     20 

RuntimeError: CUDA error: device-side assert triggered

Have you encounter this error before?

If I try to make a forward pass before training the outputs are correct

model.train()
outputs = model(d)
torch.sigmoid(outputs).cpu().detach().numpy().tolist()
/usr/local/lib/python3.6/dist-packages/apex/amp/_initialize.py:25: UserWarning: An input tensor was not cuda.
  warnings.warn("An input tensor was not cuda.")
[[0.4417332112789154],
 [0.382718026638031],
 [0.46414244174957275],
 [0.5104507803916931],
 [0.4497249126434326],
 [0.5214864015579224],
 [0.5086051225662231],
 [0.4487886130809784],
 [0.5447408556938171],
 [0.48516517877578735],
 [0.45522886514663696],
 [0.5446500778198242],
 [0.5219737887382507],
 [0.4610774517059326],
 [0.49035000801086426],
 [0.5698526501655579]]

The model

class UniterCls(UniterPreTrainedModel):

    def __init__(self, config, img_dim):
        super().__init__(config)
        self.uniter = UniterModel(config, img_dim)
        self.output = nn.Sequential(
            nn.Linear(config.hidden_size, config.hidden_size*2),
            GELU(),
            LayerNorm(config.hidden_size*2, eps=1e-12),
            nn.Linear(config.hidden_size*2, 1)
        )
        self.apply(self.init_weights)

    def forward(self, batch):
        batch = defaultdict(lambda: None, batch)
        input_ids = batch['input_ids'].to(device)
        position_ids = batch['position_ids'].to(device)
        img_feat = batch['img_feat'].to(device)
        img_pos_feat = batch['img_pos_feat'].to(device)
        attn_masks = batch['attn_masks'].to(device)
        gather_index = batch['gather_index'].to(device)
        sequence_output = self.uniter(input_ids, position_ids,
                                      img_feat, img_pos_feat,
                                      attn_masks, gather_index,
                                      output_all_encoded_layers=False)
        pooled_output = self.uniter.pooler(sequence_output)
        output = self.output(pooled_output)

        return output

Model not loading weights correctly

The following does not work -

from model import *

config_file_path='UNITER/config/uniter-base.json'
checkpoint = torch.load('UNITER/data/pretrained/uniter-base.pt')
config=UniterConfig(config_file_path)
model = UniterModel(config, img_dim = 2048).from_pretrained(config_file_path,new_ckpt,img_dim=2048)

The correct weights are not getting loaded by the model. I suspect it happens because the file uniter-base.pt has keys for UNITER weights which are of the form uniter.encoder.*, where as the model's state_dict expects keys to be of the form encoder.*.

A quick fix for it was to modify the loaded weights' keys to conform to the model.state_dict().keys()

Could you please provide the environment requirements ?

Like the above, Could you please provide the environment requirements? I'd like to build the environment on my server without docker. Thanks!

Difference between the conditional mechanism and the "Masked Language Modeling with Visual Clues" in VL-BERT

The NLVR2 data cannot be download for the download breaks

Like the above, we cannot download the nlvr2 dataset very well. The download process can be break easily, so we can't download the training dataset for that it's big, maybe it's the server problem?
Hope your response, thanks!

Question for reproducing the image retrieval performance

Hi, thanks for the great repo! I wonder would you be able to share the codes/models for reproducing the image retrieval results? Specifically, could you specify the architecture for the ITM layer?

We've encountered an issue in reproducing your results of image retrieval task on Flickr30k dataset.

I finetuned the image retrieval tasks (on Flickr30k) on your conceptual caption pretrained model (uniter-base.pt). Specifically, I finetuned on all architectures except the ITM head, as I don't know the specific architecture of the ITM head.

With the same hyperparameter in your paper, the best results for image retrieval on Flickr30k Recall@1 I could achieve is 63, which is far from the results in the paper: Table 4 IR Flickr Recall@1 73.

I wonder could you share the codes/models/architectures for image retrieval tasks?

Thanks a lot.

About the file to download the out-of-domain dataset

Hi. Could you provide the code for downloading the out-of-domain dataset (Conceptual Captions & SBU captions).

thank you!

Could you please upload the img_db/vg.tar file

Hi,

Thank you for releasing the code and checkpoints to reproduce the results of UNITER. We notice that the features of vg has still not beed uploaded yet, which is needed to run VQA fine-tuning. Could you please provide this file? Thank you very much!

About the configs for the out-of-domain and out-of-domain&in-domain pre-training

Hi,
Currently, there is only the in-domain pre-training config in this repo. Could you please provide the config files for the out-of-domain and the out-of-domain&in-domain pre-training process? Since the detection scripts and the docker image have been already provided, we can try to generate the out-of-domain feature files and do pre-train by ourselves.

Can you give a Repository Setup guide order to quickly build the environment

Vocabulary and single image-question pair prediction

Is the vocabulary available that takes the words of the questions and converts them to 'input_ids'?
Is there a function that does this for an input question?
Is there a code that take a single image-question pair and predicts the answer?

how to generate the image feature file?

I cannot find the code tools/generate_npz.py, i don't know how to generate the image feature file. Could you please upload your code? Thanks!

No "convert_imgdir.py"

When I trying to reproduce preprocessing, I find this file is missing. Can you upload that file? Thanks.

lmdb.Error: ./afs/uniter/img_db/flickr30k/feat_th0.2_max100_min10: No such device

I can't use your docker so I installed the library manually, I got this error, can you give some advice? thank you very much.

Could you provide visualization code?

Hi!

I want visualize result like your paper's Figure 7(Additional text-to-image attention visualization example).

Could you provide visualization code?

thank you:)

i got "subprocess.CalledProcessError: Command '['git', 'status', '--short']' returned non-zero exit status 128"

i got this problem.
how do i do?

thank you

root@da5f2b8bcf3b:/src# horovodrun -np 2 python train_vcr.py --config config/train-vcr-base-4gpu.json --output_dir /media/ailab/jaeyun/UNITER/output
[1,0]<stderr>:09/12/2020 19:01:03 - INFO - __main__ -   device: cuda:0 n_gpu: 2, rank: 0, 16-bits training: True
[1,1]<stderr>:09/12/2020 19:01:03 - INFO - __main__ -   device: cuda:1 n_gpu: 2, rank: 1, 16-bits training: True
[1,0]<stderr>:09/12/2020 19:01:03 - INFO - __main__ -   Loading Train Dataset ['/txt/vcr_train.db/'], ['/img/vcr_gt_train/;/img/vcr_train/']
[1,1]<stderr>:09/12/2020 19:01:03 - INFO - __main__ -   Loading Train Dataset ['/txt/vcr_train.db/'], ['/img/vcr_gt_train/;/img/vcr_train/']
[1,0]<stderr>:09/12/2020 19:01:06 - INFO - __main__ -   Loading Val Dataset /txt/vcr_val.db/, /img/vcr_gt_val/;/img/vcr_val/
[1,1]<stderr>:09/12/2020 19:01:06 - INFO - __main__ -   Loading Val Dataset /txt/vcr_val.db/, /img/vcr_gt_val/;/img/vcr_val/
[1,0]<stderr>:09/12/2020 19:01:06 - INFO - model.model -   Model config {
[1,0]<stderr>:  "attention_probs_dropout_prob": 0.1,
[1,0]<stderr>:  "hidden_act": "gelu",
[1,0]<stderr>:  "hidden_dropout_prob": 0.1,
[1,0]<stderr>:  "hidden_size": 768,
[1,0]<stderr>:  "initializer_range": 0.02,
[1,0]<stderr>:  "intermediate_size": 3072,
[1,0]<stderr>:  "max_position_embeddings": 512,
[1,0]<stderr>:  "num_attention_heads": 12,
[1,0]<stderr>:  "num_hidden_layers": 12,
[1,0]<stderr>:  "type_vocab_size": 2,
[1,0]<stderr>:  "vocab_size": 28996
[1,0]<stderr>:}
[1,0]<stderr>:
[1,1]<stderr>:09/12/2020 19:01:06 - INFO - model.model -   Model config {
[1,1]<stderr>:  "attention_probs_dropout_prob": 0.1,
[1,1]<stderr>:  "hidden_act": "gelu",
[1,1]<stderr>:  "hidden_dropout_prob": 0.1,
[1,1]<stderr>:  "hidden_size": 768,
[1,1]<stderr>:  "initializer_range": 0.02,
[1,1]<stderr>:  "intermediate_size": 3072,
[1,1]<stderr>:  "max_position_embeddings": 512,
[1,1]<stderr>:  "num_attention_heads": 12,
[1,1]<stderr>:  "num_hidden_layers": 12,
[1,1]<stderr>:  "type_vocab_size": 2,
[1,1]<stderr>:  "vocab_size": 28996
[1,1]<stderr>:}
[1,1]<stderr>:
[1,1]<stderr>:09/12/2020 19:01:08 - INFO - model.model -   Weights of UniterForVisualCommonsenseReasoning not initialized from pretrained model: ['uniter.embeddings.word_embeddings.weight', 'uniter.embeddings.position_embeddings.weight', 'uniter.embeddings.token_type_embeddings.weight', 'uniter.embeddings.LayerNorm.weight', 'uniter.embeddings.LayerNorm.bias', 'uniter.img_embeddings.img_linear.weight', 'uniter.img_embeddings.img_linear.bias', 'uniter.img_embeddings.img_layer_norm.weight', 'uniter.img_embeddings.img_layer_norm.bias', 'uniter.img_embeddings.pos_layer_norm.weight', 'uniter.img_embeddings.pos_layer_norm.bias', 'uniter.img_embeddings.pos_linear.weight', 'uniter.img_embeddings.pos_linear.bias', 'uniter.img_embeddings.mask_embedding.weight', 'uniter.img_embeddings.LayerNorm.weight', 'uniter.img_embeddings.LayerNorm.bias', 'uniter.encoder.layer.0.attention.self.query.weight', 'uniter.encoder.layer.0.attention.self.query.bias', 'uniter.encoder.layer.0.attention.self.key.weight', 'uniter.encoder.layer.0.attention.self.key.bias', 'uniter.encoder.layer.0.attention.self.value.weight', 'uniter.encoder.layer.0.attention.self.value.bias', 'uniter.encoder.layer.0.attention.output.dense.weight', 'uniter.encoder.layer.0.attention.output.dense.bias', 'uniter.encoder.layer.0.attention.output.LayerNorm.weight', 'uniter.encoder.layer.0.attention.output.LayerNorm.bias', 'uniter.encoder.layer.0.intermediate.dense.weight', 'uniter.encoder.layer.0.intermediate.dense.bias', 'uniter.encoder.layer.0.output.dense.weight', 'uniter.encoder.layer.0.output.dense.bias', 'uniter.encoder.layer.0.output.LayerNorm.weight', 'uniter.encoder.layer.0.output.LayerNorm.bias', 'uniter.encoder.layer.1.attention.self.query.weight', 'uniter.encoder.layer.1.attention.self.query.bias', 'uniter.encoder.layer.1.attention.self.key.weight', 'uniter.encoder.layer.1.attention.self.key.bias', 'uniter.encoder.layer.1.attention.self.value.weight', 'uniter.encoder.layer.1.attention.self.value.bias', 'uniter.encoder.layer.1.attention.output.dense.weight', 'uniter.encoder.layer.1.attention.output.dense.bias', 'uniter.encoder.layer.1.attention.output.LayerNorm.weight', 'uniter.encoder.layer.1.attention.output.LayerNorm.bias', 'uniter.encoder.layer.1.intermediate.dense.weight', 'uniter.encoder.layer.1.intermediate.dense.bias', 'uniter.encoder.layer.1.output.dense.weight', 'uniter.encoder.layer.1.output.dense.bias', 'uniter.encoder.layer.1.output.LayerNorm.weight', 'uniter.encoder.layer.1.output.LayerNorm.bias', 'uniter.encoder.layer.2.attention.self.query.weight', 'uniter.encoder.layer.2.attention.self.query.bias', 'uniter.encoder.layer.2.attention.self.key.weight', 'uniter.encoder.layer.2.attention.self.key.bias', 'uniter.encoder.layer.2.attention.self.value.weight', 'uniter.encoder.layer.2.attention.self.value.bias', 'uniter.encoder.layer.2.attention.output.dense.weight', 'uniter.encoder.layer.2.attention.output.dense.bias', 'uniter.encoder.layer.2.attention.output.LayerNorm.weight', 'uniter.encoder.layer.2.attention.output.LayerNorm.bias', 'uniter.encoder.layer.2.intermediate.dense.weight', 'uniter.encoder.layer.2.intermediate.dense.bias', 'uniter.encoder.layer.2.output.dense.weight', 'uniter.encoder.layer.2.output.dense.bias', 'uniter.encoder.layer.2.output.LayerNorm.weight', 'uniter.encoder.layer.2.output.LayerNorm.bias', 'uniter.encoder.layer.3.attention.self.query.weight', 'uniter.encoder.layer.3.attention.self.query.bias', 'uniter.encoder.layer.3.attention.self.key.weight', 'uniter.encoder.layer.3.attention.self.key.bias', 'uniter.encoder.layer.3.attention.self.value.weight', 'uniter.encoder.layer.3.attention.self.value.bias', 'uniter.encoder.layer.3.attention.output.dense.weight', 'uniter.encoder.layer.3.attention.output.dense.bias', 'uniter.encoder.layer.3.attention.output.LayerNorm.weight', 'uniter.encoder.layer.3.attention.output.LayerNorm.bias', 'uniter.encoder.layer.3.intermediate.dense.weight', 'uniter.encoder.layer.3.intermediate.dense.bias', 'uniter.encoder.layer.3.output.dense.weight', 'uniter.encoder.layer.3.output.dense.bias', 'uniter.encoder.layer.3.output.LayerN[1,1]<stderr>:orm.weight', 'uniter.encoder.layer.3.output.LayerNorm.bias', 'uniter.encoder.layer.4.attention.self.query.weight', 'uniter.encoder.layer.4.attention.self.query.bias', 'uniter.encoder.layer.4.attention.self.key.weight', 'uniter.encoder.layer.4.attention.self.key.bias', 'uniter.encoder.layer.4.attention.self.value.weight', 'uniter.encoder.layer.4.attention.self.value.bias', 'uniter.encoder.layer.4.attention.output.dense.weight', 'uniter.encoder.layer.4.attention.output.dense.bias', 'uniter.encoder.layer.4.attention.output.LayerNorm.weight', 'uniter.encoder.layer.4.attention.output.LayerNorm.bias', 'uniter.encoder.layer.4.intermediate.dense.weight', 'uniter.encoder.layer.4.intermediate.dense.bias', 'uniter.encoder.layer.4.output.dense.weight', 'uniter.encoder.layer.4.output.dense.bias', 'uniter.encoder.layer.4.output.LayerNorm.weight', 'uniter.encoder.layer.4.output.LayerNorm.bias', 'uniter.encoder.layer.5.attention.self.query.weight', 'uniter.encoder.layer.5.attention.self.query.bias', 'uniter.encoder.layer.5.attention.self.key.weight', 'uniter.encoder.layer.5.attention.self.key.bias', 'uniter.encoder.layer.5.attention.self.value.weight', 'uniter.encoder.layer.5.attention.self.value.bias', 'uniter.encoder.layer.5.attention.output.dense.weight', 'uniter.encoder.layer.5.attention.output.dense.bias', 'uniter.encoder.layer.5.attention.output.LayerNorm.weight', 'uniter.encoder.layer.5.attention.output.LayerNorm.bias', 'uniter.encoder.layer.5.intermediate.dense.weight', 'uniter.encoder.layer.5.intermediate.dense.bias', 'uniter.encoder.layer.5.output.dense.weight', 'uniter.encoder.layer.5.output.dense.bias', 'uniter.encoder.layer.5.output.LayerNorm.weight', 'uniter.encoder.layer.5.output.LayerNorm.bias', 'uniter.encoder.layer.6.attention.self.query.weight', 'uniter.encoder.layer.6.attention.self.query.bias', 'uniter.encoder.layer.6.attention.self.key.weight', 'uniter.encoder.layer.6.attention.self.key.bias', 'uniter.encoder.layer.6.attention.self.value.weight', 'uniter.encoder.layer.6.attention.self.value.bias', 'uniter.encoder.layer.6.attention.output.dense.weight', 'uniter.encoder.layer.6.attention.output.dense.bias', 'uniter.encoder.layer.6.attention.output.LayerNorm.weight', 'uniter.encoder.layer.6.attention.output.LayerNorm.bias', 'uniter.encoder.layer.6.intermediate.dense.weight', 'uniter.encoder.layer.6.intermediate.dense.bias', 'uniter.encoder.layer.6.output.dense.weight', 'uniter.encoder.layer.6.output.dense.bias', 'uniter.encoder.layer.6.output.LayerNorm.weight', 'uniter.encoder.layer.6.output.LayerNorm.bias', 'uniter.encoder.layer.7.attention.self.query.weight', 'uniter.encoder.layer.7.attention.self.query.bias', 'uniter.encoder.layer.7.attention.self.key.weight', 'uniter.encoder.layer.7.attention.self.key.bias', 'uniter.encoder.layer.7.attention.self.value.weight', 'uniter.encoder.layer.7.attention.self.value.bias', 'uniter.encoder.layer.7.attention.output.dense.weight', 'uniter.encoder.layer.7.attention.output.dense.bias', 'uniter.encoder.layer.7.attention.output.LayerNorm.weight', 'uniter.encoder.layer.7.attention.output.LayerNorm.bias', 'uniter.encoder.layer.7.intermediate.dense.weight', 'uniter.encoder.layer.7.intermediate.dense.bias', 'uniter.encoder.layer.7.output.dense.weight', 'uniter.encoder.layer.7.output.dense.bias', 'uniter.encoder.layer.7.output.LayerNorm.weight', 'uniter.encoder.layer.7.output.LayerNorm.bias', 'uniter.encoder.layer.8.attention.self.query.weight', 'uniter.encoder.layer.8.attention.self.query.bias', 'uniter.encoder.layer.8.attention.self.key.weight', 'uniter.encoder.layer.8.attention.self.key.bias', 'uniter.encoder.layer.8.attention.self.value.weight', 'uniter.encoder.layer.8.attention.self.value.bias', 'uniter.encoder.layer.8.attention.output.dense.weight', 'uniter.encoder.layer.8.attention.output.dense.bias', 'uniter.encoder.layer.8.attention.output.LayerNorm.weight', 'uniter.encoder.layer.8.attention.output.LayerNorm.bias', 'uniter.encoder.layer.8.intermediate.dense.weight', 'uniter.encoder.layer.8.intermediate.dense.bias', 'uniter.encoder.layer.8.output.dense.weight', 'uniter.encoder.l[1,1]<stderr>:ayer.8.output.dense.bias', 'uniter.encoder.layer.8.output.LayerNorm.weight', 'uniter.encoder.layer.8.output.LayerNorm.bias', 'uniter.encoder.layer.9.attention.self.query.weight', 'uniter.encoder.layer.9.attention.self.query.bias', 'uniter.encoder.layer.9.attention.self.key.weight', 'uniter.encoder.layer.9.attention.self.key.bias', 'uniter.encoder.layer.9.attention.self.value.weight', 'uniter.encoder.layer.9.attention.self.value.bias', 'uniter.encoder.layer.9.attention.output.dense.weight', 'uniter.encoder.layer.9.attention.output.dense.bias', 'uniter.encoder.layer.9.attention.output.LayerNorm.weight', 'uniter.encoder.layer.9.attention.output.LayerNorm.bias', 'uniter.encoder.layer.9.intermediate.dense.weight', 'uniter.encoder.layer.9.intermediate.dense.bias', 'uniter.encoder.layer.9.output.dense.weight', 'uniter.encoder.layer.9.output.dense.bias', 'uniter.encoder.layer.9.output.LayerNorm.weight', 'uniter.encoder.layer.9.output.LayerNorm.bias', 'uniter.encoder.layer.10.attention.self.query.weight', 'uniter.encoder.layer.10.attention.self.query.bias', 'uniter.encoder.layer.10.attention.self.key.weight', 'uniter.encoder.layer.10.attention.self.key.bias', 'uniter.encoder.layer.10.attention.self.value.weight', 'uniter.encoder.layer.10.attention.self.value.bias', 'uniter.encoder.layer.10.attention.output.dense.weight', 'uniter.encoder.layer.10.attention.output.dense.bias', 'uniter.encoder.layer.10.attention.output.LayerNorm.weight', 'uniter.encoder.layer.10.attention.output.LayerNorm.bias', 'uniter.encoder.layer.10.intermediate.dense.weight', 'uniter.encoder.layer.10.intermediate.dense.bias', 'uniter.encoder.layer.10.output.dense.weight', 'uniter.encoder.layer.10.output.dense.bias', 'uniter.encoder.layer.10.output.LayerNorm.weight', 'uniter.encoder.layer.10.output.LayerNorm.bias', 'uniter.encoder.layer.11.attention.self.query.weight', 'uniter.encoder.layer.11.attention.self.query.bias', 'uniter.encoder.layer.11.attention.self.key.weight', 'uniter.encoder.layer.11.attention.self.key.bias', 'uniter.encoder.layer.11.attention.self.value.weight', 'uniter.encoder.layer.11.attention.self.value.bias', 'uniter.encoder.layer.11.attention.output.dense.weight', 'uniter.encoder.layer.11.attention.output.dense.bias', 'uniter.encoder.layer.11.attention.output.LayerNorm.weight', 'uniter.encoder.layer.11.attention.output.LayerNorm.bias', 'uniter.encoder.layer.11.intermediate.dense.weight', 'uniter.encoder.layer.11.intermediate.dense.bias', 'uniter.encoder.layer.11.output.dense.weight', 'uniter.encoder.layer.11.output.dense.bias', 'uniter.encoder.layer.11.output.LayerNorm.weight', 'uniter.encoder.layer.11.output.LayerNorm.bias', 'uniter.pooler.dense.weight', 'uniter.pooler.dense.bias', 'vcr_output.0.weight', 'vcr_output.0.bias', 'vcr_output.2.weight', 'vcr_output.2.bias', 'vcr_output.3.weight', 'vcr_output.3.bias']
[1,0]<stderr>:09/12/2020 19:01:08 - INFO - model.model -   Weights of UniterForVisualCommonsenseReasoning not initialized from pretrained model: ['uniter.embeddings.word_embeddings.weight', 'uniter.embeddings.position_embeddings.weight', 'uniter.embeddings.token_type_embeddings.weight', 'uniter.embeddings.LayerNorm.weight', 'uniter.embeddings.LayerNorm.bias', 'uniter.img_embeddings.img_linear.weight', 'uniter.img_embeddings.img_linear.bias', 'uniter.img_embeddings.img_layer_norm.weight', 'uniter.img_embeddings.img_layer_norm.bias', 'uniter.img_embeddings.pos_layer_norm.weight', 'uniter.img_embeddings.pos_layer_norm.bias', 'uniter.img_embeddings.pos_linear.weight', 'uniter.img_embeddings.pos_linear.bias', 'uniter.img_embeddings.mask_embedding.weight', 'uniter.img_embeddings.LayerNorm.weight', 'uniter.img_embeddings.LayerNorm.bias', 'uniter.encoder.layer.0.attention.self.query.weight', 'uniter.encoder.layer.0.attention.self.query.bias', 'uniter.encoder.layer.0.attention.self.key.weight', 'uniter.encoder.layer.0.attention.self.key.bias', 'uniter.encoder.layer.0.attention.self.value.weight', 'uniter.encoder.layer.0.attention.self.value.bias', 'uniter.encoder.layer.0.attention.output.dense.weight', 'uniter.encoder.layer.0.attention.output.dense.bias', 'uniter.encoder.layer.0.attention.output.LayerNorm.weight', 'uniter.encoder.layer.0.attention.output.LayerNorm.bias', 'uniter.encoder.layer.0.intermediate.dense.weight', 'uniter.encoder.layer.0.intermediate.dense.bias', 'uniter.encoder.layer.0.output.dense.weight', 'uniter.encoder.layer.0.output.dense.bias', 'uniter.encoder.layer.0.output.LayerNorm.weight', 'uniter.encoder.layer.0.output.LayerNorm.bias', 'uniter.encoder.layer.1.attention.self.query.weight', 'uniter.encoder.layer.1.attention.self.query.bias', 'uniter.encoder.layer.1.attention.self.key.weight', 'uniter.encoder.layer.1.attention.self.key.bias', 'uniter.encoder.layer.1.attention.self.value.weight', 'uniter.encoder.layer.1.attention.self.value.bias', 'uniter.encoder.layer.1.attention.output.dense.weight', 'uniter.encoder.layer.1.attention.output.dense.bias', 'uniter.encoder.layer.1.attention.output.LayerNorm.weight', 'uniter.encoder.layer.1.attention.output.LayerNorm.bias', 'uniter.encoder.layer.1.intermediate.dense.weight', 'uniter.encoder.layer.1.intermediate.dense.bias', 'uniter.encoder.layer.1.output.dense.weight', 'uniter.encoder.layer.1.output.dense.bias', 'uniter.encoder.layer.1.output.LayerNorm.weight', 'uniter.encoder.layer.1.output.LayerNorm.bias', 'uniter.encoder.layer.2.attention.self.query.weight', 'uniter.encoder.layer.2.attention.self.query.bias', 'uniter.encoder.layer.2.attention.self.key.weight', 'uniter.encoder.layer.2.attention.self.key.bias', 'uniter.encoder.layer.2.attention.self.value.weight', 'uniter.encoder.layer.2.attention.self.value.bias', 'uniter.encoder.layer.2.attention.output.dense.weight', 'uniter.encoder.layer.2.attention.output.dense.bias', 'uniter.encoder.layer.2.attention.output.LayerNorm.weight', 'uniter.encoder.layer.2.attention.output.LayerNorm.bias', 'uniter.encoder.layer.2.intermediate.dense.weight', 'uniter.encoder.layer.2.intermediate.dense.bias', 'uniter.encoder.layer.2.output.dense.weight', 'uniter.encoder.layer.2.output.dense.bias', 'uniter.encoder.layer.2.output.LayerNorm.weight', 'uniter.encoder.layer.2.output.LayerNorm.bias', 'uniter.encoder.layer.3.attention.self.query.weight', 'uniter.encoder.layer.3.attention.self.query.bias', 'uniter.encoder.layer.3.attention.self.key.weight', 'uniter.encoder.layer.3.attention.self.key.bias', 'uniter.encoder.layer.3.attention.self.value.weight', 'uniter.encoder.layer.3.attention.self.value.bias', 'uniter.encoder.layer.3.attention.output.dense.weight', 'uniter.encoder.layer.3.attention.output.dense.bias', 'uniter.encoder.layer.3.attention.output.LayerNorm.weight', 'uniter.encoder.layer.3.attention.output.LayerNorm.bias', 'uniter.encoder.layer.3.intermediate.dense.weight', 'uniter.encoder.layer.3.intermediate.dense.bias', 'uniter.encoder.layer.3.output.dense.weight', 'uniter.encoder.layer.3.output.dense.bias', 'uniter.encoder.layer.3.output.LayerN[1,0]<stderr>:orm.weight', 'uniter.encoder.layer.3.output.LayerNorm.bias', 'uniter.encoder.layer.4.attention.self.query.weight', 'uniter.encoder.layer.4.attention.self.query.bias', 'uniter.encoder.layer.4.attention.self.key.weight', 'uniter.encoder.layer.4.attention.self.key.bias', 'uniter.encoder.layer.4.attention.self.value.weight', 'uniter.encoder.layer.4.attention.self.value.bias', 'uniter.encoder.layer.4.attention.output.dense.weight', 'uniter.encoder.layer.4.attention.output.dense.bias', 'uniter.encoder.layer.4.attention.output.LayerNorm.weight', 'uniter.encoder.layer.4.attention.output.LayerNorm.bias', 'uniter.encoder.layer.4.intermediate.dense.weight', 'uniter.encoder.layer.4.intermediate.dense.bias', 'uniter.encoder.layer.4.output.dense.weight', 'uniter.encoder.layer.4.output.dense.bias', 'uniter.encoder.layer.4.output.LayerNorm.weight', 'uniter.encoder.layer.4.output.LayerNorm.bias', 'uniter.encoder.layer.5.attention.self.query.weight', 'uniter.encoder.layer.5.attention.self.query.bias', 'uniter.encoder.layer.5.attention.self.key.weight', 'uniter.encoder.layer.5.attention.self.key.bias', 'uniter.encoder.layer.5.attention.self.value.weight', 'uniter.encoder.layer.5.attention.self.value.bias', 'uniter.encoder.layer.5.attention.output.dense.weight', 'uniter.encoder.layer.5.attention.output.dense.bias', 'uniter.encoder.layer.5.attention.output.LayerNorm.weight', 'uniter.encoder.layer.5.attention.output.LayerNorm.bias', 'uniter.encoder.layer.5.intermediate.dense.weight', 'uniter.encoder.layer.5.intermediate.dense.bias', 'uniter.encoder.layer.5.output.dense.weight', 'uniter.encoder.layer.5.output.dense.bias', 'uniter.encoder.layer.5.output.LayerNorm.weight', 'uniter.encoder.layer.5.output.LayerNorm.bias', 'uniter.encoder.layer.6.attention.self.query.weight', 'uniter.encoder.layer.6.attention.self.query.bias', 'uniter.encoder.layer.6.attention.self.key.weight', 'uniter.encoder.layer.6.attention.self.key.bias', 'uniter.encoder.layer.6.attention.self.value.weight', 'uniter.encoder.layer.6.attention.self.value.bias', 'uniter.encoder.layer.6.attention.output.dense.weight', 'uniter.encoder.layer.6.attention.output.dense.bias', 'uniter.encoder.layer.6.attention.output.LayerNorm.weight', 'uniter.encoder.layer.6.attention.output.LayerNorm.bias', 'uniter.encoder.layer.6.intermediate.dense.weight', 'uniter.encoder.layer.6.intermediate.dense.bias', 'uniter.encoder.layer.6.output.dense.weight', 'uniter.encoder.layer.6.output.dense.bias', 'uniter.encoder.layer.6.output.LayerNorm.weight', 'uniter.encoder.layer.6.output.LayerNorm.bias', 'uniter.encoder.layer.7.attention.self.query.weight', 'uniter.encoder.layer.7.attention.self.query.bias', 'uniter.encoder.layer.7.attention.self.key.weight', 'uniter.encoder.layer.7.attention.self.key.bias', 'uniter.encoder.layer.7.attention.self.value.weight', 'uniter.encoder.layer.7.attention.self.value.bias', 'uniter.encoder.layer.7.attention.output.dense.weight', 'uniter.encoder.layer.7.attention.output.dense.bias', 'uniter.encoder.layer.7.attention.output.LayerNorm.weight', 'uniter.encoder.layer.7.attention.output.LayerNorm.bias', 'uniter.encoder.layer.7.intermediate.dense.weight', 'uniter.encoder.layer.7.intermediate.dense.bias', 'uniter.encoder.layer.7.output.dense.weight', 'uniter.encoder.layer.7.output.dense.bias', 'uniter.encoder.layer.7.output.LayerNorm.weight', 'uniter.encoder.layer.7.output.LayerNorm.bias', 'uniter.encoder.layer.8.attention.self.query.weight', 'uniter.encoder.layer.8.attention.self.query.bias', 'uniter.encoder.layer.8.attention.self.key.weight', 'uniter.encoder.layer.8.attention.self.key.bias', 'uniter.encoder.layer.8.attention.self.value.weight', 'uniter.encoder.layer.8.attention.self.value.bias', 'uniter.encoder.layer.8.attention.output.dense.weight', 'uniter.encoder.layer.8.attention.output.dense.bias', 'uniter.encoder.layer.8.attention.output.LayerNorm.weight', 'uniter.encoder.layer.8.attention.output.LayerNorm.bias', 'uniter.encoder.layer.8.intermediate.dense.weight', 'uniter.encoder.layer.8.intermediate.dense.bias', 'uniter.encoder.layer.8.output.dense.weight', 'uniter.encoder.l[1,0]<stderr>:ayer.8.output.dense.bias', 'uniter.encoder.layer.8.output.LayerNorm.weight', 'uniter.encoder.layer.8.output.LayerNorm.bias', 'uniter.encoder.layer.9.attention.self.query.weight', 'uniter.encoder.layer.9.attention.self.query.bias', 'uniter.encoder.layer.9.attention.self.key.weight', 'uniter.encoder.layer.9.attention.self.key.bias', 'uniter.encoder.layer.9.attention.self.value.weight', 'uniter.encoder.layer.9.attention.self.value.bias', 'uniter.encoder.layer.9.attention.output.dense.weight', 'uniter.encoder.layer.9.attention.output.dense.bias', 'uniter.encoder.layer.9.attention.output.LayerNorm.weight', 'uniter.encoder.layer.9.attention.output.LayerNorm.bias', 'uniter.encoder.layer.9.intermediate.dense.weight', 'uniter.encoder.layer.9.intermediate.dense.bias', 'uniter.encoder.layer.9.output.dense.weight', 'uniter.encoder.layer.9.output.dense.bias', 'uniter.encoder.layer.9.output.LayerNorm.weight', 'uniter.encoder.layer.9.output.LayerNorm.bias', 'uniter.encoder.layer.10.attention.self.query.weight', 'uniter.encoder.layer.10.attention.self.query.bias', 'uniter.encoder.layer.10.attention.self.key.weight', 'uniter.encoder.layer.10.attention.self.key.bias', 'uniter.encoder.layer.10.attention.self.value.weight', 'uniter.encoder.layer.10.attention.self.value.bias', 'uniter.encoder.layer.10.attention.output.dense.weight', 'uniter.encoder.layer.10.attention.output.dense.bias', 'uniter.encoder.layer.10.attention.output.LayerNorm.weight', 'uniter.encoder.layer.10.attention.output.LayerNorm.bias', 'uniter.encoder.layer.10.intermediate.dense.weight', 'uniter.encoder.layer.10.intermediate.dense.bias', 'uniter.encoder.layer.10.output.dense.weight', 'uniter.encoder.layer.10.output.dense.bias', 'uniter.encoder.layer.10.output.LayerNorm.weight', 'uniter.encoder.layer.10.output.LayerNorm.bias', 'uniter.encoder.layer.11.attention.self.query.weight', 'uniter.encoder.layer.11.attention.self.query.bias', 'uniter.encoder.layer.11.attention.self.key.weight', 'uniter.encoder.layer.11.attention.self.key.bias', 'uniter.encoder.layer.11.attention.self.value.weight', 'uniter.encoder.layer.11.attention.self.value.bias', 'uniter.encoder.layer.11.attention.output.dense.weight', 'uniter.encoder.layer.11.attention.output.dense.bias', 'uniter.encoder.layer.11.attention.output.LayerNorm.weight', 'uniter.encoder.layer.11.attention.output.LayerNorm.bias', 'uniter.encoder.layer.11.intermediate.dense.weight', 'uniter.encoder.layer.11.intermediate.dense.bias', 'uniter.encoder.layer.11.output.dense.weight', 'uniter.encoder.layer.11.output.dense.bias', 'uniter.encoder.layer.11.output.LayerNorm.weight', 'uniter.encoder.layer.11.output.LayerNorm.bias', 'uniter.pooler.dense.weight', 'uniter.pooler.dense.bias', 'vcr_output.0.weight', 'vcr_output.0.bias', 'vcr_output.2.weight', 'vcr_output.2.bias', 'vcr_output.3.weight', 'vcr_output.3.bias']
[1,0]<stdout>:Unexpected_keys: ['cls.predictions.decoder.weight', 'cls.predictions.transform.LayerNorm.bias', 'region_classifier.net.2.weight', 'itm_output.bias', 'feat_regress.net.0.weight', 'feat_regress.net.2.weight', 'cls.predictions.transform.dense.bias', 'feat_regress.net.3.weight', 'feat_regress.net.2.bias', 'region_classifier.net.3.weight', 'feat_regress.net.3.bias', 'region_classifier.net.2.bias', 'itm_output.weight', 'feat_regress.net.0.bias', 'region_classifier.net.0.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.bias', 'cls.predictions.transform.LayerNorm.weight', 'region_classifier.net.3.bias', 'region_classifier.net.0.weight']
[1,0]<stdout>:Missing_keys: ['vcr_output.3.weight', 'vcr_output.3.bias', 'vcr_output.2.weight', 'vcr_output.2.bias', 'vcr_output.0.weight', 'uniter.img_embeddings.mask_embedding.weight', 'vcr_output.0.bias']
[1,1]<stdout>:Unexpected_keys: ['feat_regress.net.0.bias', 'region_classifier.net.0.weight', 'cls.predictions.decoder.weight', 'cls.predictions.transform.dense.weight', 'feat_regress.net.3.weight', 'region_classifier.net.3.bias', 'cls.predictions.transform.LayerNorm.weight', 'region_classifier.net.3.weight', 'itm_output.weight', 'region_classifier.net.0.bias', 'region_classifier.net.2.bias', 'cls.predictions.transform.dense.bias', 'feat_regress.net.2.weight', 'feat_regress.net.2.bias', 'feat_regress.net.3.bias', 'region_classifier.net.2.weight', 'feat_regress.net.0.weight', 'cls.predictions.transform.LayerNorm.bias', 'itm_output.bias', 'cls.predictions.bias']
[1,1]<stdout>:Missing_keys: ['vcr_output.3.weight', 'vcr_output.0.bias', 'vcr_output.3.bias', 'uniter.img_embeddings.mask_embedding.weight', 'vcr_output.2.weight', 'vcr_output.2.bias', 'vcr_output.0.weight']
[1,0]<stdout>:Selected optimization level O2:  FP16 training with FP32 batchnorm and FP32 master weights.
[1,0]<stdout>:
[1,0]<stdout>:Defaults for this optimization level are:
[1,0]<stdout>:enabled                : True
[1,0]<stdout>:opt_level              : O2
[1,0]<stdout>:cast_model_type        : torch.float16
[1,0]<stdout>:patch_torch_functions  : False
[1,0]<stdout>:keep_batchnorm_fp32    : True
[1,0]<stdout>:master_weights         : True
[1,0]<stdout>:loss_scale             : dynamic
[1,0]<stdout>:Processing user overrides (additional kwargs that are not None)...
[1,0]<stdout>:After processing overrides, optimization options are:
[1,0]<stdout>:enabled                : True
[1,0]<stdout>:opt_level              : O2
[1,0]<stdout>:cast_model_type        : torch.float16
[1,0]<stdout>:patch_torch_functions  : False
[1,0]<stdout>:keep_batchnorm_fp32    : True
[1,0]<stdout>:master_weights         : True
[1,0]<stdout>:loss_scale             : dynamic
[1,1]<stdout>:Selected optimization level O2:  FP16 training with FP32 batchnorm and FP32 master weights.
[1,1]<stdout>:
[1,1]<stdout>:Defaults for this optimization level are:
[1,1]<stdout>:enabled                : True
[1,1]<stdout>:opt_level              : O2
[1,1]<stdout>:cast_model_type        : torch.float16
[1,1]<stdout>:patch_torch_functions  : False
[1,1]<stdout>:keep_batchnorm_fp32    : True
[1,1]<stdout>:master_weights         : True
[1,1]<stdout>:loss_scale             : dynamic
[1,1]<stdout>:Processing user overrides (additional kwargs that are not None)...
[1,1]<stdout>:After processing overrides, optimization options are:
[1,1]<stdout>:enabled                : True
[1,1]<stdout>:opt_level              : O2
[1,1]<stdout>:cast_model_type        : torch.float16
[1,1]<stdout>:patch_torch_functions  : False
[1,1]<stdout>:keep_batchnorm_fp32    : True
[1,1]<stdout>:master_weights         : True
[1,1]<stdout>:loss_scale             : dynamic
[1,0]<stderr>:09/12/2020 19:01:09 - INFO - __main__ -   Waiting on git info....
[1,0]<stderr>:fatal: Not a git repository (or any parent up to mount point /src)
[1,0]<stderr>:Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
[1,0]<stderr>:09/12/2020 19:01:09 - INFO - __main__ -   Git branch: 
[1,0]<stderr>:fatal: Not a git repository (or any parent up to mount point /src)
[1,0]<stderr>:Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
[1,0]<stderr>:09/12/2020 19:01:09 - INFO - __main__ -   Git SHA: 
[1,0]<stderr>:fatal: Not a git repository (or any parent up to mount point /src)
[1,0]<stderr>:Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
[1,0]<stderr>:Traceback (most recent call last):
[1,0]<stderr>:  File "train_vcr.py", line 489, in <module>
[1,0]<stderr>:    main(args)
[1,0]<stderr>:  File "train_vcr.py", line 210, in main
[1,0]<stderr>:    save_training_meta(opts)
[1,0]<stderr>:  File "/src/utils/save.py", line 44, in save_training_meta
[1,0]<stderr>:    cwd=git_dir, universal_newlines=True).strip()
[1,0]<stderr>:  File "/opt/conda/lib/python3.6/subprocess.py", line 356, in check_output
[1,0]<stderr>:    **kwargs).stdout
[1,0]<stderr>:  File "/opt/conda/lib/python3.6/subprocess.py", line 438, in run
[1,0]<stderr>:    output=stdout, stderr=stderr)
[1,0]<stderr>:subprocess.CalledProcessError: Command '['git', 'status', '--short']' returned non-zero exit status 128.
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[41971,1],0]
  Exit code:    1
--------------------------------------------------------------------------

Do you know this error?

hi!
i try convert vcr dataset to itm task.
( i make data [CLS] question [SEP] answer [SEP] or [CLS] question [SEP] answer [SEP] rationale [SEP])
but i got error below.
the error start at "model/UniterTextEmbeddings".( maybe "words_embeddings = self.word_embeddings(input_ids)")
Have you seen this error?
i tried to solve this problem during three days, but i can't find why this error occured.

thank you :)

root@c3425fd1fd9c:/src# horovodrun -np 2 python pretrain.py --config config/pretrain-indomain-base-8gpu.json --output_dir /src/output
[1,0]<stderr>:09/30/2020 09:41:57 - INFO - __main__ -   device: cuda:0 n_gpu: 2, rank: 0, 16-bits training: True
[1,1]<stderr>:09/30/2020 09:41:57 - INFO - __main__ -   device: cuda:1 n_gpu: 2, rank: 1, 16-bits training: True
  0%|          | 0/200000 [00:00<?, ?it/s][1,0]<stderr>:09/30/2020 09:41:57 - INFO - __main__ -   Loading itm_vcr train dataset ['/txt/vcr_train.db/'], ['/img/vcr_gt_train/;/img/vcr_train/']
[1,0]<stderr>:09/30/2020 09:41:58 - INFO - __main__ -   425182 samples loaded
[1,0]<stderr>:09/30/2020 09:41:58 - INFO - __main__ -   Loading itm_vcr validation dataset, ['/txt/vcr_val.db/'], ['/img/vcr_gt_val/;/img/vcr_val/']
[1,0]<stderr>:09/30/2020 09:41:59 - INFO - __main__ -   52976 samples loaded
[1,1]<stderr>:09/30/2020 09:42:00 - INFO - model.model -   Model config {
[1,1]<stderr>:  "attention_probs_dropout_prob": 0.1,
[1,1]<stderr>:  "hidden_act": "gelu",
[1,1]<stderr>:  "hidden_dropout_prob": 0.1,
[1,1]<stderr>:  "hidden_size": 768,
[1,1]<stderr>:  "initializer_range": 0.02,
[1,1]<stderr>:  "intermediate_size": 3072,
[1,1]<stderr>:  "max_position_embeddings": 512,
[1,1]<stderr>:  "num_attention_heads": 12,
[1,1]<stderr>:  "num_hidden_layers": 12,
[1,1]<stderr>:  "type_vocab_size": 2,
[1,1]<stderr>:  "vocab_size": 28996
[1,1]<stderr>:}
[1,1]<stderr>:
[1,0]<stderr>:09/30/2020 09:42:00 - INFO - model.model -   Model config {
[1,0]<stderr>:  "attention_probs_dropout_prob": 0.1,
[1,0]<stderr>:  "hidden_act": "gelu",
[1,0]<stderr>:  "hidden_dropout_prob": 0.1,
[1,0]<stderr>:  "hidden_size": 768,
[1,0]<stderr>:  "initializer_range": 0.02,
[1,0]<stderr>:  "intermediate_size": 3072,
[1,0]<stderr>:  "max_position_embeddings": 512,
[1,0]<stderr>:  "num_attention_heads": 12,
[1,0]<stderr>:  "num_hidden_layers": 12,
[1,0]<stderr>:  "type_vocab_size": 2,
[1,0]<stderr>:  "vocab_size": 28996
[1,0]<stderr>:}
[1,0]<stderr>:
[1,1]<stderr>:09/30/2020 09:42:02 - INFO - model.model -   Weights of UniterForPretraining not initialized from pretrained model: ['uniter.img_embeddings.img_linear.weight', 'uniter.img_embeddings.img_linear.bias', 'uniter.img_embeddings.img_layer_norm.weight', 'uniter.img_embeddings.img_layer_norm.bias', 'uniter.img_embeddings.pos_layer_norm.weight', 'uniter.img_embeddings.pos_layer_norm.bias', 'uniter.img_embeddings.pos_linear.weight', 'uniter.img_embeddings.pos_linear.bias', 'uniter.img_embeddings.mask_embedding.weight', 'uniter.img_embeddings.LayerNorm.weight', 'uniter.img_embeddings.LayerNorm.bias', 'uniter.know_embeddins.position_embeddings.weight', 'uniter.know_embeddins.token_type_embeddings.weight', 'uniter.know_embeddins.LayerNorm.weight', 'uniter.know_embeddins.LayerNorm.bias', 'uniter.know_embeddins.gcn.gc1.weight', 'uniter.know_embeddins.gcn.gc2.weight', 'uniter.know_embeddins.gcn.gc3.weight', 'feat_regress.weight', 'feat_regress.bias', 'feat_regress.net.0.weight', 'feat_regress.net.0.bias', 'feat_regress.net.2.weight', 'feat_regress.net.2.bias', 'region_classifier.net.0.weight', 'region_classifier.net.0.bias', 'region_classifier.net.2.weight', 'region_classifier.net.2.bias', 'region_classifier.net.3.weight', 'region_classifier.net.3.bias', 'itm_output.weight', 'itm_output.bias']
[1,1]<stderr>:09/30/2020 09:42:02 - INFO - model.model -   Weights from pretrained model not used in UniterForPretraining: ['cls.seq_relationship.weight', 'cls.seq_relationship.bias']
[1,0]<stderr>:09/30/2020 09:42:03 - INFO - model.model -   Weights of UniterForPretraining not initialized from pretrained model: ['uniter.img_embeddings.img_linear.weight', 'uniter.img_embeddings.img_linear.bias', 'uniter.img_embeddings.img_layer_norm.weight', 'uniter.img_embeddings.img_layer_norm.bias', 'uniter.img_embeddings.pos_layer_norm.weight', 'uniter.img_embeddings.pos_layer_norm.bias', 'uniter.img_embeddings.pos_linear.weight', 'uniter.img_embeddings.pos_linear.bias', 'uniter.img_embeddings.mask_embedding.weight', 'uniter.img_embeddings.LayerNorm.weight', 'uniter.img_embeddings.LayerNorm.bias', 'uniter.know_embeddins.position_embeddings.weight', 'uniter.know_embeddins.token_type_embeddings.weight', 'uniter.know_embeddins.LayerNorm.weight', 'uniter.know_embeddins.LayerNorm.bias', 'uniter.know_embeddins.gcn.gc1.weight', 'uniter.know_embeddins.gcn.gc2.weight', 'uniter.know_embeddins.gcn.gc3.weight', 'feat_regress.weight', 'feat_regress.bias', 'feat_regress.net.0.weight', 'feat_regress.net.0.bias', 'feat_regress.net.2.weight', 'feat_regress.net.2.bias', 'region_classifier.net.0.weight', 'region_classifier.net.0.bias', 'region_classifier.net.2.weight', 'region_classifier.net.2.bias', 'region_classifier.net.3.weight', 'region_classifier.net.3.bias', 'itm_output.weight', 'itm_output.bias']
[1,0]<stderr>:09/30/2020 09:42:03 - INFO - model.model -   Weights from pretrained model not used in UniterForPretraining: ['cls.seq_relationship.weight', 'cls.seq_relationship.bias']
[1,0]<stdout>:Selected optimization level O2:  FP16 training with FP32 batchnorm and FP32 master weights.
[1,0]<stdout>:
[1,0]<stdout>:Defaults for this optimization level are:
[1,0]<stdout>:enabled                : True
[1,0]<stdout>:opt_level              : O2
[1,0]<stdout>:cast_model_type        : torch.float16
[1,0]<stdout>:patch_torch_functions  : False
[1,0]<stdout>:keep_batchnorm_fp32    : True
[1,0]<stdout>:master_weights         : True
[1,0]<stdout>:loss_scale             : dynamic
[1,0]<stdout>:Processing user overrides (additional kwargs that are not None)...
[1,0]<stdout>:After processing overrides, optimization options are:
[1,0]<stdout>:enabled                : True
[1,0]<stdout>:opt_level              : O2
[1,0]<stdout>:cast_model_type        : torch.float16
[1,0]<stdout>:patch_torch_functions  : False
[1,0]<stdout>:keep_batchnorm_fp32    : True
[1,0]<stdout>:master_weights         : True
[1,0]<stdout>:loss_scale             : dynamic
[1,1]<stdout>:Selected optimization level O2:  FP16 training with FP32 batchnorm and FP32 master weights.
[1,1]<stdout>:
[1,1]<stdout>:Defaults for this optimization level are:
[1,1]<stdout>:enabled                : True
[1,1]<stdout>:opt_level              : O2
[1,1]<stdout>:cast_model_type        : torch.float16
[1,1]<stdout>:patch_torch_functions  : False
[1,1]<stdout>:keep_batchnorm_fp32    : True
[1,1]<stdout>:master_weights         : True
[1,1]<stdout>:loss_scale             : dynamic
[1,1]<stdout>:Processing user overrides (additional kwargs that are not None)...
[1,1]<stdout>:After processing overrides, optimization options are:
[1,1]<stdout>:enabled                : True
[1,1]<stdout>:opt_level              : O2
[1,1]<stdout>:cast_model_type        : torch.float16
[1,1]<stdout>:patch_torch_functions  : False
[1,1]<stdout>:keep_batchnorm_fp32    : True
[1,1]<stdout>:master_weights         : True
[1,1]<stdout>:loss_scale             : dynamic
[1,0]<stderr>:09/30/2020 09:42:03 - INFO - __main__ -   ***** Running training with 2 GPUs *****
[1,0]<stderr>:09/30/2020 09:42:03 - INFO - __main__ -     Batch size = 10240
[1,0]<stderr>:09/30/2020 09:42:03 - INFO - __main__ -     Accumulate steps = 2
[1,0]<stderr>:09/30/2020 09:42:03 - INFO - __main__ -     Num steps = 200000
[1,1]<stdout>:#########################input_ids  torch.Size([24, 67])
[1,1]<stdout>:#########################position_ids  torch.Size([1, 67])
[1,1]<stdout>:#########################token_type_ids  torch.Size([24, 67])
[1,0]<stdout>:#########################input_ids  torch.Size([24, 65])
[1,0]<stdout>:#########################position_ids  torch.Size([1, 65])
[1,0]<stdout>:#########################token_type_ids  torch.Size([24, 65])
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [96,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [97,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [98,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [99,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191[1,1]<stderr>:,0,0], thread: [100,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [101,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [102,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [103,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [104,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [105,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [106,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [107,0,0] Assertion `srcIndex < srcSelectDimSize[1,1]<stderr>:` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [108,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [109,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [110,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [111,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [112,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0[1,1]<stderr>:,0], thread: [113,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [114,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [115,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [116,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [117,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362[1,1]<stderr>:: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [118,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [119,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [120,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [121,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [122,0,0] Assertion `srcIndex < srcSelectDimSize[1,1]<stderr>:` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [123,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [124,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [125,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [126,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362[1,1]<stderr>:: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [127,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [0,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu[1,1]<stderr>::362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [1,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [2,0,0] Assertion `srcIndex < srcSelectDimSize[1,1]<stderr>:` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [3,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [4,0,0] Assertion `srcIndex < srcSelectDimSize[1,1]<stderr>:` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [5,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [6,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [7,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [8,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0[1,1]<stderr>:], thread: [9,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [10,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [11,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0[1,1]<stderr>:], thread: [12,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [13,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [14,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [15,0,0[1,1]<stderr>:] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [16,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [17,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [18,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362[1,1]<stderr>:: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [19,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [20,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [21,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0[1,1]<stderr>:,0], thread: [22,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [23,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [24,0,0[1,1]<stderr>:] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [25,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191[1,1]<stderr>:,0,0], thread: [26,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [27,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0[1,1]<stderr>:,0], thread: [28,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [29,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0[1,1]<stderr>:,0], thread: [30,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [191,0,0], thread: [31,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,1]<stderr>:THCudaCheck FAIL file=/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCReduceAll.cuh line=321 error=710 : device-side assert triggered
[1,1]<stdout>:#########################words_embeddings  [1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [66,0,0], thread: [96,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [66,0,0], thread: [97,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [66,0,0], thread: [98,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [66,0,0], thread: [99,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [66,0,0], thread: [100,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [66,0,0], thread: [101,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu[1,0]<stderr>::362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [66,0,0], thread: [102,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [66,0,0], thread: [103,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [66,0,0], thread: [104,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [66,0,0], thread: [105,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [66,0,0], thread: [106,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [66,0,0], thread: [107,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [66,0,0], thread: [108,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [66,0,0], thread: [109,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [66,0,0], thread: [110,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [66,0,0], thread: [111[1,0]<stderr>:,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [66,0,0], thread: [112,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [66,0,0], thread: [113,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [66,0,0], thread: [114,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [66,0,0], thread: [115,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [66,0,0], thread: [116,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [66,0,0], thread: [117,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [66,0,0], thread: [118,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [66,0,0], thread: [119,0[1,0]<stderr>:,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [66,0,0], thread: [120,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [66,0,0], thread: [121,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [66,0,0], thread: [122,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [66,0,0], thread: [123,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [66,0,0], thread: [124,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [66,0,0], thread: [125,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [66,0,0], thread: [126,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [66,0,0], thread: [127,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [122,0,0], thread: [0,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [122,0,0], thread: [1,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [122,0,0], thread: [2,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [122,0,0], thread: [3,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362[1,0]<stderr>:: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [122,0,0], thread: [4,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [122,0,0], thread: [5,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [122,0,0], thread: [6,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [122,0,0], thread: [7,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [122,0,0], thread: [8,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [122,0,0], thread: [9,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [122,0,0], thread: [10,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [122,0,0], thread: [11,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [122,0,0], thread: [12,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [122,0,0], thread: [13,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [122,0,0], thread: [14,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [122,0,0], thread: [15,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [122,0,0], thread: [16,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [122,0,0], thread: [17[1,0]<stderr>:,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [122,0,0], thread: [18,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [122,0,0], thread: [19,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [122,0,0], thread: [20,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [122,0,0], thread: [21,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [122,0,0], thread: [22,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [122,0,0], thread: [23,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [122,0,0], thread: [24,0[1,0]<stderr>:,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [122,0,0], thread: [25,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [122,0,0], thread: [26,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [122,0,0], thread: [27,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [122,0,0], thread: [28,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [122,0,0], thread: [29,0,0[1,0]<stderr>:] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [122,0,0], thread: [30,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [122,0,0], thread: [31,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [64,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [65,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [66,0,0] Assertion `srcIndex < srcSelectDimSize[1,0]<stderr>:` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [67,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [68,0[1,0]<stderr>:,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [69,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0[1,0]<stderr>:], thread: [70,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [71,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [72,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [73,0,0] Assertion `srcIndex < srcSelectDimSize[1,0]<stderr>:` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [74,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [75,0,0[1,0]<stderr>:] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [76,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [77,0,0[1,0]<stderr>:] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [78,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [79,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362[1,0]<stderr>:: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [80,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [81,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [82,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16[1,0]<stderr>:,0,0], thread: [83,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [84,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [85,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [86,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [87,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [88,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [89,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [90,0,0[1,0]<stderr>:] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [91,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [92,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [93,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [94,0,0] Assertion `srcIndex < srcSelectDimSize[1,0]<stderr>:` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [95,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [32,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [33,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [34,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362[1,0]<stderr>:: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [35,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [36,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [37,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [38,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu[1,0]<stderr>::362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [39,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [40,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [41,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [42,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu[1,0]<stderr>::362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [43,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [44,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [45,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0[1,0]<stderr>:], thread: [46,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [47,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [48,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [49,0[1,0]<stderr>:,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [50,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [51,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0[1,0]<stderr>:], thread: [52,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [53,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [54,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [55,0,0[1,0]<stderr>:] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [56,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [57,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [58,0,0[1,0]<stderr>:] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [59,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [60,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [61,0,0[1,0]<stderr>:] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [62,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [63,0,0] Assertion `srcIndex < srcSelectDimSize[1,0]<stderr>:` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [96,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [97,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [98,0,0[1,0]<stderr>:] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [99,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [100,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [101,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [102,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0[1,0]<stderr>:,0], thread: [103,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [104,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [105,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [106,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [107,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu[1,0]<stderr>::362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [108,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [109,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362[1,0]<stderr>:: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [110,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [111,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362[1,0]<stderr>:: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [112,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [113,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu[1,0]<stderr>::362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [114,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [115,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362[1,0]<stderr>:: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [116,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [117,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16[1,0]<stderr>:,0,0], thread: [118,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [119,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [120,0,0] Assertion `srcIndex < srcSelectDimSize[1,0]<stderr>:` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [121,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [122,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu[1,0]<stderr>::362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [123,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [124,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true][1,0]<stderr>:: block: [16,0,0], thread: [125,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [126,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [127,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [0[1,0]<stderr>:,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [1,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [2,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362[1,0]<stderr>:: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [3,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [4,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [5,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [6,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [7,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [8,0,0[1,0]<stderr>:] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [9,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [10,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0[1,0]<stderr>:], thread: [11,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [12,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [13,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [14[1,0]<stderr>:,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [15,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [16,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [17[1,0]<stderr>:,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [18,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [19,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [20,0,0[1,0]<stderr>:] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [21,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [22,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true][1,0]<stderr>:: block: [16,0,0], thread: [23,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [24,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [25,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [26,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [27,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true][1,0]<stderr>:: block: [16,0,0], thread: [28,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [29,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [30,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [16,0,0], thread: [31,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [212,0,0], thread: [96,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [212,0,0], thread: [97,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [212,0,0], thread: [98[1,0]<stderr>:,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [212,0,0], thread: [99,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [212,0,0], thread: [100,0,0[1,0]<stderr>:] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [212,0,0], thread: [101,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [212,0,0], thread: [102,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [212,0,0], thread: [103[1,0]<stderr>:,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [212,0,0], thread: [104,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [212,0,0], thread: [105,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [212,0,0], thread: [106,0,0[1,0]<stderr>:] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [212,0,0], thread: [107,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [212,0,0], thread: [108,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [212,0,0], thread: [109,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [212[1,0]<stderr>:,0,0], thread: [110,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [212,0,0], thread: [111,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [212,0,0], thread: [112,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [212[1,0]<stderr>:,0,0], thread: [113,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [212,0,0], thread: [114,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [212,0,0], thread: [115,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu[1,0]<stderr>::362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [212,0,0], thread: [116,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [212,0,0], thread: [117,0,0] Assertion `srcIndex < srcSelectDimSize[1,0]<stderr>:` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [212,0,0], thread: [118,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362[1,0]<stderr>:: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [212,0,0], thread: [119,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [212,0,0[1,0]<stderr>:], thread: [120,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [212,0,0], thread: [121,0,0] Assertion `srcIndex < srcSelectDimSize[1,0]<stderr>:` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [212,0,0], thread: [122,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu[1,0]<stderr>::362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [212,0,0], thread: [123,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [212,0[1,0]<stderr>:,0], thread: [124,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [212,0,0], thread: [125,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [212,0,0], thread: [126,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362[1,0]<stderr>:: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [212,0,0], thread: [127,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:THCudaCheck FAIL file=/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCReduceAll.cuh line=321 error=710 : device-side assert triggered
[1,0]<stdout>:#########################words_embeddings  [1,0]<stderr>:Traceback (most recent call last):
[1,0]<stderr>:  File "pretrain.py", line 641, in <module>
[1,0]<stderr>:    main(args)
[1,0]<stderr>:  File "pretrain.py", line 268, in main
[1,0]<stderr>:    loss = model(batch, task=task, compute_loss=True)
[1,0]<stderr>:  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
[1,0]<stderr>:    result = self.forward(*input, **kwargs)
[1,0]<stderr>:  File "/opt/conda/lib/python3.6/site-packages/apex/amp/_initialize.py", line 177, in new_fwd
[1,0]<stderr>:    **applier(kwargs, input_caster))
[1,0]<stderr>:  File "/src/model/pretrain.py", line 99, in forward
[1,0]<stderr>:    targets, ot_inputs, compute_loss, txt_type_ids)
[1,0]<stderr>:  File "/src/model/pretrain.py", line 167, in forward_itm
[1,0]<stderr>:    output_all_encoded_layers=False, txt_type_ids=txt_type_ids)
[1,0]<stderr>:  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
[1,0]<stderr>:    result = self.forward(*input, **kwargs)
[1,0]<stderr>:  File "/src/model/model.py", line 420, in forward
[1,0]<stderr>:    input_ids, position_ids, txt_type_ids)
[1,0]<stderr>:  File "/src/model/model.py", line 355, in _compute_txt_embeddings
[1,0]<stderr>:    output = self.embeddings(input_ids, position_ids, txt_type_ids)
[1,0]<stderr>:  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
[1,0]<stderr>:    result = self.forward(*input, **kwargs)
[1,0]<stderr>:  File "/src/model/model.py", line 241, in forward
[1,0]<stderr>:    print('#########################words_embeddings ', words_embeddings)
[1,0]<stderr>:  File "/opt/conda/lib/python3.6/site-packages/torch/tensor.py", line 71, in __repr__
[1,0]<stderr>:    return torch._tensor_str._str(self)
[1,0]<stderr>:  File "/opt/conda/lib/python3.6/site-packages/torch/_tensor_str.py", line 283, in _str
[1,0]<stderr>:    tensor_str = _tensor_str(self, indent)
[1,0]<stderr>:  File "/opt/conda/lib/python3.6/site-packages/torch/_tensor_str.py", line 201, in _tensor_str
[1,0]<stderr>:    formatter = _Formatter(get_summarized_data(self) if summarize else self)
[1,0]<stderr>:  File "/opt/conda/lib/python3.6/site-packages/torch/_tensor_str.py", line 87, in __init__
[1,0]<stderr>:    nonzero_finite_vals = torch.masked_select(tensor_view, torch.isfinite(tensor_view) & tensor_view.ne(0))
[1,0]<stderr>:RuntimeError: cuda runtime error (710) : device-side assert triggered at /tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCReduceAll.cuh:321
[1,1]<stderr>:Traceback (most recent call last):
[1,1]<stderr>:  File "pretrain.py", line 641, in <module>
[1,1]<stderr>:    main(args)
[1,1]<stderr>:  File "pretrain.py", line 268, in main
[1,1]<stderr>:    loss = model(batch, task=task, compute_loss=True)
[1,1]<stderr>:  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
[1,1]<stderr>:    result = self.forward(*input, **kwargs)
[1,1]<stderr>:  File "/opt/conda/lib/python3.6/site-packages/apex/amp/_initialize.py", line 177, in new_fwd
[1,1]<stderr>:    **applier(kwargs, input_caster))
[1,1]<stderr>:  File "/src/model/pretrain.py", line 99, in forward
[1,1]<stderr>:    targets, ot_inputs, compute_loss, txt_type_ids)
[1,1]<stderr>:  File "/src/model/pretrain.py", line 167, in forward_itm
[1,1]<stderr>:    output_all_encoded_layers=False, txt_type_ids=txt_type_ids)
[1,1]<stderr>:  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
[1,1]<stderr>:    result = self.forward(*input, **kwargs)
[1,1]<stderr>:  File "/src/model/model.py", line 420, in forward
[1,1]<stderr>:    input_ids, position_ids, txt_type_ids)
[1,1]<stderr>:  File "/src/model/model.py", line 355, in _compute_txt_embeddings
[1,1]<stderr>:    output = self.embeddings(input_ids, position_ids, txt_type_ids)
[1,1]<stderr>:  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
[1,1]<stderr>:    result = self.forward(*input, **kwargs)
[1,1]<stderr>:  File "/src/model/model.py", line 241, in forward
[1,1]<stderr>:    print('#########################words_embeddings ', words_embeddings)
[1,1]<stderr>:  File "/opt/conda/lib/python3.6/site-packages/torch/tensor.py", line 71, in __repr__
[1,1]<stderr>:    return torch._tensor_str._str(self)
[1,1]<stderr>:  File "/opt/conda/lib/python3.6/site-packages/torch/_tensor_str.py", line 283, in _str
[1,1]<stderr>:    tensor_str = _tensor_str(self, indent)
[1,1]<stderr>:  File "/opt/conda/lib/python3.6/site-packages/torch/_tensor_str.py", line 201, in _tensor_str
[1,1]<stderr>:    formatter = _Formatter(get_summarized_data(self) if summarize else self)
[1,1]<stderr>:  File "/opt/conda/lib/python3.6/site-packages/torch/_tensor_str.py", line 87, in __init__
[1,1]<stderr>:    nonzero_finite_vals = torch.masked_select(tensor_view, torch.isfinite(tensor_view) & tensor_view.ne(0))
[1,1]<stderr>:RuntimeError: cuda runtime error (710) : device-side assert triggered at /tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCReduceAll.cuh:321
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [4,0,0], thread: [96,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [4,0,0], thread: [97,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [4,0,0], thread: [98,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [4,0,0], thread: [99,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [4,0,0], thread: [100,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true][1,0]<stderr>:: block: [4,0,0], thread: [101,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [4,0,0], thread: [102,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [4,0,0], thread: [103,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [4,0,0], thread: [104,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [4,0,0], thread: [105,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [4,0,0], thread: [106,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [4,0,0], thread: [107,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [4,0,0], thread: [108,0,0] Assertion `srcIndex < srcSelectDimSize[1,0]<stderr>:` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [4,0,0], thread: [109,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [4,0,0], thread: [110,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [4,0,0], thread: [111,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [4,0,0], thread: [112,0[1,0]<stderr>:,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [4,0,0], thread: [113,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [4,0,0], thread: [114,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [4,0[1,0]<stderr>:,0], thread: [115,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [4,0,0], thread: [116,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true][1,0]<stderr>:: block: [4,0,0], thread: [117,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [4[1,0]<stderr>:,0,0], thread: [118,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [4,0,0[1,0]<stderr>:], thread: [119,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [4,0,0], thread: [120[1,0]<stderr>:,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [4,0,0[1,0]<stderr>:], thread: [121,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [4,0,0[1,0]<stderr>:], thread: [122,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [4,0,0], thread: [123[1,0]<stderr>:,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [4,0,0], thread: [124,0[1,0]<stderr>:,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [4,0,0], thread: [125,0[1,0]<stderr>:,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [4,0,0], thread: [126,0[1,0]<stderr>:,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:/tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = c10::Half, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [4,0,0], thread: [127,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
[1,0]<stderr>:terminate called after throwing an instance of 'c10::Error'
[1,0]<stderr>:  what():  CUDA error: device-side assert triggered (insert_events at ../c10/cuda/CUDACachingAllocator.cpp:533)
[1,0]<stderr>:frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0x6a (0x7f4ad4e9693a in /opt/conda/lib/python3.6/site-packages/torch/lib/libc10.so)
[1,0]<stderr>:frame #1: <unknown function> + 0xcafc (0x7f4accdfcafc in /opt/conda/lib/python3.6/site-packages/torch/lib/libc10_cuda.so)
[1,0]<stderr>:frame #2: <unknown function> + 0x10291 (0x7f4acce00291 in /opt/conda/lib/python3.6/site-packages/torch/lib/libc10_cuda.so)
[1,0]<stderr>:frame #3: c10::TensorImpl::release_resources() + 0x61 (0x7f4ad4e893c1 in /opt/conda/lib/python3.6/site-packages/torch/lib/libc10.so)
[1,0]<stderr>:frame #4: torch::autograd::Variable::Impl::release_resources() + 0x5e (0x7f4ad436b1ee in /opt/conda/lib/python3.6/site-packages/torch/lib/libtorch.so.1)
[1,0]<stderr>:frame #5: <unknown function> + 0x136d7b (0x7f4b2adbbd7b in /opt/conda/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
[1,0]<stderr>:frame #6: <unknown function> + 0x3540b4 (0x7f4b2afd90b4 in /opt/conda/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
[1,0]<stderr>:frame #7: <unknown function> + 0x354111 (0x7f4b2afd9111 in /opt/conda/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
[1,0]<stderr>:<omitting python frames>
[1,0]<stderr>:frame #19: __libc_start_main + 0xf0 (0x7f4b39e92830 in /lib/x86_64-linux-gnu/libc.so.6)
[1,0]<stderr>:
[1,0]<stderr>:[c3425fd1fd9c:01577] *** Process received signal ***
[1,0]<stderr>:[c3425fd1fd9c:01577] Signal: Aborted (6)
[1,0]<stderr>:[c3425fd1fd9c:01577] Signal code:  (-6)
[1,0]<stderr>:[c3425fd1fd9c:01577] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f4b3a24d390]
[1,0]<stderr>:[c3425fd1fd9c:01577] [1,0]<stderr>:[ 1] [1,0]<stderr>:/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x38)[0x7f4b39ea7428]
[1,0]<stderr>:[c3425fd1fd9c:01577] [ 2] [1,0]<stderr>:/lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7f4b39ea902a]
[1,0]<stderr>:[c3425fd1fd9c:01577] [ 3] [1,0]<stderr>:/opt/conda/lib/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0xbc)[0x7f4ad4dcc3df]
[1,0]<stderr>:[c3425fd1fd9c:01577] [ 4] [1,0]<stderr>:/opt/conda/lib/libstdc++.so.6(+0x9cb16)[0x7f4ad4dcab16]
[1,0]<stderr>:[c3425fd1fd9c:01577] [ 5] [1,0]<stderr>:/opt/conda/lib/libstdc++.so.6(+0x9bf91)[0x7f4ad4dc9f91]
[1,0]<stderr>:[c3425fd1fd9c:01577] [ 6] [1,0]<stderr>:/opt/conda/lib/libstdc++.so.6(__gxx_personality_v0+0x33e)[0x7f4ad4dca79d]
[1,0]<stderr>:[c3425fd1fd9c:01577] [ 7] /opt/conda/bin/../lib/libgcc_s.so.1(+0xcf56)[0x7f4b39246f56]
[1,0]<stderr>:[c3425fd1fd9c:01577] [ 8] [1,0]<stderr>:/opt/conda/bin/../lib/libgcc_s.so.1(_Unwind_Resume+0x61)[0x7f4b392473e9]
[1,0]<stderr>:[c3425fd1fd9c:01577] [ 9] /opt/conda/lib/python3.6/site-packages/torch/lib/libc10_cuda.so(+0x106e6)[0x7f4acce006e6]
[1,0]<stderr>:[c3425fd1fd9c:01577] [10] [1,0]<stderr>:/opt/conda/lib/python3.6/site-packages/torch/lib/libc10.so(_ZN3c1010TensorImpl17release_resourcesEv+0x61)[0x7f4ad4e893c1]
[1,0]<stderr>:[c3425fd1fd9c:01577] [11] [1,0]<stderr>:/opt/conda/lib/python3.6/site-packages/torch/lib/libtorch.so.1(_ZN5torch8autograd8Variable4Impl17release_resourcesEv+0x5e)[0x7f4ad436b1ee]
[1,0]<stderr>:[c3425fd1fd9c:01577] [12] [1,0]<stderr>:/opt/conda/lib/python3.6/site-packages/torch/lib/libtorch_python.so(+0x136d7b)[0x7f4b2adbbd7b]
[1,0]<stderr>:[c3425fd1fd9c:01577] [13] [1,0]<stderr>:/opt/conda/lib/python3.6/site-packages/torch/lib/libtorch_python.so(+0x3540b4)[0x7f4b2afd90b4]
[1,0]<stderr>:[c3425fd1fd9c:01577] [14] [1,0]<stderr>:/opt/conda/lib/python3.6/site-packages/torch/lib/libtorch_python.so(+0x354111)[0x7f4b2afd9111]
[1,0]<stderr>:[c3425fd1fd9c:01577] [15] [1,0]<stderr>:python(+0x1993cf)[0x55953a7043cf]
[1,0]<stderr>:[c3425fd1fd9c:01577] [16] [1,0]<stderr>:python(+0xf18e8)[0x55953a65c8e8]
[1,0]<stderr>:[c3425fd1fd9c:01577] [17] [1,0]<stderr>:python(+0xf12b7)[0x55953a65c2b7]
[1,0]<stderr>:[c3425fd1fd9c:01577] [18] [1,0]<stderr>:python(+0xf1147)[0x55953a65c147]
[1,0]<stderr>:[c3425fd1fd9c:01577] [19] [1,0]<stderr>:python(+0xf115d)[0x55953a65c15d]
[1,0]<stderr>:[c3425fd1fd9c:01577] [20] [1,0]<stderr>:python(PyDict_SetItem+0x3da)[0x55953a6a1e7a]
[1,0]<stderr>:[c3425fd1fd9c:01577] [21] [1,0]<stderr>:python(PyDict_SetItemString+0x4f)[0x55953a6aa78f]
[1,0]<stderr>:[c3425fd1fd9c:01577] [22] [1,0]<stderr>:python(PyImport_Cleanup+0x99)[0x55953a70e709]
[1,0]<stderr>:[c3425fd1fd9c:01577] [23] [1,0]<stderr>:python(Py_FinalizeEx+0x61)[0x55953a77a5f1]
[1,0]<stderr>:[c3425fd1fd9c:01577] [24] [1,0]<stderr>:python(Py_Main+0x35e)[0x55953a7851fe]
[1,0]<stderr>:[c3425fd1fd9c:01577] [25] [1,0]<stderr>:python(main+0xee)[0x55953a64e02e]
[1,0]<stderr>:[c3425fd1fd9c:01577] [26] [1,0]<stderr>:/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7f4b39e92830]
[1,0]<stderr>:[c3425fd1fd9c:01577] [27] [1,0]<stderr>:python(+0x1c3e0e)[0x55953a72ee0e]
[1,0]<stderr>:[c3425fd1fd9c:01577] *** End of error message ***
[1,1]<stderr>:terminate called after throwing an instance of 'c10::Error'
[1,1]<stderr>:  what():  CUDA error: device-side assert triggered (insert_events at ../c10/cuda/CUDACachingAllocator.cpp:533)
[1,1]<stderr>:frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0x6a (0x7f6a835fe93a in /opt/conda/lib/python3.6/site-packages/torch/lib/libc10.so)
[1,1]<stderr>:frame #1: <unknown function> + 0xcafc (0x7f6a7b564afc in /opt/conda/lib/python3.6/site-packages/torch/lib/libc10_cuda.so)
[1,1]<stderr>:frame #2: <unknown function> + 0x10291 (0x7f6a7b568291 in /opt/conda/lib/python3.6/site-packages/torch/lib/libc10_cuda.so)
[1,1]<stderr>:frame #3: c10::TensorImpl::release_resources() + 0x61 (0x7f6a835f13c1 in /opt/conda/lib/python3.6/site-packages/torch/lib/libc10.so)
[1,1]<stderr>:frame #4: torch::autograd::Variable::Impl::release_resources() + 0x5e (0x7f6a82ad31ee in /opt/conda/lib/python3.6/site-packages/torch/lib/libtorch.so.1)
[1,1]<stderr>:frame #5: <unknown function> + 0x136d7b (0x7f6ad9523d7b in /opt/conda/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
[1,1]<stderr>:frame #6: <unknown function> + 0x3540b4 (0x7f6ad97410b4 in /opt/conda/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
[1,1]<stderr>:frame #7: <unknown function> + 0x354111 (0x7f6ad9741111 in /opt/conda/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
[1,1]<stderr>:<omitting python frames>
[1,1]<stderr>:frame #19: __libc_start_main + 0xf0 (0x7f6ae85fa830 in /lib/x86_64-linux-gnu/libc.so.6)
[1,1]<stderr>:
[1,1]<stderr>:[c3425fd1fd9c:01578] *** Process received signal ***
[1,1]<stderr>:[c3425fd1fd9c:01578] Signal: Aborted (6)
[1,1]<stderr>:[c3425fd1fd9c:01578] Signal code:  (-6)
[1,1]<stderr>:[c3425fd1fd9c:01578] [ 0] [1,1]<stderr>:/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f6ae89b5390]
[1,1]<stderr>:[c3425fd1fd9c:01578] [ 1] [1,1]<stderr>:/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x38)[0x7f6ae860f428]
[1,1]<stderr>:[c3425fd1fd9c:01578] [ 2] [1,1]<stderr>:/lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7f6ae861102a]
[1,1]<stderr>:[c3425fd1fd9c:01578] [ 3] [1,1]<stderr>:/opt/conda/lib/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0xbc)[0x7f6a835343df]
[1,1]<stderr>:[c3425fd1fd9c:01578] [ 4] [1,1]<stderr>:/opt/conda/lib/libstdc++.so.6(+0x9cb16)[0x7f6a83532b16]
[1,1]<stderr>:[c3425fd1fd9c:01578] [ 5] [1,1]<stderr>:/opt/conda/lib/libstdc++.so.6(+0x9bf91)[0x7f6a83531f91]
[1,1]<stderr>:[c3425fd1fd9c:01578] [ 6] [1,1]<stderr>:/opt/conda/lib/libstdc++.so.6(__gxx_personality_v0+0x33e)[0x7f6a8353279d]
[1,1]<stderr>:[c3425fd1fd9c:01578] [ 7] /opt/conda/bin/../lib/libgcc_s.so.1(+0xcf56)[0x7f6ae79aef56]
[1,1]<stderr>:[c3425fd1fd9c:01578] [1,1]<stderr>:[ 8] /opt/conda/bin/../lib/libgcc_s.so.1(_Unwind_Resume+0x61)[0x7f6ae79af3e9]
[1,1]<stderr>:[c3425fd1fd9c:01578] [ 9] /opt/conda/lib/python3.6/site-packages/torch/lib/libc10_cuda.so(+0x106e6)[0x7f6a7b5686e6]
[1,1]<stderr>:[c3425fd1fd9c:01578] [10] [1,1]<stderr>:/opt/conda/lib/python3.6/site-packages/torch/lib/libc10.so(_ZN3c1010TensorImpl17release_resourcesEv+0x61)[0x7f6a835f13c1]
[1,1]<stderr>:[c3425fd1fd9c:01578] [11] [1,1]<stderr>:/opt/conda/lib/python3.6/site-packages/torch/lib/libtorch.so.1(_ZN5torch8autograd8Variable4Impl17release_resourcesEv+0x5e)[0x7f6a82ad31ee]
[1,1]<stderr>:[c3425fd1fd9c:01578] [12] [1,1]<stderr>:/opt/conda/lib/python3.6/site-packages/torch/lib/libtorch_python.so(+0x136d7b)[0x7f6ad9523d7b]
[1,1]<stderr>:[c3425fd1fd9c:01578] [13] [1,1]<stderr>:/opt/conda/lib/python3.6/site-packages/torch/lib/libtorch_python.so(+0x3540b4)[0x7f6ad97410b4]
[1,1]<stderr>:[c3425fd1fd9c:01578] [14] [1,1]<stderr>:/opt/conda/lib/python3.6/site-packages/torch/lib/libtorch_python.so(+0x354111)[0x7f6ad9741111]
[1,1]<stderr>:[c3425fd1fd9c:01578] [15] [1,1]<stderr>:python(+0x1993cf)[0x55f6d02033cf]
[1,1]<stderr>:[c3425fd1fd9c:01578] [16] [1,1]<stderr>:python(+0xf18e8)[0x55f6d015b8e8]
[1,1]<stderr>:[c3425fd1fd9c:01578] [17] [1,1]<stderr>:python(+0xf12b7)[0x55f6d015b2b7]
[1,1]<stderr>:[c3425fd1fd9c:01578] [18] [1,1]<stderr>:python(+0xf1147)[0x55f6d015b147]
[1,1]<stderr>:[c3425fd1fd9c:01578] [19] [1,1]<stderr>:python(+0xf115d)[0x55f6d015b15d]
[1,1]<stderr>:[c3425fd1fd9c:01578] [20] [1,1]<stderr>:python(PyDict_SetItem+0x3da)[0x55f6d01a0e7a]
[1,1]<stderr>:[c3425fd1fd9c:01578] [21] [1,1]<stderr>:python(PyDict_SetItemString+0x4f)[0x55f6d01a978f]
[1,1]<stderr>:[c3425fd1fd9c:01578] [22] [1,1]<stderr>:python(PyImport_Cleanup+0x99)[0x55f6d020d709]
[1,1]<stderr>:[c3425fd1fd9c:01578] [23] [1,1]<stderr>:python(Py_FinalizeEx+0x61)[0x55f6d02795f1]
[1,1]<stderr>:[c3425fd1fd9c:01578] [24] [1,1]<stderr>:python(Py_Main+0x35e)[0x55f6d02841fe]
[1,1]<stderr>:[c3425fd1fd9c:01578] [25] [1,1]<stderr>:python(main+0xee)[0x55f6d014d02e]
[1,1]<stderr>:[c3425fd1fd9c:01578] [26] [1,1]<stderr>:/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7f6ae85fa830]
[1,1]<stderr>:[c3425fd1fd9c:01578] [27] [1,1]<stderr>:python(+0x1c3e0e)[0x55f6d022de0e]
[1,1]<stderr>:[c3425fd1fd9c:01578] *** End of error message ***
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 1 with PID 0 on node c3425fd1fd9c exited on signal 6 (Aborted).

Why use this restrict? How to do when batch size is small on my machine?

UNITER/data/sampler.py

Line 41 in 80d3602

if (max_len * (len(batch_indices) + self._size_mul)

[VCR task] Does horovodrun have to be installed?

can it train without horovodrun?

thank you!

TypeError: forward() got an unexpected keyword argument 'input_ids'

Hello I am just onboarding this repo and am stuck at the following step:

Run inference for the NLVR2 task and then evaluate.

inference

python inf_nlvr2.py --txt_db /txt/nlvr2_test1.db/ --img_db /img/nlvr2_test/
--train_dir /storage/nlvr-base/ --ckpt 6500 --output_dir . --fp16

.

I first got an hvd error that I resolved by adding a hvd.init() in the /data/data.py after importing hvd.

But now I get the error below. Can you please let me know what I could be doing incorrectly?

Traceback (most recent call last):
File "inf_nlvr2.py", line 138, in
main(args)
File "inf_nlvr2.py", line 71, in main
results = evaluate(model, eval_dataloader, device)
File "/opt/conda/lib/python3.6/site-packages/torch/autograd/grad_mode.py", line 43, in decorate_no_grad
return func(*args, **kwargs)
File "inf_nlvr2.py", line 92, in evaluate
scores = model(**batch, targets=None, compute_loss=False)
File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/opt/conda/lib/python3.6/site-packages/apex/amp/_initialize.py", line 177, in new_fwd
**applier(kwargs, input_caster))
TypeError: forward() got an unexpected keyword argument 'input_ids'

inf_vcr.py i got all score 0.00

i run inf_vcr.py for original model.
i got this result
Do you know what the problem is?

.....
[1,0]<stderr>:Stalled ranks:
[1,0]<stderr>:0: [allgather.noname.1]
 97%|█████████▋| 1222/1264 [18:55<00:35,  1.19it/s][1,0]<stderr>:[2020-09-22 17:25:14.668682: W horovod/common/operations.cc:779] One or more tensors were submitted to be reduced, gathered or broadcasted by subset of ranks and are waiting for remainder of ranks for more than 60 seconds. This may indicate that different ranks are trying to submit different tensors or that only subset of ranks is submitting tensors, which will cause deadlock. 
[1,0]<stderr>:Stalled ranks:
[1,0]<stderr>:0: [allgather.noname.1]
100%|██████████| 1264/1264 [19:28<00:00,  1.76it/s][1,0]<stderr>:09/22/2020 17:25:47 - INFO - __main__ -   **evaluation finished in 1168 seconds, score_qa: 0.00 score_qar: 0.00 score: 0.00**

The download data of running code

Hello,thanks for your code：
when I run the second step"Launch the Docker container for running the experiments",it downloads some image slowly,how can I download it locally?thanks again！

Pretraining Process

Thank you for your great work.
Could you please provide the codes of the pretraining process in detail? Specifically, the MLM, MRM, and ITM.

Could you provide a smaller sample for easier reference to the expected format for pretraining?

A great work!

I find it is difficult to prepare the pretraining data for the code. Could you please release a sample of pretraining data? It would be better if you could provide the code or procedures for data pre-processing.

Thanks a lot!

The format of pre-training data

Thanks for your great work！I wanna ask :
In the pre-training of the second stage of the VCR task, what is the format of pre-training data？
Thanks again!

The code on the VCR task.

Thanks for your great work！
Could you please share the code on VCR task including the pre-training code on VCR ？
Thanks again！

"Warning: NaN or Inf found in input tensor" and "Gradient overflow. Skipping step, loss ..."

i got this during vcr training.
("Warning: NaN or Inf found in input tensor" and "Gradient overflow. Skipping step, loss scaler 0 reducing loss scale to 32768.0")
Is the problem caused by hardware?

thank you!

[1,1]<stdout>:Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 32768.0
[1,0]<stdout>:Warning: NaN or Inf found in input tensor.
[1,0]<stdout>:Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 32768.0
  0%|          | 1/8000 [00:02<5:51:34,  2.64s/it][1,0]<stdout>:Warning: NaN or Inf found in input tensor.
[1,0]<stdout>:Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 16384.0
  0%|          | 2/8000 [00:03<4:48:42,  2.17s/it][1,1]<stdout>:Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 16384.0
  1%|▏         | 100/8000 [02:22<3:09:47,  1.44s/it][1,0]<stderr>:09/13/2020 04:37:50 - INFO - __main__ -   ============Step 100=============
[1,0]<stderr>:09/13/2020 04:37:50 - INFO - __main__ -   32000 examples trained at 225 ex/s
[1,0]<stderr>:09/13/2020 04:37:50 - INFO - __main__ -   ===========================================
  2%|▎         | 200/8000 [04:52<3:25:59,  1.58s/it][1,0]<stderr>:09/13/2020 04:40:21 - INFO - __main__ -   ============Step 200=============
[1,0]<stderr>:09/13/2020 04:40:21 - INFO - __main__ -   64000 examples trained at 218 ex/s
[1,0]<stderr>:09/13/2020 04:40:21 - INFO - __main__ -   ===========================================
  4%|▎         | 299/8000 [07:46<3:55:29,  1.83s/it][1,1]<stdout>:Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 8192.0
[1,0]<stdout>:Warning: NaN or Inf found in input tensor.
[1,0]<stdout>:Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 8192.0
  4%|▍         | 300/8000 [07:48<4:01:47,  1.88s/it][1,0]<stderr>:09/13/2020 04:43:17 - INFO - __main__ -   ============Step 300=============
[1,0]<stderr>:09/13/2020 04:43:17 - INFO - __main__ -   96000 examples trained at 204 ex/s
[1,0]<stderr>:09/13/2020 04:43:17 - INFO - __main__ -   ===========================================
  5%|▌         | 400/8000 [10:57<4:05:42,  1.94s/it][1,0]<stderr>:09/13/2020 04:46:26 - INFO - __main__ -   ============Step 400=============
[1,0]<stderr>:09/13/2020 04:46:26 - INFO - __main__ -   128000 examples trained at 194 ex/s
[1,0]<stderr>:09/13/2020 04:46:26 - INFO - __main__ -   ===========================================

Image-text retrieval results can't be reproduced

Thanks for your contribution!
I run your code and can not reproduce the performance in your paper.
There my results in two different setting：
Finetuning Image-text retrieval with "train-itm-flickr-base-8gpu.json"

Finetuning Image-text retrieval with"train-itm-flickr-base-16gpu-hn.json"

There is a gap between the results and those in your paper. What's the difference between the experiment you do and the code you released? For example, training steps and learning rate. Thanks!

ailab@ailab:~/UNITER$ bash scripts/download_vcr.sh /media/ailab/jaeyun/UNITER
--2020-09-11 02:25:19--  https://convaisharables.blob.core.windows.net/uniter/txt_db/vcr_train.db.tar
Resolving convaisharables.blob.core.windows.net (convaisharables.blob.core.windows.net)... 13.77.184.64
Connecting to convaisharables.blob.core.windows.net (convaisharables.blob.core.windows.net)|13.77.184.64|:443... connected.
HTTP request sent, awaiting response... 404 The specified blob does not exist.
2020-09-11 02:25:20 ERROR 404: The specified blob does not exist..

tar: /media/ailab/jaeyun/UNITER/txt_db/vcr_train.db.tar: Cannot open: No such file or directory
tar: Error is not recoverable: exiting now
--2020-09-11 02:25:20--  https://convaisharables.blob.core.windows.net/uniter/txt_db/vcr_val.db.tar
Resolving convaisharables.blob.core.windows.net (convaisharables.blob.core.windows.net)... 13.77.184.64
Connecting to convaisharables.blob.core.windows.net (convaisharables.blob.core.windows.net)|13.77.184.64|:443... connected.
HTTP request sent, awaiting response... 404 The specified blob does not exist.
2020-09-11 02:25:20 ERROR 404: The specified blob does not exist..

tar: /media/ailab/jaeyun/UNITER/txt_db/vcr_val.db.tar: Cannot open: No such file or directory
tar: Error is not recoverable: exiting now
--2020-09-11 02:25:20--  https://convaisharables.blob.core.windows.net/uniter/txt_db/vcr_test.db.tar
Resolving convaisharables.blob.core.windows.net (convaisharables.blob.core.windows.net)... 13.77.184.64
Connecting to convaisharables.blob.core.windows.net (convaisharables.blob.core.windows.net)|13.77.184.64|:443... connected.
HTTP request sent, awaiting response... 404 The specified blob does not exist.
2020-09-11 02:25:21 ERROR 404: The specified blob does not exist..

tar: /media/ailab/jaeyun/UNITER/txt_db/vcr_test.db.tar: Cannot open: No such file or directory
tar: Error is not recoverable: exiting now

Do you have a plan for releasing a version w/o docker requirement?

Hi there,

Thank you for sharing of this great repo. Do you have a plan to release a version without docker requirement?

Best,

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.

	embedding_output = torch.gather(torch.cat([txt_emb, img_emb], dim=1),
	dim=1, index=gather_index)