jinhan / tacotron2-gst Goto Github PK

Traceback (most recent call last):

  File "train.py", line 285, in <module>

    args.warm_start, args.n_gpus, args.rank, args.group_name, hparams)

  File "train.py", line 222, in train

    optimizer.backward(loss)
    
  File "/projects/playground/tacotron2-gst/fp16_optimizer.py", line 362, in backward

    self.loss_scaler.backward(loss.float())
    
  File "/projects/playground/tacotron2-gst/loss_scaler.py", line 80, in backward

    scaled_loss.backward()
    
  File "/pyenvs/taco/lib/python3.6/site-packages/torch/tensor.py", line 118, in backward

    torch.autograd.backward(self, gradient, retain_graph, create_graph)
    
  File "/pyenvs/taco/lib/python3.6/site-packages/torch/autograd/__init__.py", line 93, in backward

    allow_unreachable=True)  # allow_unreachable flag

  File "/projects/playground/tacotron2-gst/distributed.py", line 157, in allreduce_params

    coalesced = _flatten_dense_tensors(grads)
    
  File "/projects/playground/tacotron2-gst/distributed.py", line 19, in _flatten_dense_tensors
    
    flat = torch.cat([t.contiguous().view(-1) for t in tensors], dim=0)

RuntimeError: Expected object of scalar type Half but got scalar type Float for sequence element 3 in sequence argument at position #1 'tensors'

    ```

/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [119,0,0], thread: [95,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
Traceback (most recent call last):
  File "train.py", line 285, in <module>
    args.warm_start, args.n_gpus, args.rank, args.group_name, hparams)
  File "train.py", line 213, in train
    y_pred = model(x)
  File "/home/rishikesh/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/rishikesh/Dev/tacotron2-gst/model.py", line 516, in forward
    transcript_outputs = self.encoder(embedded_inputs, input_lengths)
  File "/home/rishikesh/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/rishikesh/Dev/tacotron2-gst/model.py", line 177, in forward
    x = F.dropout(F.relu(conv(x)), drop_rate, self.training)
  File "/home/rishikesh/.local/lib/python3.6/site-packages/torch/nn/functional.py", line 749, in dropout
    else _VF.dropout(input, p, training))
RuntimeError: Creating MTGP constants failed. at /pytorch/aten/src/THC/THCTensorRandom.cu:35

python : 3.6
OS: Ubuntu 18.04
Pytorch : 1.0
tensorflow : 1.6