The training loss is a constant from beginning about transformer HOT 5 CLOSED

Hi, having a constant loss throughout the training can be a real headache, let's try a couple of things:

• What is the exact shape of a batch returned by your dataloader ?
• Have you properly normalized your data ?
• Could you try using one of the LSTM/GRU in the benchmark to see if the problem persists ?

You should keep the MSELoss for now, the OZELoss is proper to my use case and may not fit your problem anyway.

from transformer.

Hi, having a constant loss throughout the training can be a real headache, let's try a couple of things:

• What is the exact shape of a batch returned by your dataloader ?
• Have you properly normalized your data ?
• Could you try using one of the LSTM/GRU in the benchmark to see if the problem persists ?

You should keep the MSELoss for now, the OZELoss is proper to my use case and may not fit your problem anyway.

Thanks for your reply.
Firstly, i used LSTM which was same with LSTM in benchmark,and the results were good.So i want to know the results
with Transformer,
Secondly, i have used MinMax and log normalization,and the log normalization worked better on LSTM,so i used it.
Thirdly, shape returned by dataloader is [batch_size, time_step, input_size]
So should I adjust the other parameters,or the transformer is just not fit with my data?

from transformer.

• LSTM working properly is not a good sign at all
• Either should be fine
• Which is the correct shape.

One thing you could try is reducing the number of layers of the Transformer. Firstly, because smaller models are usually easier to converge, secondly so it's easier to pinpoint at which point does the gradient vanishes, or explodes (if that is indeed the reason for the model being stuck).

from transformer.

• LSTM working properly is not a good sign at all
• Either should be fine
• Which is the correct shape.

One thing you could try is reducing the number of layers of the Transformer. Firstly, because smaller models are usually easier to converge, secondly so it's easier to pinpoint at which point does the gradient vanishes, or explodes (if that is indeed the reason for the model being stuck).

Thanks,
I deleted part of layernorms and residual connections in sub-layer, and then the loss decreased normally.Besides,the performance in Transformer is a bit better than LSTM on my data.

from transformer.

It's funny how these layers you removed supposedly improved convergence on all networks. I guess the transformer architecture still isn't that well understood in that regard.

If you want to get to the bottom of this convergence issue, you could now try plotting attention maps. You could also add these layers back, one at a time, to isolate the problem.

from transformer.