prediction problems about transformer HOT 3 CLOSED

As I said, in order to adapt this repo to your problem you could start by:

* I would start, as you suggested, by releasing the equality condition on the input and output sequences. This can also be achieved by adding an output embedding layer after the last decoder.

* The auto regressive nature of the Transformer becomes essential, so make sure to mask subsequent predictions in the decoder.

* Most alternative implementations aiming at reducing the quadratic complexity, such as `ChunkMHA` or `WindowMHA` in this repo, should be avoided, as the model is no longer coherent.

from transformer.

Hi, I understand the problem you are trying to solve. I fixed K to be equal in both input and output shapes in order to avoid having to deal with predictions problem, as neither me nor this repo are qualified to address these problems. All the modifications of the original transformer that I implemented are made for many to many coherent time series problems, when the prediction of time step k depends mostly on input time steps k:k-Δ.

That being said, it doesn't mean that the Transformer can't be adapted for predictions problems, but it will require some modifications.

• I would start, as you suggested, by releasing the equality condition on the input and output sequences. This can also be achieved by adding an output embedding layer after the last decoder.
• The auto regressive nature of the Transformer becomes essential, so make sure to mask subsequent predictions in the decoder.
• Most alternative implementations aiming at reducing the quadratic complexity, such as ChunkMHA or WindowMHA in this repo, should be avoided, as the model is no longer coherent.

from transformer.

thank for your answer. Though your repo is not applicable for the prediction problems. Then how should I modificate the model to apply to my problem. I have changed the embedding layer and replaced it by Linear layer, also, I changed the output layer with the sigmoid layer. What elso should I do to apply the transformer model to my problem. What's more,

I fixed K to be equal in both input and output shapes in order to avoid having to deal with predictions problem, as neither me nor this repo are qualified to address these problems
I can't quite make out what you mean. Wheather your repo can solve the prediction problem. If no, why?

from transformer.