[Feature Request] Add a Transformer about srs-benchmark HOT 15 CLOSED

just a reminder to start working on this

I have. But it's too slow and run out of my CPU.

from srs-benchmark.

Transformer has many different implementations and hyperparameters. And it's mainly used for NLP, instead of common time-series prediction. I don't know what's the best practice for our case.

from srs-benchmark.

How about a bidirectional LSTM? According to pytorch documentation, it's very easy to do, there is a hyperparameter for that.

from srs-benchmark.

I try to add the code of transformer: 4801583

But the training is very slow. Maybe I will train the transformer in the next month.

from srs-benchmark.

Btw, to make the results comparable ensure that the transformer has roughly the same number of parameters as the LSTM.

from srs-benchmark.

It only has 492 parameters.

from srs-benchmark.

Yes. So the transformer should have around 500. If the transformer has much more parameters, it will be impossible to tell whether RMSE is different because there are more parameters or because the architecture is different. By making the number of parameters more or less the same, we can see how much the architecture matters.

from srs-benchmark.

I mean my current implementation of transformer has 492 parameters. But I forget how many parameters the LSTM has.

from srs-benchmark.

from srs-benchmark.

OK. Now their sizes of parameters are very close.

from srs-benchmark.

@L-M-Sherlock just a reminder to start working on this

from srs-benchmark.

Progress: 50%

It's worse than LSTM. It's not surprised. Here is a related work: Are Transformers Effective for Time Series Forecasting?

from srs-benchmark.

Interesting, I'll take a look at the article later.

from srs-benchmark.

What is this?

from srs-benchmark.

What is this?

It sets the size at 1 for each collection.

from srs-benchmark.