MSSE definition incosistent with Hyndman and Koehler 2005 about hierarchicalforecast HOT 6 CLOSED

Thanks for the comment,
I will incorporate into the library renaming MSSE into RelMSE.

from hierarchicalforecast.

Sorry, RMSSE formula rendered incorrectly, here is a screenshot from the article

from hierarchicalforecast.

Hey @nickto,

Thanks for the comment.

I think the RMSSE equation is consistent with the definition of HierarchicalForecast's MSSE because $\hat{y}^{naive}=y_{t-1}$ is the Naive forecast that is scaling, and that is relativizing the error measure.

The MSSE satisfies both being scaled and being a relative metric simultaneously.

from hierarchicalforecast.

Hi @kdgutier ,

I think the inconsistency is only in the period on which the scaling factor is computed.

Hyndman and Koehler (2006) and Makridakis et al. (2002) compute the scaling factor on the training (in-sample) data data only.

However, in HierarchicalForecast the scaling factor is computed on the out-of-sample data:

y_naive = np.repeat(y_train[:,[-1]], horizon, axis=1) # rolling naive forecast for the whole forecasting horizon
norm = mse(y=y, y_hat=y_naive)                        # compare to true out-of-sample values

If I understand correctly, this metric would have been called "relative MSE", according to Hyndman and Koehler (2006). Relative MSE is the ratio between our model MSE and some benchmark MSE (rolling naive forecast in this case).

In order to make msse() more consistent with Hyndman and Koehler (2006) and Makridakis et al. (2022), I guess it should be something along the lines of

y_naive = y_train[:, :-1]                 # make 1 step ahead naive forecast of the in-sample values
y_naive_true = y_train[:, 1:]             # in-sample true values
norm = mse(y=y_naive_true, y_hat=y_naive) # compare to true in-sample values

If my reasoning is wrong (which can very well be), I would be thankful if you could point out what I have overlooked or misunderstood.

from hierarchicalforecast.

@nickto
Would you suggest that going beyond a forecast of $t+1$ would merit to change the scale nomenclature into a relative nomenclature? But with a single step ahead forecast it is scaled?

Is that the difference?

from hierarchicalforecast.

@kdgutier, I tried creating a PR-185, but not entirely sure if it works with your PR. But hope it could be useful :)

Also, thank you for such a an incredibly quick feedback!

from hierarchicalforecast.