Please ensure proper citation practice and proper assignment of credit in your academic papers about hierarchicalforecast HOT 14 CLOSED

@fkiraly, we updated the README and included a section on open-source libraries in the HierarchicalForecast: A Reference Framework for Hierarchical Forecasting in Python. We think this closes the issue, but feel free to comment if something is missing.

Inspired by your comment and in the spirit of advancing the field, we take you up on your offer and will ask you some further questions regarding best scientific practice:

1. We did not find any validations or experiments related to your hts re-implementation. We were curious if that aligns with the best scientific practice and, furthermore, with the recommendations of your own paper, which encourages replicability and demonstrations of effectiveness.

2. Given that we weren’t able to find validations for the hts methods, we did some experiments ourselves; the comparison was incomplete due to sktime’s exclusion of TopDown, MiddleOut, and ERM reconciliation strategies. For the methods that we could compare, and as shown in the results table, we found they might be some problems in your implementation.

2.1 We believe it is best scientific practice (or at least professional courtesy) to double-check with the specialized colleagues involved before assuming some malpractice or being condescending. In that spirit, we would like to invite you to help us identify possible mistakes we made in our experiments. We hope we did something wrong, but in the unlikely absence of experimental errors or mistakes from our side, we think it would be appropriate to fix the issue that might be (silently) affecting thousands of sktime users. As you can see in the table, the errors are particularly big for some datasets.

3. One final note on proper citation practice and proper assignment of credit, after reading your pydata Berlin presentation, it seems that you might also have omitted some references in your notebook. Wouldn’t it be convenient to include further citations besides mentioning Hyndman’s book? To complement your notebook, feel free to look at our literature review.

from hierarchicalforecast.

Thanks! Also great package! I will be using it a lot.

from hierarchicalforecast.

In the spirit of transparency and collaboration,@alexhallam also pointed out the README section didn't include a proper References section. (BTW: congrats on tablespoon)

We have updated the README accordingly and have also submitted a vaster lit review in the paper. @fkiraly, we will let you know once the changes are publicly available in case you have any further observations.

Furthermore, this thread inspired us to make an empirical comparison of (some of) the available implementations, and we will be publishing the results of the experiments in the following days.

from hierarchicalforecast.

Thank you very much for your kind comments, @fkiraly.

We are still working on the details of the papers, and your comments are highly welcomed. We will include the proper citations to sktime, R, Darts, and to the valuable contributions that @carlomazzaferro and @AngelPone have made.

If you find any further omissions, we are more than happy to amend them in the spirit of best scientific practice.

from hierarchicalforecast.

@ciaran-g, thanks for the input we will try to make that clearer.

from hierarchicalforecast.

@mergenthaler, apologies, I forgot about this thread. Let me answer one by one.

We did not find any validations or experiments related to your hts re-implementation. We were curious if that aligns with the best scientific practice and, furthermore, with the recommendations of your own paper, which encourages replicability and demonstrations of effectiveness.

The way this aligns with best scientific practice:

• sktime have not published a paper on the hts functionality, there are no claims of correctness made. You are right, in-principle you would want to write a paper of the kind eventually, and the burden of proof is with sktime authors.
• implementation is by an expert in the field, @ciaran-g. That is lowest level of empirical evidence (D), your level of evidence is higher and justifies writing a paper.

from hierarchicalforecast.

We believe it is best scientific practice (or at least professional courtesy) to double-check with the specialized colleagues involved before assuming some malpractice or being condescending.

There are multiple things here that you are mixing up:

• bad citation practice -> nixtla
• violation of best open source collaboration practice -> nixtla
• honest scientific errors -> sktime

sktime might be accused of honest scientific errors, e.g., a mistake in the implementation. I don't think that is ethically problematic though, as long as these are honestly addressed and discussed.

Ethical issues though surround what I think you were doing:

• bad citation practice against better knowledge, not acknowledging properly people who are working on the same thing earlier. Ideas of implementing a methon in an open source package, interface designs, etc, are valuable in itself. You can't just ignore prior work in the software space, even if the implementation is faulty.

• sktime is an openly governed open source community, while nixtla is commercially controlled. You chose to reimplement hierarchical method and then position them in social media etc as "first of their kind" and the best thing since sliced bread, instead of participating in the community effort. Even though there is nothing wrong with it, it violates the communitarian/collaborative expectations of the open source field, because you are trying to take ownership and put it under the soft control of a company, instead of working towards a common good together.

from hierarchicalforecast.

One final note on proper citation practice and proper assignment of credit, after reading your pydata Berlin presentation, it seems that you might also have omitted some references in your notebook. Wouldn’t it be convenient to include further citations besides mentioning Hyndman’s book?

The notebook is not a primary academic publication, it is a software tutorial.

For a software tutorial on methodology, I think a textbook reference is enough, or a reference to pertinent academic paper if applicable. But we didn't write an academic paper, and (as far as I remember) didn't make any scientific claims about our implementation being special or new, except that it's there and this is how you use it.

If you require a full review on all methods, we would have had write basically multiple chapters from Hyndman's book in an updated form to accompany the tutorial, which seems absurd.

Conversely, if you write a paper on state-of-art in implementing hierarchical methods - like you did - you need to do your homework.

from hierarchicalforecast.

PS: speaking about collaborative open source expectations: if you suspect an error in sktime reconcilers, it would be great if you would open a bug report on sktime and ping the author. Or, help to diagnose and fix.

For instance, might the issues in your experiment with sktime be rooted in your use of a base estimator with a known bug: sktime/sktime#3162 (not mentioned in the table) rather than the reconciliation methods themselves?

from hierarchicalforecast.

Hi @fkiraly, thanks for taking the time to write such a complete response.

As mentioned three months ago, we have included all the proper citations in the README and the hierarchical forecasting work-in-progress paper. If you find anything missing, please let us know.

Regarding the rest of this thread: we feel your tone and intent are not aiming for improvement or collaboration. We appreciate your opinions, but we simply disagree with them.

Fortunately enough, neither you nor we are in the position to declare a final verdict on the other issues discussed:
Luckily the eventual publication and validity of the paper will be decided by a peer of experts, and whether or not we contribute to the python time series community and satisfy their expectations will also not be defined by you but by the community itself. (If you still have a problem with the current status of the wip paper being on Arvix, you can alert them.)

We hope you have a great fall developer's day!

from hierarchicalforecast.

Hi, I know you have closed this issue but I'm just posting my thoughts on it all. I'm also looking forward trying out the probabilistic coherent methods you have implemented - v. nice! 👍

While I was implementing the hierarchical reconcilers earlier this year, I was informally checking that it was consistent with fable via a notebook and some synthetic data. Having a look at your own experiments I decided to clean it up a bit and expand it to use the labour dataset from your case study as well. I used a simple but equivalent AR forecaster for both sktime and fable, and you can see the results here.

The results confirm that the difference between fable and sktime in your experiments linked here, in terms of forecast accuracy, are down to problems with the base forecaster (as has been discussed above and reported), at least for the labour dataset. I think that it would be fair to make that clear. Also, I see you plan to include the same base forecasts for all comparisons in the TODO section, and I think this is probably the best way to compare only the hierarchical implementation of the different libraries.

Hope this is helpful and thanks for uploading the datasets and your experiments

from hierarchicalforecast.

Hi @ciaran-g

Do you know what was the problem with sktime's previous base forecasts?
Thanks for sharing such a careful study.

from hierarchicalforecast.

@ciaran-g, the disclaimer, and links to your experiment have been included here. Is there anything else we should note or include?

from hierarchicalforecast.

Do you know what was the problem with sktime's previous base forecasts?

I think that issue is still active @kdgutier unfortunately and it's more to do with the statsmodels forecaster, but having a look here it seems like setting some starting parameters for the model might be a quick way forward

Is there anything else we should note or include?

Looks good to me @mergenthaler, thanks for adding that!

from hierarchicalforecast.