2 Issues: Getting data from the future and integrity of the datasets about meteostat-python HOT 6 CLOSED

@clampr Thanks for all your work.

Steps I did afterwards:

1. I used python lib to get some sample data for stations in Dresden.
2. Get the data from dwd within the same timeframe
3. Compare both datasets --> They are identical --> Proof: routine fix works correctly for stations in Dresden, should probably work for all.

from meteostat-python.

I understand the confusion and I will add an FAQ page on the dev site soon to explain some of these common questions. Regrading the first issue: this is caused by the fact, that model=True by default. At Meteostat we use (MOSMIX) model data to fill gaps in the historical time series. As we do have these forecasts anyway, they're provided along with the historical data. If you don't want model data, you can opt-out on both the website and the Python lib.

Issue 2 is a bit more special and has to do with an order/priority fix in the latest version of the bulk data interface. We now choose SYNOP data over METAR reports because they're more accurate. This will be fixed in the next release of the web app. Until then, I'd prefer the Python results.

from meteostat-python.

@clampr Thanks for your answer.

1. Issue:
   As far as it conserns my usecase. I support the fact model=True by default for historically data while you can still opt-out. It will generate more uniform data. But still including the forecast into historically data is a bad idea and needs to be differentiated. There might be a better implementation splitting historical and forcast data. A second button for forcast data on the website might a solution (fetching all forcast data for a station/ region), for python maybe a forcast class?

2. Issue:
   Python is my prefered choice anyway. But the integrity is still given?

from meteostat-python.

Well, the problem is that we rely on static station dumps. Data isn't queried directly from the Meteostat DB when you make a request through the Python lib. This approach allows the library to scale with minimal to no restrictions.

I agree that mixing historical data and forecasts is confusing. Also, I think there are better alternatives when it comes to weather forecasts. If we want to keep the ability of getting data for the current day, which I think is relevant to many users, we need to have at least a short-term forecast (48 hours) in the station dumps (because of update frequency and caching). For now, I limited the forecasts to 48 hours for all dumps. I think it's fair to say that Meteostat has a focus on historical data with short term forecasts for more comfortable analysis. As 48 hour forecasts are very accurate most of the time, I think it's okay to use them for analysis in certain situations.

If you're only using the Python interface, integrity should be given.

from meteostat-python.

Ok this is a good limitation.

If you're only using the Python interface, integrity should be given.

Perfect thanks.

An other question:

When comparing the output and just changing model=True to False I expect to get less rows/ or attributes with NaN/null values.
Steps to reproduce:

"""define the periode for fetching"""
start = datetime(2020, 3, 1)
end = datetime(2020, 5, 31, 23, 59)
station = "10488"

"""Get hourly data and opt-out for model data"""
df_weatherdata = Hourly(station, start, end, model=False)
df_weatherdata = df_weatherdata.fetch()
df_weatherdata.reset_index(inplace=True)
df_weatherdata.rename(columns={"time": "datetime"}, inplace=True)
df_weatherdata=df_weatherdata.assign(id_national=station)
print(df_weatherdata)

Results: While model=False I still get values or other values while model=True. A model might be even used when data is present?

For comparising:
10488_2020-03-01_2020-05-31_True.csv
10488_2020-03-01_2020-05-31_False.csv

compared csv files online:
https://www.textcompare.org/csv/compare?id=61165041052c9b0015ebb514

from meteostat-python.

New docs on data quality. https://dev.meteostat.net/quality.html

from meteostat-python.