aggregate function about meteostat-python HOT 8 CLOSED

@edesz you are right. It is calculating the sum for each weather station first and then averages across all stations selected by the user. I don't have a problem with this and I think it totally makes sense. I just want to point out it is calculating the mean across weather stations since the official documentation is not particularly clear on this point...

I was trying to do things manually in pandas for a single station, which was not necessary at all - when trying to adapt this to handle multiple stations, it becomes unnecessarily complex. Actually, you were thinking of aggregating data from multiple stations the right way (by using .aggregate()).

...you can set its spatial parameter to true if you want a regional average. In that case, Meteostat will first calculate the precipitation sum for each weather station and then the mean across all stations...

Oh okay, so I was wrong. meteostat DOES come with that functionality built in. It is right there in the docs too. Thanks for pointing it out. It is quite convenient to be able to do aggregating for multiple stations. Nice!

from meteostat-python.

@clampr That will be fantastic! Thank you!

from meteostat-python.

Unfortunately, I can't seem to replicate this problem.

Here is an MRE, for Daily() and Hourly(), to demonstrate manually aggregating the data to monthly frequency compared to calling .aggregate() on Daily() and Monthly() respectively

Imports

import numpy as np
from meteostat import Stations, Daily, Hourly

Define start and end dates

start = datetime(2018, 1, 1)
end = datetime(2018, 12, 31)

Daily data

Define daily aggregation dict for some of the columns (using meteostat documentation)

agg_dict_daily = {'tavg': 'mean', 'tmin': 'min', "tmax": "max", "prcp": "sum"}

Manual aggregation for daily data

data = Daily('10637', start=start, end=end)
data = data.normalize()
data = data.fetch()
data_manual_agg = data.resample("1W").agg(agg_dict_daily).round(1)
print(data_manual_agg.head())

            tavg  tmin  tmax  prcp
time                              
2018-01-07   7.3   4.1  11.2  37.0
2018-01-14   5.4  -1.8  10.3   2.0
2018-01-21   3.7  -2.7  12.2  15.6
2018-01-28   6.6   0.3  11.5   9.6
2018-02-04   4.4  -1.6  10.8  12.6

Calling .aggregate() on Daily()

data = Daily('10637', start=start, end=end)
data = data.normalize()
data_agg = data.aggregate('1W')
data_agg = data_agg.fetch()
print(data_agg[list(agg_dict_daily)].head())

            tavg  tmin  tmax  prcp
time                              
2018-01-07   7.3   4.1  11.2  37.0
2018-01-14   5.4  -1.8  10.3   2.0
2018-01-21   3.7  -2.7  12.2  15.6
2018-01-28   6.6   0.3  11.5   9.6
2018-02-04   4.4  -1.6  10.8  12.6

Verify equality between the "prcp" column obtained using

• manual aggregation of daily data to monthly frequency
• calling .aggregate() on Daily()

assert not np.testing.assert_array_almost_equal(
    data_manual_agg["prcp"].to_numpy(),
    data_agg["prcp"].to_numpy(),
)

The manual resampling of daily data to monthly frequency, using the expected aggregations, agrees with the output of calling .aggregate() on Hourly().

Hourly data

Define hourly aggregation dict (using meteostat documentation)

agg_dict_hourly = {
    'temp': 'mean',
    'dwpt': 'mean',
    "rhum": "mean",
    "prcp": "sum",
}

Manual aggregation for hourly data

data = Hourly('10637', start=start, end=end)
data = data.normalize()
data = data.fetch()
data_manual_agg = data.resample("1W").agg(agg_dict_hourly).round(1)
print(data_manual_agg.head())

            temp  dwpt  rhum  prcp
time                              
2018-01-07   7.3   3.8  79.2  37.1
2018-01-14   5.4   2.3  80.6   2.0
2018-01-21   3.7   0.0  78.0  14.0
2018-01-28   6.6   4.5  86.8  11.2
2018-02-04   4.4   1.5  82.1  12.6

Calling .aggregate() on Hourly()

data = Hourly('10637', start=start, end=end)
data = data.normalize()
data_agg = data.aggregate('1W')
data_agg = data_agg.fetch()
print(data_agg[list(agg_dict_hourly)].head())

            temp  dwpt  rhum  prcp
time                              
2018-01-07   7.3   3.8  79.2  37.1
2018-01-14   5.4   2.3  80.6   2.0
2018-01-21   3.7   0.0  78.0  14.0
2018-01-28   6.6   4.5  86.8  11.2
2018-02-04   4.4   1.5  82.1  12.6

Verify equality between the "prcp" column obtained using

• manual aggregation of hourly data to monthly frequency
• calling .aggregate() on Hourly()

assert not np.testing.assert_array_almost_equal(
    data_manual_agg["prcp"].to_numpy(),
    data_agg["prcp"].to_numpy(),
)

So, manual resampling of hourly data to monthly, using the expected aggregations, agrees with calling .aggregate() on Hourly().

Versions used

import meteostat
print(f"meteostat=={meteostat.__version__}")

meteostat==1.5.10

Do you have a specific example you could show to demonstrate the problem you noticed?

from meteostat-python.

I think my problem is I'm using multiple stations. So instead of calculating the sum of precipitation of, say, all the weather stations in the UK on a particular day, it calculates the average of precipitation across all weather stations using that day's data.

from meteostat-python.

That does make sense. I'm not sure meteostat. comes with that functionality built in. It's probably up to the user to use pandas to achieve their desired transformations of the raw data. If you'll be working with multiple stations, it is certainly worth looking into which transformations work for your use-case.

from meteostat-python.

I think @edesz is correct. Usually, aggregate() groups by station. However, you can set its spatial parameter to true if you want a regional average. In that case, Meteostat will first calculate the precipitation sum for each weather station and then the mean across all stations. Of course you can always fetch() the result and use the power of Pandas 😉

from meteostat-python.

@edesz @clampr Thanks for the comment. I did calculate the precipitation sum manually. @edesz you are right. It is calculating the sum for each weather station first and then averages across all stations selected by the user. I don't have a problem with this and I think it totally makes sense. I just want to point out it is calculating the mean across weather stations since the official documentation is not particularly clear on this point, and it would be nice if the author can add just one sentence of explanation on that. Anyway, thank you all!

from meteostat-python.

Good point @cshenicct 👍 It currently reads:

[spatial parameter] Calculate averages across weather stations

Maybe add a sentence on the grouping by station here?

from meteostat-python.