ecmwf-opendata
is a package to simplify the download of ECMWF open data. It implements a request-based interface to the dataset using ECMWF's MARS language to select meteorological fields, similar to the existing ecmwf-api-client Python package.
A collection of Jupyter Notebooks that make use of that package is available here.
The example below will download the latest available 10-day forecast for the mean sea level pressure (msl
) into a local file called data.grib2
:
from ecmwf.opendata import Client
client = Client()
client.retrieve(
step=240,
type="fc",
param="msl",
target="data.grib2",
)
❗ NOTE: This package is designed for users that want to download a subset of the whole dataset. If you plan to download a large percentage of each data file, it may be more efficient to download whole files and filter out the data you want locally. See the documentation on the file naming convention for more information. Alternatively, you can use this tool to download whole files by only specifying
date
,time
,step
,stream
andtype
. Please be aware that all data for a full day is in the order of 726 GiB.
The constructor of the client object takes the following options:
client = Client(
source="ecmwf",
beta=True,
preserve_request_order=False,
infer_stream_keyword=True,
)
where:
-
source
is either the name of server to contact or a fully qualified URL. Possible values areecmwf
to access ECMWF's servers, orazure
to access data hosted on Microsoft's Azure. Default isecmwf
. -
beta
is a boolean that indicates whether to access the beta or the production version of the dataset. Current onlybeta=True
is supported. -
preserve_request_order
. If this flag is set toTrue
, the library will attempt to write the retrieved data into the target file in the order specified by the request. For example, if the request specifiesparam=[2t,msl]
the library will ensure that the field2t
is first in the target file, while withparam=[msl,2t]
, the fieldmsl
will be first. This also works across different keywords:...,levelist=[500,100],param=[z,t],...
will produce different output to...,param=[z,t],levelist=[500,100],...
If the flag is set toFalse
, the library will sort the request to minimise the number of HTTP requests made to the server, leading to faster download speeds. Default isFalse
. -
infer_stream_keyword
. Thestream
keyword represents the ECMWF forecasting system that creates the data. Setting it properly requires knowledge of how ECMWF runs its operations. If this boolean is set toTrue
, the library will try to infer the correct value for thestream
keyword based on the rest of the request. Default isTrue
.
⚠️ NOTE: It is recommended not to set thepreserve_request_order
flag toTrue
when downloading a large number of fields as this will add extra load on the servers.
Client.retrieve()
The Client.retrieve()
method takes request as input and will retrieve the corresponding data from the server and write them in the user's target file.
A request is a list of keyword/value pairs used to select the desired data. It is possible to specify a list of values for a given keyword.
The request can either be specified as a dictionary:
from ecmwf.opendata import Client
client = Client(source="ecmwf")
request = {
"time": 0,
"type": "fc",
"step": 24,
"param": ["2t", "msl"],
}
client.retrieve(request, "data.grib2")
# or:
client.retrieve(
request=request,
target="data.grib2",
)
or directly as arguments to the retrieve()
method:
from ecmwf.opendata import Client
client = Client(source="ecmwf")
client.retrieve(
time=0,
type="fc",
step=24,
param=["2t", "msl"],
target="data.grib2",
)
The date
and time
keyword are used to select the date and time of the forecast run (see Date and time below). If date
or both date
and time
are not specified, the library will query the server for the most recent matching data. The date
and time
of the downloaded forecast is returned by the retrieve()
method.
from ecmwf.opendata import Client
client = Client(source="ecmwf")
result = client.retrieve(
type="fc",
step=24,
param=["2t", "msl"],
target="data.grib2",
)
print(result.datetime)
may print 2022-01-23 00:00:00
.
Client.download()
The Client.download()
method takes the same parameters as the Client.retrieve()
method, but will download the whole data files from the server, ignoring keywords like param
, levelist
or number
.
The example below will download all field from the latest time step 24, ignoring the keyword param
:
from ecmwf.opendata import Client
client = Client(source="ecmwf")
client.download(
param="msl",
type="fc",
step=24,
target="data.grib2",
)
Client.latest()
The Client.latest()
method takes the same parameters as the Client.retrieve()
method, and returns the date of the most recent matching forecast without downloading the data:
from ecmwf.opendata import Client
client = Client(source="ecmwf")
print(client.latest(
type="fc",
step=24,
param=["2t", "msl"],
target="data.grib2",
))
may print 2022-01-23 00:00:00
.
⏰ NOTE: The data is available between 7 and 9 hours after the forecast starting date and time, depending on the forecasting system and the time step specified.
The supported keywords are:
type
: the type of data (compulsory, defaults tofc
).stream
: the forecast system (optional if unambiguous, compulsory otherwise). See theinfer_stream_keyword
above.date
: the date at which the forecast starts.time
: the time at which the forecast starts.step
: the forecast time step in hours, orfcmonth
, the time step in months for the seasonal forecast (compulsory, default to0
and1
, respectively).
and (all optional, with no defaults):
param
: the meteorological parameters, such as wind, pressure or humidity.levtype
: select between single level parameters and parameters on pressure levels.levelist
: the list of pressure levels when relevant.number
: the list of ensemble member numbers when relevant.
The keywords in the first list are used to identify which file to access, while the second list is used to identify which parts of the files need to be actually downloaded. Some HTTP servers are able to return multiple parts of a file, while other can only return a single part from a file. In the latter case, the library may perform many HTTP requests to the server. If you wish to download whole files, only provide keywords from the first list.
The date and time parameters refer to the starting time of the forecast. All date and time are expressed in UTC.
There are several ways to specify the date and time in a request.
Date can be specified using strings, numbers and Python datetime.datetime
or datetime.date
objects:
...
date='20220125',
time=12,
...
date='2022-01-25',
time=12,
...
date='2022-01-25 12:00:00',
...
date=20220125,
time=12,
...
date=datetime.datetime(2022, 1, 25, 12, 0, 0),
...
date=datetime.date(2022, 1, 25),
time=12,
...
Dates can also be given as a number less than or equal to zero. In this case, it is equivalent to the current UTC date minus the given number of days:
...
date=0, # today
date=-1, # yesterday
date=-2, # the day before yesterday
...
The keyword time
can be given as a string or an integer, or a Python datetime.time
object. All values of time below are equivalent:
...
time=12,
...
time=1200,
...
time='12',
...
time='1200',
...
time=datetime.time(12),
...
List of valid values for time |
---|
0, 6, 12 and 18 |
If time
is not specified, the time is extracted from the date.
...
date='2022-01-25 12:00:00',
...
is equivalent to:
...
date='2022-01-25',
time=12,
...
If the time
keyword is specified, it overrides any time given in the request.
...
date='2022-01-25 12:00:00',
time=18,
...
is equivalent to:
...
date='2022-01-25',
time=18,
...
As stated before, if date
or both date
and time
are not specified, the library will query the server for the most recent matching data. The date
and time
of the downloaded forecast is returned by the retrieve()
method:
Example without the date
keyword:
from ecmwf.opendata import Client
client = Client(source="ecmwf")
result = client.retrieve(
time=12,
type="fc",
param="2t",
step="24",
target="data.grib2",
)
print(result.datetime)
will print 2022-01-22 12:00:00
if run in the morning of 2022-01-23.
Example without the date
and time
keywords:
from ecmwf.opendata import Client
client = Client(source="ecmwf")
result = client.retrieve(
type="fc",
param="2t",
step="24",
target="data.grib2",
)
print(result.datetime)
will print 2022-01-23 00:00:00
if run in the morning of 2022-01-23.
ECMWF runs several forecasting systems:
Each of these forecasts also produces several types of products, that are referred to using the keywords
stream
and type
.
Valid values for
type
are:
HRES:
fc
: Forecast.
ENS:
cf
: Control forecast.pf
: Perturbed forecast.em
: Ensemble mean.es
: Ensemble standard deviation.ep
: Probabilities.
Valid values for
stream
are:
oper
: Atmospheric fields from HRES - 00 UTC and 12 UTC.wave
: Ocean wave fields from HRES - 00 UTC and 12 UTC.enfo
: Atmospheric fields from ENS.waef
: Ocean wave fields from ENS.scda
: Atmospheric fields from HRES - 06 UTC and 18 UTC.scwv
: Ocean wave fields from HRES - 06 UTC and 18 UTC.
📌 NOTE: if the client's flag
infer_stream_keyword
is set toTrue
, the library will infer the stream from thetype
andtime
. In that case, you just need to specifystream=wave
to access ocean wave products, and don't provide a value forstream
in other cases.
To select a time step, use the step
keyword:
...
step=24,
...
step=[24, 48],
...
Forecasting system | Time | List of time steps |
---|---|---|
HRES | 00 and 12 | 0 to 144 by 3, 144 to 240 by 6 |
ENS | 00 and 12 | 0 to 144 by 3, 144 to 360 by 6 |
HRES | 06 and 18 | 0 to 90 by 3 |
ENS | 06 and 18 | 0 to 144 by 3 |
Probabilities - Instantaneous weather events | 00 and 12 | 0 to 360 by 12 |
Probabilities - Daily weather events | 00 and 12 | 0-24 to 336-360 by 12 |
📌 NOTE: Not specifying
step
will return all available time steps.
To select a parameter, use the param
keyword:
...
param="msl",
...
param=["2t", "msl"]
...
for pressure level parameters, use the levelist
keyword:
...
param="t",
levelist=850,
...
param=["u", "v"],
levelist=[1000, 850, 500],
...
📌 NOTE: Not specifying
levelist
will return all available levels, and not specifyingparam
will return all available parameters.
List of pressure levels (hPa) |
---|
1000, 925, 850, 700, 500, 300, 250, 200 and 50 |
Below is the list of all parameters:
Atmospheric fields on pressure levels
Parameter | Description | Units |
---|---|---|
d | Divergence | s-1 |
gh | Geopotential height | gpm |
q | Specific humidity | kg kg-1 |
r | Relative humidity | % |
t | Temperature | K |
u | U component of wind | m s-1 |
v | V component of wind | m s-1 |
vo | Vorticity (relative) | s-1 |
Atmospheric fields on a single level
Parameter | Description | Units |
---|---|---|
10u | 10 metre U wind component | m s-1 |
10v | 10 metre V wind component | m s-1 |
2t | 2 metre temperature | K |
msl | Mean sea level pressure | Pa |
ro | Runoff | m |
skt | Skin temperature | K |
sp | Surface pressure | Pa |
st | Soil Temperature | K |
stl1 | Soil temperature level 1 | K |
tcwv | Total column vertically-integrated water vapour | kg m-2 |
tp | Total Precipitation | m |
Ocean waves fields
Parameter | Description | Units |
---|---|---|
mp2 | Mean zero-crossing wave period | s |
mwd | Mean wave direction | Degree true |
mwp | Mean wave period | s |
pp1d | Peak wave period | s |
swh | Significant height of combined wind waves and swell | m |
Ensemble mean and standard deviation - pressure levels
Parameter | Description | Units |
---|---|---|
gh | Geopotential height | gpm |
t | Temperature | K |
ws | Wind speed | m s-1 |
Ensemble mean and standard deviation - single level
Parameter | Description | Units |
---|---|---|
msl | Mean sea level pressure | Pa |
Instantaneous weather events - atmospheric fields - 850 hPa
Parameter | Description | Units |
---|---|---|
ptsa_gt_1p5stdev | Probability of temperature standardized anomaly greater than 1.5 standard deviation | % |
ptsa_gt_1stdev | Probability of temperature standardized anomaly greater than 1 standard deviation | % |
ptsa_gt_2stdev | Probability of temperature standardized anomaly greater than 2 standard deviation | % |
ptsa_lt_1p5stdev | Probability of temperature standardized anomaly less than -1.5 standard deviation | % |
ptsa_lt_1stdev | Probability of temperature standardized anomaly less than -1 standard deviation | % |
ptsa_lt_2stdev | Probability of temperature standardized anomaly less than -2 standard deviation | % |
Daily weather events - atmospheric fields - single level
Parameter | Description | Units |
---|---|---|
10fgg10 | 10 metre wind gust of at least 10 m/s | % |
10fgg15 | 10 metre wind gust of at least 15 m/s | % |
10fgg25 | 10 metre wind gust of at least 25 m/s | % |
tpg1 | Total precipitation of at least 1 mm | % |
tpg10 | Total precipitation of at least 10 mm | % |
tpg100 | Total precipitation of at least 100 mm | % |
tpg20 | Total precipitation of at least 20 mm | % |
tpg25 | Total precipitation of at least 25 mm | % |
tpg5 | Total precipitation of at least 5 mm | % |
tpg50 | Total precipitation of at least 50 mm | % |
Instantaneous weather events - ocean waves fields
Parameter | Description | Units |
---|---|---|
swhg2 | Significant wave height of at least 2 m | % |
swhg4 | Significant wave height of at least 4 m | % |
swhg6 | Significant wave height of at least 6 m | % |
swhg8 | Significant wave height of at least 8 m | % |
You can select individual members of the ensemble forecast use the keyword number
.
...
stream="enfo",
step=24,
param="msl",
number=1,
...
stream="enfo",
step=24,
param="msl",
number=[1, 10, 20],
...
List of ensemble numbers |
---|
1 to 50 |
📌 NOTE: Not specifying
number
will return all ensemble forecast members.
from ecmwf.opendata import Client
client = Client(source="ecmwf")
client.retrieve(
time=0,
stream="oper",
type="fc",
step=24,
param="2t",
target="data.grib2",
)
from ecmwf.opendata import Client
client = Client(source="ecmwf")
client.retrieve(
time=0,
stream="oper",
type="tf",
step=240,
target="data.bufr",
)
- The downloaded data are encoded in BUFR edition 4
- For the HRES Tropical Cyclone tracks at time=06 and time=18 use:
...
step = 90,
...
❗ NOTE: Tropical cyclone tracks products are only available when there are tropical cyclones observed or forecast.
Download a single surface parameter at a single forecast step for all ensemble members from ECMWF's 12UTC 00UTC ENS forecast
from ecmwf.opendata import Client
client = Client(source="ecmwf")
client.retrieve(
time=0,
stream="enfo",
type="pf",
param="msl",
target="data.grib2",
)
- To download a single ensemble member, use the
number
keyword:number=1
. - All of the odd numbered ensemble members use
number=[num for num in range(1,51,2)]
. - To download the control member, use
type="cf"
.
The Tropical Cyclone tracks are identified by the keyword type="tf"
.
from ecmwf.opendata import Client
client = Client(source="ecmwf")
client.retrieve(
time=0,
stream="enfo",
type="tf",
step=240,
target="data.bufr",
)
- The downloaded data are encoded in BUFR edition 4
- For the ENS Tropical Cyclone tracks at time=06 and time=18 replace
step=240
withstep=144
.
Download the ensemble mean and standard deviation for all parameters at a single forecast step from ECMWF's 00UTC ENS forecast
The ensemble mean and standard deviation are identified by the keywords type="em"
:
from ecmwf.opendata import Client
client = Client(source="ecmwf")
client.retrieve(
time=0,
stream="enfo",
type="em",
step=24,
target="data.grib2",
)
and type="es"
, respectively:
from ecmwf.opendata import Client
client = Client(source="ecmwf")
client.retrieve(
time=0,
stream="enfo",
type="es",
step=24,
target="data.grib2",
)
The ensemble probability products are identified by the keyword type="ep"
. The probability products are available only for time=00
and time=12
.
Two different products are available.
The probability of temperature standardized anomalies at a constant pressure level of 850hPa are available at 12 hourly forecast steps.
from ecmwf.opendata import Client
client = Client(source="ecmwf")
client.retrieve(
time=0,
stream="enfo",
type="ep",
step=[i for i in range(12, 361, 12)],
levelist=850,
param=[
"ptsa_gt_1stdev",
"ptsa_gt_1p5stdev",
"ptsa_gt_2stdev",
"ptsa_lt_1stdev",
"ptsa_lt_1p5stdev",
"ptsa_lt_2stdev",
],
target="data.grib2",
)
The probabilities of total precipitation and wind gusts exceeding specified thresholds in a 24 hour period are available for step ranges 0-24 to 336-360 by 12. These are specified in the retrieval request using, e.g.: step=["0-24", "12-36", "24-48"]
.
from ecmwf.opendata import Client
client = Client(source="ecmwf")
steps = [f"{12 * i}-{ 12 * i + 24}" for i in range(29)]
client.retrieve(
time=0,
stream="enfo",
type="ep",
step=steps,
param=["tpg1", "tpg5", "10fgg10"],
target="data.grib2",
)
By downloading data from the ECMWF open data dataset, you agree to the their terms: Attribution 4.0 International (CC BY 4.0). If you do not agree with such terms, do not download the data. Visit this page for more information.
Apache License 2.0 In applying this licence, ECMWF does not waive the privileges and immunities granted to it by virtue of its status as an intergovernmental organisation nor does it submit to any jurisdiction.