pypsa / atlite Goto Github PK
View Code? Open in Web Editor NEWAtlite: A Lightweight Python Package for Calculating Renewable Power Potentials and Time Series
Home Page: https://atlite.readthedocs.io
Atlite: A Lightweight Python Package for Calculating Renewable Power Potentials and Time Series
Home Page: https://atlite.readthedocs.io
When passing odd bounds as cutout parameters, the retrieved data is misaligned to the built coordinate system.
For example
cutout = atlite.Cutout(path="uk-2011-01",
module="era5",
bounds=(-14.1, 50.34, 1.512, 61.123),
time="2011-01-01")
cutout.prepare()
will try to retrieve data for a grid beginning with -14.1 followed by steps of 0.25. However the coordinate system set up (automaticaly before cutout.prepare()
) begins with -14.0 followed by steps of 0.25.
atilite
merges the original grid with the retrieved data without an error, which ends up in a lot of nan's
When running the standard pattern for creating an ERA5 cutout (https://atlite.readthedocs.io/en/latest/examples/plotting_with_atlite.html)
import geopandas as gpd
import cartopy.io.shapereader as shpreader
import atlite
import logging
logging.basicConfig(level=logging.INFO)
shpfilename = shpreader.natural_earth(resolution='10m',
category='cultural',
name='admin_0_countries')
reader = shpreader.Reader(shpfilename)
UkIr = gpd.GeoSeries({r.attributes['NAME_EN']: r.geometry
for r in reader.records()},
crs={'init': 'epsg:4326'}
).reindex(['United Kingdom', 'Ireland'])
# Define the cutout; this will not yet trigger any major operations
cutout = atlite.Cutout(name="uk-2011-01",
cutout_dir="./",
module="era5",
bounds=UkIr.unary_union.bounds,
years=slice(2011,2011),
months=slice(1,1))
# This is where all the work happens.
cutout.prepare()
I got the following message when I run the code in Spyder:
> C:\Users\Yiping\Miniconda3\lib\site-packages\pyproj\crs\crs.py:55: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method. When making the change, be mindful of axis order changes: https://pyproj4.github.io/pyproj/stable/gotchas.html#axis-order-changes-in-proj-6
> return _prepare_from_string(" ".join(pjargs))
> INFO:atlite.cutout:Cutout uk-2011-01 not found in directory ./, building new one
> INFO:cdsapi:Sending request to https://cds.climate.copernicus.eu/api/v2/resources/reanalysis-era5-single-levels
> INFO:cdsapi:Request is completed
> INFO:atlite.datasets.era5:Downloading request for 9 variables to C:\Users\Yiping\AppData\Local\Temp\tmpqrvzeks3.nc
> INFO:cdsapi:Downloading http://136.156.132.235/cache-compute-0006/cache/data7/adaptor.mars.internal-1583414461.1578808-27797-3-b18c0f99-b457-440f-990b-0daa709defa9.nc to C:\Users\Yiping\AppData\Local\Temp\tmpqrvzeks3.nc (126.6K)
> INFO:cdsapi:Download rate 846.1K/s
> INFO:atlite.preparation:Starting preparation of cutout 'uk-2011-01'
> INFO:atlite.preparation:1 tasks have been collected. Starting running them on all processors.
It seems to be working, but the problem is:
my computer CPU keeps running with full speed but without any output for 2 hours.
Then, I run the code in the command line and get the following error. It seems it is the problem of the Initialization of Cutout with "months = slice(1,1)". But it is exactly the same as the given example (see examples/create_cutout.py). Can anyone help with this? Thanks a lot!**
> C:\Users\Yiping\Miniconda3\lib\site-packages\pyproj\crs.py:77: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method.
> return _prepare_from_string(" ".join(pjargs))
> C:\Users\Yiping\Miniconda3\lib\site-packages\pyproj\crs.py:77: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method.
> return _prepare_from_string(" ".join(pjargs))
> C:\Users\Yiping\Miniconda3\lib\site-packages\pyproj\crs.py:77: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method.
> return _prepare_from_string(" ".join(pjargs))
> C:\Users\Yiping\Miniconda3\lib\site-packages\pyproj\crs.py:77: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method.
> return _prepare_from_string(" ".join(pjargs))
> C:\Users\Yiping\Miniconda3\lib\site-packages\pyproj\crs.py:77: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method.
> return _prepare_from_string(" ".join(pjargs))
> C:\Users\Yiping\Miniconda3\lib\site-packages\pyproj\crs.py:77: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method.
> return _prepare_from_string(" ".join(pjargs))
> C:\Users\Yiping\Miniconda3\lib\site-packages\pyproj\crs.py:77: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method.
> return _prepare_from_string(" ".join(pjargs))
> C:\Users\Yiping\Miniconda3\lib\site-packages\pyproj\crs.py:77: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method.
> return _prepare_from_string(" ".join(pjargs))
> C:\Users\Yiping\Miniconda3\lib\site-packages\pyproj\crs.py:77: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method.
> return _prepare_from_string(" ".join(pjargs))
> C:\Users\Yiping\Miniconda3\lib\site-packages\pyproj\crs.py:77: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method.
> return _prepare_from_string(" ".join(pjargs))
> C:\Users\Yiping\Miniconda3\lib\site-packages\pyproj\crs.py:77: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method.
> return _prepare_from_string(" ".join(pjargs))
> C:\Users\Yiping\Miniconda3\lib\site-packages\pyproj\crs.py:77: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method.
> return _prepare_from_string(" ".join(pjargs))
> Traceback (most recent call last):
> File "<string>", line 1, in <module>
> File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
> exitcode = _main(fd)
> File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 114, in _main
> prepare(preparation_data)
> File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 225, in prepare
> Traceback (most recent call last):
> File "<string>", line 1, in <module>
> _fixup_main_from_path(data['init_main_from_path'])
> File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
> File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
> exitcode = _main(fd)run_name="__mp_main__")
>
> File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 114, in _main
> File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 263, in run_path
> prepare(preparation_data)
> File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 225, in prepare
> pkg_name=pkg_name, script_name=fname)
> File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 96, in _run_module_code
> _fixup_main_from_path(data['init_main_from_path'])mod_name, mod_spec, pkg_name, script_name)
> File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
>
> File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 85, in _run_code
> exec(code, run_globals)run_name="__mp_main__")
> File "C:\Users\Yiping\Documents\Python Workspace\REUTE\climate\atlite_example.py", line 29, in <module>
>
> months=slice(1,1)) File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 263, in run_path
>
> File "C:\Users\Yiping\Miniconda3\lib\site-packages\atlite\cutout.py", line 60, in __init__
> pkg_name=pkg_name, script_name=fname) assert False
> AssertionError
>
> File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 96, in _run_module_code
> Traceback (most recent call last):
> File "<string>", line 1, in <module>
> mod_name, mod_spec, pkg_name, script_name) File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
>
> File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 85, in _run_code
> exitcode = _main(fd)
> File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 114, in _main
> exec(code, run_globals)
> Traceback (most recent call last):
> File "C:\Users\Yiping\Documents\Python Workspace\REUTE\climate\atlite_example.py", line 29, in <module>
> File "<string>", line 1, in <module>
> prepare(preparation_data) File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
>
> File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 225, in prepare
> months=slice(1,1))
> File "C:\Users\Yiping\Miniconda3\lib\site-packages\atlite\cutout.py", line 60, in __init__
> exitcode = _main(fd) Traceback (most recent call last):
> _fixup_main_from_path(data['init_main_from_path'])
>
> Traceback (most recent call last):
> File "<string>", line 1, in <module>
> Traceback (most recent call last):
> File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 114, in _main
> File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
> File "<string>", line 1, in <module>
> assert False File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
> File "<string>", line 1, in <module>
> Traceback (most recent call last):
> File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
>
> prepare(preparation_data) File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
>
> File "<string>", line 1, in <module>
> AssertionErrorexitcode = _main(fd)run_name="__mp_main__") File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 225, in prepare
>
> exitcode = _main(fd) File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
>
> exitcode = _main(fd)Traceback (most recent call last):
>
>
> File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 263, in run_path
>
> File "<string>", line 1, in <module>
> exitcode = _main(fd) File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 114, in _main
> _fixup_main_from_path(data['init_main_from_path']) File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 114, in _main
> File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 114, in _main
> Traceback (most recent call last):
>
> File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
>
> pkg_name=pkg_name, script_name=fname) File "<string>", line 1, in <module>
> prepare(preparation_data) File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 114, in _main
> File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
> prepare(preparation_data)
> Traceback (most recent call last):
> File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
>
> prepare(preparation_data) exitcode = _main(fd)
> File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 96, in _run_module_code
> File "<string>", line 1, in <module>
> File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 225, in prepare
>
> exitcode = _main(fd)
> File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 225, in prepare
> run_name="__mp_main__") File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
> prepare(preparation_data) File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 225, in prepare
>
> _fixup_main_from_path(data['init_main_from_path']) mod_name, mod_spec, pkg_name, script_name)
>
> File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 114, in _main
> File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 114, in _main
>
> _fixup_main_from_path(data['init_main_from_path'])
> File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 263, in run_path
> File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 225, in prepare
> exitcode = _main(fd)Traceback (most recent call last):
> _fixup_main_from_path(data['init_main_from_path']) File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
>
> File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 85, in _run_code
>
> pkg_name=pkg_name, script_name=fname) prepare(preparation_data)
>
> File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
> File "<string>", line 1, in <module>
>
> File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 114, in _main
> _fixup_main_from_path(data['init_main_from_path'])prepare(preparation_data)
>
> run_name="__mp_main__")exec(code, run_globals) File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
> File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 96, in _run_module_code
> File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
> File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 225, in prepare
> File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
> File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 225, in prepare
>
>
> run_name="__mp_main__") prepare(preparation_data) File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 263, in run_path
> File "C:\Users\Yiping\Documents\Python Workspace\REUTE\climate\atlite_example.py", line 29, in <module>
> mod_name, mod_spec, pkg_name, script_name)exitcode = _main(fd)
>
> _fixup_main_from_path(data['init_main_from_path'])
> run_name="__mp_main__") _fixup_main_from_path(data['init_main_from_path'])
> run_name="__mp_main__") File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 263, in run_path
>
>
> File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 225, in prepare
>
> pkg_name=pkg_name, script_name=fname)
> months=slice(1,1)) File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 85, in _run_code
> File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 114, in _main
> File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 263, in run_path
> File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
> File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 263, in run_path
>
> File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
>
> pkg_name=pkg_name, script_name=fname)
> File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 96, in _run_module_code
> _fixup_main_from_path(data['init_main_from_path']) File "C:\Users\Yiping\Miniconda3\lib\site-packages\atlite\cutout.py", line 60, in __init__
> exec(code, run_globals) prepare(preparation_data) File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 96, in _run_module_code
> run_name="__mp_main__")pkg_name=pkg_name, script_name=fname)
>
> run_name="__mp_main__")
> pkg_name=pkg_name, script_name=fname)
>
>
> File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 263, in run_path
> File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
> assert False
> File "C:\Users\Yiping\Documents\Python Workspace\REUTE\climate\atlite_example.py", line 29, in <module>
> File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 225, in prepare
> File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 96, in _run_module_code
> mod_name, mod_spec, pkg_name, script_name) File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 96, in _run_module_code
>
> pkg_name=pkg_name, script_name=fname) mod_name, mod_spec, pkg_name, script_name)months=slice(1,1))
> File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 263, in run_path
> AssertionError
> run_name="__mp_main__")
>
> _fixup_main_from_path(data['init_main_from_path']) File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 85, in _run_code
> mod_name, mod_spec, pkg_name, script_name)
> pkg_name=pkg_name, script_name=fname)
> mod_name, mod_spec, pkg_name, script_name) File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 263, in run_path
> File "C:\Users\Yiping\Miniconda3\lib\site-packages\atlite\cutout.py", line 60, in __init__
>
>
> File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 96, in _run_module_code
>
> File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 85, in _run_code
>
> File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
> exec(code, run_globals) File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 85, in _run_code
> File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 96, in _run_module_code
> mod_name, mod_spec, pkg_name, script_name) File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 85, in _run_code
> pkg_name=pkg_name, script_name=fname)assert False
>
>
> exec(code, run_globals) File "C:\Users\Yiping\Documents\Python Workspace\REUTE\climate\atlite_example.py", line 29, in <module>
> exec(code, run_globals)exec(code, run_globals)
> mod_name, mod_spec, pkg_name, script_name) File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 85, in _run_code
>
> File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 96, in _run_module_code
>
> run_name="__mp_main__") File "C:\Users\Yiping\Documents\Python Workspace\REUTE\climate\atlite_example.py", line 29, in <module>
>
>
> AssertionError File "C:\Users\Yiping\Documents\Python Workspace\REUTE\climate\atlite_example.py", line 29, in <module>
> months=slice(1,1))
>
> File "C:\Users\Yiping\Documents\Python Workspace\REUTE\climate\atlite_example.py", line 29, in <module>
> File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 85, in _run_code
>
> exec(code, run_globals)mod_name, mod_spec, pkg_name, script_name) File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 263, in run_path
> File "C:\Users\Yiping\Miniconda3\lib\site-packages\atlite\cutout.py", line 60, in __init__
>
> months=slice(1,1))months=slice(1,1))
> months=slice(1,1))exec(code, run_globals)assert False File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 85, in _run_code
>
>
> pkg_name=pkg_name, script_name=fname) File "C:\Users\Yiping\Documents\Python Workspace\REUTE\climate\atlite_example.py", line 29, in <module>
>
>
> File "C:\Users\Yiping\Miniconda3\lib\site-packages\atlite\cutout.py", line 60, in __init__
> File "C:\Users\Yiping\Miniconda3\lib\site-packages\atlite\cutout.py", line 60, in __init__
>
>
> File "C:\Users\Yiping\Miniconda3\lib\site-packages\atlite\cutout.py", line 60, in __init__
> months=slice(1,1))AssertionErrorexec(code, run_globals) File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 96, in _run_module_code
> File "C:\Users\Yiping\Documents\Python Workspace\REUTE\climate\atlite_example.py", line 29, in <module>
>
>
>
> assert Falseassert False File "C:\Users\Yiping\Miniconda3\lib\site-packages\atlite\cutout.py", line 60, in __init__
> assert False File "C:\Users\Yiping\Documents\Python Workspace\REUTE\climate\atlite_example.py", line 29, in <module>
>
>
> mod_name, mod_spec, pkg_name, script_name)months=slice(1,1))
> AssertionErrorAssertionError
>
> assert FalseAssertionErrormonths=slice(1,1))
>
> File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 85, in _run_code
> File "C:\Users\Yiping\Miniconda3\lib\site-packages\atlite\cutout.py", line 60, in __init__
>
>
>
> AssertionErrorassert False File "C:\Users\Yiping\Miniconda3\lib\site-packages\atlite\cutout.py", line 60, in __init__
> exec(code, run_globals)
>
>
> assert False File "C:\Users\Yiping\Documents\Python Workspace\REUTE\climate\atlite_example.py", line 29, in <module>
> AssertionError
>
> AssertionErrormonths=slice(1,1))
>
> File "C:\Users\Yiping\Miniconda3\lib\site-packages\atlite\cutout.py", line 60, in __init__
> assert False
> AssertionError
[edit by @euronion : Formatting. Surronding your output with ``` ... ``` preserves the error message formatting and makes it easier to read.]
An error, saying 'Not implemented for negative latitudes', was raised when I converted weather data in regions with negative latitudes.
Can Atlite still get the right converted results after the 'abs()' function is added to 'orientation.py'? Cheers.
The cutout creation using environments with python 3.7 sometimes stops during the writing of the netcdf file. The progress then stays at a low percentage 0 to 20 % and noting more happens. When killing the process, one sees that dask was waiting for further calculation. I am absolutely not sure why this is the case.
Trying again helps and using a python 3.8 environment remedies.
cyordereddict is obsolete! Since Python 3.5 it is replaced by built-in collections.OrderedDict
https://github.com/shoyer/cyordereddict
Is it still in use at atlite?
I had an already prepared cutout in the old atlite version and opened it with the new atlite version:
cutout = atlite.Cutout(config['cutout'], cutout_dir=snakemake.input.cutout_dir)
The script took some time and then failed with the following error message:
WARNING:atlite.utils:Found an old-style directory-like cutout. Use `prepare` to transfer that data.
/home/vagrant/miniconda/envs/atlite-v0.2/lib/python3.7/site-packages/xarray/backends/api.py:783: FutureWarning: In xarray version 0.13 `auto_combine` will be deprecated.
coords=coords)
/home/vagrant/miniconda/envs/atlite-v0.2/lib/python3.7/site-packages/xarray/backends/api.py:783: FutureWarning: The datasets supplied have global dimension coordinates. You may want
to use the new `combine_by_coords` function (or the
`combine='by_coords'` option to `open_mfdataset` to order the datasets
before concatenation. Alternatively, to continue concatenating based
on the order the datasets are supplied in in future, please use the
new `combine_nested` function (or the `combine='nested'` option to
open_mfdataset).The datasets supplied require both concatenation and merging. From
xarray version 0.14 this will operation will require either using the
new `combine_nested` function (or the `combine='nested'` option to
open_mfdataset), with a nested list structure such that you can combine
along the dimensions None. Alternatively if your datasets have global
dimension coordinates then you can use the new `combine_by_coords`
function.
coords=coords)
---------------------------------------------------------------------------
MergeError Traceback (most recent call last)
<ipython-input-7-042299d21e33> in <module>
1 # Load atlite cutout
----> 2 cutout = atlite.Cutout(config['cutout'], cutout_dir=snakemake.input.cutout_dir)
~/miniconda/envs/atlite-v0.2/lib/python3.7/site-packages/atlite/cutout.py in __init__(self, name, data, cutout_dir, **cutoutparams)
92 elif os.path.isdir(os.path.join(self.cutout_dir, self.name)):
93 data = utils.migrate_from_cutout_directory(os.path.join(self.cutout_dir, self.name),
---> 94 self.name, self.cutout_fn, cutoutparams)
95 self.is_view = True
96 else:
~/miniconda/envs/atlite-v0.2/lib/python3.7/site-packages/atlite/utils.py in migrate_from_cutout_directory(old_cutout_dir, name, cutout_fn, cutoutparams)
59 old_cutout_dir = Path(old_cutout_dir)
60 with xr.open_dataset(old_cutout_dir / "meta.nc") as meta:
---> 61 data = xr.open_mfdataset(str(old_cutout_dir / "[12]*.nc"))
62 data.attrs.update(meta.attrs)
63 data.attrs['prepared_features'] = list(sys.modules['atlite.datasets.' + data.attrs["module"]].features)
~/miniconda/envs/atlite-v0.2/lib/python3.7/site-packages/xarray/backends/api.py in open_mfdataset(paths, chunks, concat_dim, compat, preprocess, engine, lock, data_vars, coords, combine, autoclose, parallel, **kwargs)
781 combined = auto_combine(datasets, concat_dim=concat_dim,
782 compat=compat, data_vars=data_vars,
--> 783 coords=coords)
784 elif combine == 'nested':
785 # Combined nested list by successive concat and merge operations
~/miniconda/envs/atlite-v0.2/lib/python3.7/site-packages/xarray/core/combine.py in auto_combine(datasets, concat_dim, compat, data_vars, coords, fill_value)
627 return _old_auto_combine(datasets, concat_dim=concat_dim,
628 compat=compat, data_vars=data_vars,
--> 629 coords=coords, fill_value=fill_value)
630
631
~/miniconda/envs/atlite-v0.2/lib/python3.7/site-packages/xarray/core/combine.py in _old_auto_combine(datasets, concat_dim, compat, data_vars, coords, fill_value)
681 else:
682 concatenated = datasets
--> 683 merged = merge(concatenated, compat=compat, fill_value=fill_value)
684 return merged
685
~/miniconda/envs/atlite-v0.2/lib/python3.7/site-packages/xarray/core/merge.py in merge(objects, compat, join, fill_value)
549
550 variables, coord_names, dims = merge_core(dict_like_objects, compat, join,
--> 551 fill_value=fill_value)
552 # TODO: don't always recompute indexes
553 merged = Dataset._construct_direct(
~/miniconda/envs/atlite-v0.2/lib/python3.7/site-packages/xarray/core/merge.py in merge_core(objs, compat, join, priority_arg, explicit_coords, indexes, fill_value)
460
461 priority_vars = _get_priority_vars(aligned, priority_arg, compat=compat)
--> 462 variables = merge_variables(expanded, priority_vars, compat=compat)
463 assert_unique_multiindex_level_names(variables)
464
~/miniconda/envs/atlite-v0.2/lib/python3.7/site-packages/xarray/core/merge.py in merge_variables(list_of_variables_dicts, priority_vars, compat)
170 else:
171 try:
--> 172 merged[name] = unique_variable(name, var_list, compat)
173 except MergeError:
174 if compat != 'minimal':
~/miniconda/envs/atlite-v0.2/lib/python3.7/site-packages/xarray/core/merge.py in unique_variable(name, variables, compat)
90 'objects to be combined:\n'
91 'first value: %r\nsecond value: %r'
---> 92 % (name, out, var))
93 if combine_method:
94 # TODO: add preservation of attrs into fillna
MergeError: conflicting values for variable 'roughness' on objects to be combined:
first value: <xarray.Variable (y: 112, x: 132)>
dask.array<shape=(112, 132), dtype=float32, chunksize=(112, 132)>
Attributes:
units: m
long_name: Forecast surface roughness
second value: <xarray.Variable (time: 8784, y: 112, x: 132)>
dask.array<shape=(8784, 112, 132), dtype=float32, chunksize=(1440, 112, 132)>
Attributes:
units: m
long_name: Forecast surface roughness
PV should work fine and fully parallelized even over world-spanning cutouts by using the relatively new map_blocks
in convert_pv.
module
into modules
?#77 (comment)
see discussion there
gc.collect
in prepare function due to possible file handling errors?Originally posted by @FabianHofmann in #77 (comment)
I have already filed the issue with pyproj-eur and was asked to file it here as well.
The error happens when importing atlite with pyproj version 1.9.6 installed.
Steps to reproduce the error:
import atlite
.Add ellps=XXX
(e.g. ellps='WGS84'
, if WGS84 is correct) to the call projection = RotProj(dict(proj='ob_tran', o_proj='latlong', lon_0=180, o_lon_p=-162, o_lat_p=39.25))
on line 41/42 in cordex.py
.
>>> import atlite
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\Svosw\miniconda3\envs\pyproj_test\lib\site-packages\atlite\__init__.py", line 26, in <module>
from .cutout import Cutout
File "C:\Users\Svosw\miniconda3\envs\pyproj_test\lib\site-packages\atlite\cutout.py", line 33, in <module>
from . import config, datasets
File "C:\Users\Svosw\miniconda3\envs\pyproj_test\lib\site-packages\atlite\datasets\__init__.py", line 3, in <module>
from . import cordex, ncep, era5, sarah
File "C:\Users\Svosw\miniconda3\envs\pyproj_test\lib\site-packages\atlite\datasets\cordex.py", line 42, in <module>
o_lon_p=-162, o_lat_p=39.25))
File "C:\Users\Svosw\miniconda3\envs\pyproj_test\lib\site-packages\pyproj\__init__.py", line 362, in __new__
return _proj.Proj.__new__(self, projstring)
File "_proj.pyx", line 129, in _proj.Proj.__cinit__
RuntimeError: b'major axis or radius = 0 or not given'
Occuring in new version branch v0.2
When running the standard pattern for creating a ERA5 cutout (see examples/create_cutout.py):
cutout = atlite.Cutout(name="europe-2011-01",
module="era5",
xs=slice(-12.18798349, 41.56244222),
ys=slice(71.65648314, 33.56459975),
years=slice(2011, 2011),
months=slice(1,1))
I get this exception when atlite tries to delete the temporary files downloaded for creating the cutout:
INFO:atlite.datasets.common:Deleting file C:\Users<User>\AppData\Local\Temp\tmpmsdmnmk8.nc
Exception ignored in: <finalize object at 0x25f484f2850; dead>
Traceback (most recent call last):
File "C:\anaconda\envs\atlite\lib\weakref.py", line 548, in call
return info.func(*info.args, **(info.kwargs or {}))
File "C:\Users<User>\Documents\GitHub\atlite\atlite\datasets\common.py", line 19, in noisy_unlink
os.unlink(path)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\Users\\AppData\Local\Temp\tmpmsdmnmk8.nc'
The error occurs multiple times for all the temp files created. They are not properly cleaned up.
Should be related to the new way of handling the files with:
and
-unrelated line of code removed-
RESkit, Severin Ryberg's atlite :), has three very interesting features that one should evaluate, whether they can be adopted or reused:
simulate
method work with xarray/dask.Together those two features would allow having a range of different turbines or the possibility to have a weak wind turbine as a second choice.
WARNING: Because the original data is provided every month, at the month boundaries there is untidiness if you use a time shift. The resulting xarray will have duplicates in the index for the parts of the day in each month at the boundary. You will have to re-average these based on the number of hours in each month for the duplicated day.
Hi Jonas, do you have an example of how to deal with the duplicated days? If not, I can try and add a function for it.
Hi, @euronion, I have several questions regarding the documentation of Atlite with Cordex dataset, which I think maybe suitable to put here.
Do you have any example/document of Atlite (the current version V0.0.2 or the upcoming V0.2) working with the Cordex data?
It seems that I should download all the 9 variables (influx, outflux, etc...) and store them into 9 folders in the same directory?
Do the variable "CWT" mean "compute working team"? I cannot find this variable in the Cordex output variable list.
Is it possible to prepare a cutout with only a subset of the 9 variables with Cordex?
Thank you in advance!
Originally posted by @fang1103 in #57 (comment)
Should we add the concept of a projection into this get_coords
function? Think rotpole transform for cordex for example.
I'd propose to have this as a separate issue, though!
Originally posted by @coroa in https://github.com/PyPSA/atlite/diffs
Hi,
i am running the following code:
CODE
import atlite
import logging
logging.basicConfig(level=logging.INFO)
cutout = atlite.Cutout(path="HH-2019.nc",
module="era5",
x=slice(8.3, 11.37),
y=slice(52.5, 54.32),
time=slice("2019-01","2019-12")
)
cutout.prepare()
ERROR
the download process is executed as described in the documentation, but then appears the following Error:
Traceback (most recent call last):
File "D:\Bachelorarbeit\conda\envs\atlite\lib\site-packages\atlite\data.py", line 146, in cutout_prepare
fd, tmp = mkstemp(suffix=filename, dir=directory)
File "D:\Bachelorarbeit\conda\envs\atlite\lib\tempfile.py", line 331, in mkstemp
return _mkstemp_inner(dir, prefix, suffix, flags, output_type)
File "D:\Bachelorarbeit\conda\envs\atlite\lib\tempfile.py", line 250, in _mkstemp_inner
fd = _os.open(file, flags, 0o600)
FileNotFoundError: [Errno 2] No such file or directory: 'HH-2019-01-01\\tmp1jwrobky02.nc'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "D:/Bachelorarbeit/Bachelorarbeit/tryouts/try_atlite.py", line 12, in <module>
cutout.prepare()
File "D:\Bachelorarbeit\conda\envs\atlite\lib\site-packages\atlite\data.py", line 170, in cutout_prepare
rmtree(tmpdir)
File "D:\Bachelorarbeit\conda\envs\atlite\lib\shutil.py", line 737, in rmtree
return _rmtree_unsafe(path, onerror)
File "D:\Bachelorarbeit\conda\envs\atlite\lib\shutil.py", line 615, in _rmtree_unsafe
onerror(os.unlink, fullname, sys.exc_info())
File "D:\Bachelorarbeit\conda\envs\atlite\lib\shutil.py", line 613, in _rmtree_unsafe
os.unlink(fullname)
PermissionError: [WinError 32]The process cannot access the file because it is being used by another process: 'C:\\Users\\cedko\\AppData\\Local\\Temp\\tmp5dg_g04i\\tmp7g_scis6.nc'
Is there any known Bug generating this Error or is my PC causing this Error?
I need to include the first and second standard parallels for the Albers Equal Area projection to work when calculating runoff.
runoff *= (1000. / 24.) * xr.DataArray(basins.shapes.to_crs(dict(proj="aea")).area)
should be
runoff *= (1000. / 24.) * xr.DataArray(basins.shapes.to_crs(dict(proj="aea", lat_1=35, lat_2=60)).area)
Iโm not sure if it is not an issue with the newer version, but I mentioned this here just in case.
As discussed with @euronion xarray supports sparse arrays since v0.14 see the release notes.
It works pretty decent though the functionality is not well documented.
For its use the sparse package has to be installed
The two functions I use the netallocation package are the following
import xarray as xr
from sparse import as_coo, COO
def as_sparse(ds):
"""
Convert dense dataset/dataarray into a sparse dataset.
Parameters
----------
ds : xr.Dataset or xr.DataArray
Returns
-------
xr.Dataset or xr.DataArray
Dataset or DataArray with sparse data.
"""
func = lambda data: COO(data) if not isinstance(data, COO) else data
return xr.apply_ufunc(func, ds)
def as_dense(ds):
"""
Convert sparse dataset/dataarray into a dense dataset.
Parameters
----------
ds : xr.Dataset or xr.DataArray
Returns
-------
xr.Dataset or xr.DataArray
Dataset or DataArray with dense data.
"""
func = lambda data: data.todense() if isinstance(data, COO) else data
return xr.apply_ufunc(func, ds)
Using a cutout
which is already prepared, i.e. with
> cutout.available_features == cutout.prepared_features
True
and then calling
cutout.prepare()
results in a lot of activity and finally some errors, e.g.
---------------------------------------------------------------------------
FileNotFoundError Traceback (most recent call last)
<ipython-input-4-eee9441f41e9> in <module>
----> 1 cutout.prepare()
~/miniconda/envs/atlite-v0.2/lib/python3.7/site-packages/atlite/data.py in cutout_prepare(cutout, features, monthly, overwrite)
204 if os.path.exists(cutout.cutout_fn):
205 os.unlink(cutout.cutout_fn)
--> 206 os.rename(target, cutout.cutout_fn)
207
208 # Re-open
FileNotFoundError: [Errno 2] No such file or directory: '/reanalysis/atlite/cutouts/tmp4zn24h_sv0.2_western-asia-2016-era5.nc' -> '../cutouts/v0.2_western-asia-2016-era5.nc'
I think the problem is the data.cutout_prepare(...)
function, which causes a lot of overhead processing, temp-file creation and other things, even if the missing_features
set at this point is empty
https://github.com/FRESNA/atlite/blob/9f6d0316f20ecd6fffe731d01cd731466845a801/atlite/data.py#L185-L186
I would expect this to run through cleanly in the blink of an eye.
Possibly reintroduce the "Cutout is already fully prepared" fast track from the earlier version?
Hello,
I seem to be having trouble creating a cutout for SARAH-2 data. I was able to get ERA5 to work fine, but for some reason SARAH-2 seems to be giving me problems.
Script:
`import logging
import atlite
logging.basicConfig(level=logging.DEBUG)
'''SARAH-2 DOESNT SEEM TO BE WORKING. ERA5 is okay'''
cutout = atlite.Cutout(name="SARAH_SthAfr_2011_01",
cutout_dir="/Users/lennon/Documents/GitHub/Sites/obeles.github.io/atlite/atlite/datasets/ERA5_datasets/cutouts",
module="sarah",
xs=slice(27.25, 32.25),
ys=slice(-30.75, -35.75),
years=slice(2011, 2011),
months=slice(1,1)
)
cutout.prepare()`
Script Error:
`INFO:atlite.cutout:Cutout SARAH_SthAfr_2011_01 not found in directory /Users/lennon/Documents/GitHub/Sites/obeles.github.io/atlite/atlite/datasets/ERA5_datasets/cutouts, building new one
Traceback (most recent call last):
File "/Users/lennon/Dropbox/Scripts/solar_atlite_cutout.py", line 24, in
months=slice(1,1)
File "/Users/lennon/anaconda3/envs/geo_env/lib/python3.7/site-packages/atlite/cutout.py", line 88, in init
self.meta = self.get_meta(**cutoutparams)
File "/Users/lennon/anaconda3/envs/geo_env/lib/python3.7/site-packages/atlite/preparation.py", line 171, in cutout_get_meta
ds = prepare_func(xs=xs, ys=ys, year=years.stop, month=months.stop, **meta_kwds)
File "/Users/lennon/anaconda3/envs/geo_env/lib/python3.7/site-packages/atlite/datasets/sarah.py", line 70, in prepare_meta_sarah
for t in (template_sis, template_sid)]
File "/Users/lennon/anaconda3/envs/geo_env/lib/python3.7/site-packages/atlite/datasets/sarah.py", line 70, in
for t in (template_sis, template_sid)]
StopIteration`
Suggested by @FabianHofmann during our recent work on the FOSS paper.
Implement an alternative to the currently used clearsky model based on Bayesian statistics.
n/a
Relevance/Importance = ?
Reference:
Philippe Lauret, John Boland, Barbara Ridley,
Bayesian statistical analysis applied to solar radiation modelling,
2013,
https://doi.org/10.1016/j.renene.2012.01.049.
Transmission capacities are influenced by the weather. We should add a conversion function for the cooling effect of transmission lines.
Goal:
Add a conversion function transmission
/transmission_availability
(or different) in atlite.convert
.
Idea: Assume a set of transmission lines given as line shapes.
indicator_matrix
if it works between properly between line shapes and area shapes. Otherwise define a intersection matrix (booleans would probably be sufficient) .atlite.aggregate
. For a first shot, we could instead use the mean temperature over the whole line.I encountered a slicing issue during cutout creation:
atlite\atlite\data.py in cutout_prepare(cutout, features, tmpdir, overwrite)
124
125 # target is series of all available variables for given module and features
--> 126 target = available_features(modules).loc[:, features].drop_duplicates()
127
128 for module in target.index.unique('module'):
~\.julia\conda\3\envs\atlite\lib\site-packages\pandas\core\indexing.py in __getitem__(self, key)
871 # AttributeError for IntervalTree get_value
872 pass
--> 873 return self._getitem_tuple(key)
874 else:
875 # we by definition only have the 0th axis
~\.julia\conda\3\envs\atlite\lib\site-packages\pandas\core\indexing.py in _getitem_tuple(self, tup)
1042 def _getitem_tuple(self, tup: Tuple):
1043 try:
-> 1044 return self._getitem_lowerdim(tup)
1045 except IndexingError:
1046 pass
~\.julia\conda\3\envs\atlite\lib\site-packages\pandas\core\indexing.py in _getitem_lowerdim(self, tup)
764 # we may have a nested tuples indexer here
765 if self._is_nested_tuple_indexer(tup):
--> 766 return self._getitem_nested_tuple(tup)
767
768 # we maybe be using a tuple to represent multiple dimensions here
~\.julia\conda\3\envs\atlite\lib\site-packages\pandas\core\indexing.py in _getitem_nested_tuple(self, tup)
832 # selectors
833 axis = self.axis or 0
--> 834 return self._getitem_axis(tup, axis=axis)
835
836 # handle the multi-axis by taking sections and reducing
~\.julia\conda\3\envs\atlite\lib\site-packages\pandas\core\indexing.py in _getitem_axis(self, key, axis)
1101 # nested tuple slicing
1102 if is_nested_tuple(key, labels):
-> 1103 locs = labels.get_locs(key)
1104 indexer = [slice(None)] * self.ndim
1105 indexer[axis] = locs
~\.julia\conda\3\envs\atlite\lib\site-packages\pandas\core\indexes\multi.py in get_locs(self, seq)
3110
3111 assert isinstance(indexer, Int64Index), type(indexer)
-> 3112 indexer = self._reorder_indexer(seq, indexer)
3113
3114 return indexer._values
~\.julia\conda\3\envs\atlite\lib\site-packages\pandas\core\indexes\multi.py in _reorder_indexer(self, seq, indexer)
3139 if is_list_like(k):
3140 if not need_sort:
-> 3141 k_codes = self.levels[i].get_indexer(k)
3142 k_codes = k_codes[k_codes >= 0] # Filter absent keys
3143 # True if the given codes are not ordered
~\.julia\conda\3\envs\atlite\lib\site-packages\pandas\core\indexes\base.py in get_indexer(self, target, method, limit, tolerance)
3004 )
3005
-> 3006 indexer = self._engine.get_indexer(target._get_engine_target())
3007
3008 return ensure_platform_int(indexer)
pandas\_libs\index.pyx in pandas._libs.index.IndexEngine.get_indexer()
pandas\_libs\hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.lookup()
TypeError: unhashable type: 'slice'
Caused by this line
Lines 125 to 126 in ba5daae
due to the multi-index access in .loc
with an array of an empty slice.
Removing
Line 121 in ba5daae
(for which I don't see a need anyways) resolved this issue.
Can you reproduce @FabianHofmann ?
Was there a change in pandas
?
Referencing #107 for pandas
version.
I am unable to locally install the current version with pip
(pip install .
).
I am using the environment.yaml
file provided.
The following error message pops up when trying install it:
(atlite) pypsa@vm-pypsa:~/share/GitHub/atlite$ git remote -v
origin https://github.com/PyPSA/atlite.git (fetch)
origin https://github.com/PyPSA/atlite.git (push)
(atlite) pypsa@vm-pypsa:~/share/GitHub/atlite$ git status
On branch v0.2
Your branch is up-to-date with 'origin/v0.2'.
nothing to commit, working tree clean
(atlite) pypsa@vm-pypsa:~/share/GitHub/atlite$ pip install .
Processing /home/pypsa/share/GitHub/atlite
ERROR: Command errored out with exit status 1:
command: /home/pypsa/miniconda3/envs/atlite/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-egexmm80/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-egexmm80/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-req-build-egexmm80/pip-egg-info
cwd: /tmp/pip-req-build-egexmm80/
Complete output (27 lines):
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-req-build-egexmm80/setup.py", line 52, in <module>
'Operating System :: OS Independent',
File "/home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages/setuptools/__init__.py", line 160, in setup
_install_setup_requires(attrs)
File "/home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages/setuptools/__init__.py", line 155, in _install_setup_requires
dist.fetch_build_eggs(dist.setup_requires)
File "/home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages/setuptools/dist.py", line 698, in fetch_build_eggs
replace_conflicting=True,
File "/home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages/pkg_resources/__init__.py", line 783, in resolve
replace_conflicting=replace_conflicting
File "/home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1066, in best_match
return self.obtain(req, installer)
File "/home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1078, in obtain
return installer(requirement)
File "/home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages/setuptools/dist.py", line 754, in fetch_build_egg
return fetch_build_egg(self, req)
File "/home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages/setuptools/installer.py", line 133, in fetch_build_egg
wheel.install_as_egg(dist_location)
File "/home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages/setuptools/wheel.py", line 99, in install_as_egg
self._install_as_egg(destination_eggdir, zf)
File "/home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages/setuptools/wheel.py", line 107, in _install_as_egg
self._convert_metadata(zf, destination_eggdir, dist_info, egg_info)
File "/home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages/setuptools/wheel.py", line 128, in _convert_metadata
os.mkdir(destination_eggdir)
FileExistsError: [Errno 17] File exists: '/tmp/pip-req-build-egexmm80/.eggs/setuptools_scm-4.1.2-py3.7.egg'
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
Installing it directly from GitHub works fine, i.e. this works w/o problems:
(atlite) pypsa@vm-pypsa:~/share/GitHub/atlite$ pip install git+https://github.com/pypsa/[email protected]
Collecting git+https://github.com/pypsa/[email protected]
Cloning https://github.com/pypsa/atlite.git (to revision v0.2) to /tmp/pip-req-build-v4bvq5v1
Running command git clone -q https://github.com/pypsa/atlite.git /tmp/pip-req-build-v4bvq5v1
Running command git checkout -b v0.2 --track origin/v0.2
Switched to a new branch 'v0.2'
Branch 'v0.2' set up to track remote branch 'v0.2' from 'origin'.
Requirement already satisfied: numpy in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from atlite==0.0.4.dev330+g2fbdd9f) (1.18.1)
Requirement already satisfied: scipy in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from atlite==0.0.4.dev330+g2fbdd9f) (1.4.1)
Requirement already satisfied: pandas>=0.22 in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from atlite==0.0.4.dev330+g2fbdd9f) (1.0.3)
Requirement already satisfied: bottleneck in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from atlite==0.0.4.dev330+g2fbdd9f) (1.3.2)
Requirement already satisfied: numexpr in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from atlite==0.0.4.dev330+g2fbdd9f) (2.7.1)
Requirement already satisfied: xarray>=0.11.2 in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from atlite==0.0.4.dev330+g2fbdd9f) (0.15.1)
Requirement already satisfied: netcdf4 in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from atlite==0.0.4.dev330+g2fbdd9f) (1.4.2)
Requirement already satisfied: dask>=0.18.0 in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from atlite==0.0.4.dev330+g2fbdd9f) (2.17.2)
Requirement already satisfied: requests in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from atlite==0.0.4.dev330+g2fbdd9f) (2.23.0)
Requirement already satisfied: rtree in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from atlite==0.0.4.dev330+g2fbdd9f) (0.9.4)
Requirement already satisfied: pyyaml in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from atlite==0.0.4.dev330+g2fbdd9f) (5.3.1)
Requirement already satisfied: rasterio>=1.0 in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from atlite==0.0.4.dev330+g2fbdd9f) (1.1.0)
Requirement already satisfied: shapely in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from atlite==0.0.4.dev330+g2fbdd9f) (1.7.0)
Requirement already satisfied: progressbar2 in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from atlite==0.0.4.dev330+g2fbdd9f) (3.37.1)
Requirement already satisfied: geopandas in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from atlite==0.0.4.dev330+g2fbdd9f) (0.6.1)
Requirement already satisfied: cdsapi in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from atlite==0.0.4.dev330+g2fbdd9f) (0.2.7)
Requirement already satisfied: pytz>=2017.2 in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from pandas>=0.22->atlite==0.0.4.dev330+g2fbdd9f) (2020.1)
Requirement already satisfied: python-dateutil>=2.6.1 in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from pandas>=0.22->atlite==0.0.4.dev330+g2fbdd9f) (2.8.1)
Requirement already satisfied: setuptools>=41.2 in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from xarray>=0.11.2->atlite==0.0.4.dev330+g2fbdd9f) (47.1.1.post20200604)
Requirement already satisfied: cftime in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from netcdf4->atlite==0.0.4.dev330+g2fbdd9f) (1.1.2)
Requirement already satisfied: idna<3,>=2.5 in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from requests->atlite==0.0.4.dev330+g2fbdd9f) (2.9)
Requirement already satisfied: chardet<4,>=3.0.2 in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from requests->atlite==0.0.4.dev330+g2fbdd9f) (3.0.4)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from requests->atlite==0.0.4.dev330+g2fbdd9f) (1.25.8)
Requirement already satisfied: certifi>=2017.4.17 in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from requests->atlite==0.0.4.dev330+g2fbdd9f) (2020.4.5.1)
Requirement already satisfied: attrs in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from rasterio>=1.0->atlite==0.0.4.dev330+g2fbdd9f) (19.3.0)
Requirement already satisfied: snuggs>=1.4.1 in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from rasterio>=1.0->atlite==0.0.4.dev330+g2fbdd9f) (1.4.7)
Requirement already satisfied: cligj>=0.5 in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from rasterio>=1.0->atlite==0.0.4.dev330+g2fbdd9f) (0.5.0)
Requirement already satisfied: affine in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from rasterio>=1.0->atlite==0.0.4.dev330+g2fbdd9f) (2.3.0)
Requirement already satisfied: click<8,>=4.0 in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from rasterio>=1.0->atlite==0.0.4.dev330+g2fbdd9f) (7.1.2)
Requirement already satisfied: click-plugins in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from rasterio>=1.0->atlite==0.0.4.dev330+g2fbdd9f) (1.1.1)
Requirement already satisfied: python-utils>=2.3.0 in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from progressbar2->atlite==0.0.4.dev330+g2fbdd9f) (2.3.0)
Requirement already satisfied: six in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from progressbar2->atlite==0.0.4.dev330+g2fbdd9f) (1.15.0)
Requirement already satisfied: fiona in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from geopandas->atlite==0.0.4.dev330+g2fbdd9f) (1.8.11)
Requirement already satisfied: pyproj in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from geopandas->atlite==0.0.4.dev330+g2fbdd9f) (2.6.1.post1)
Requirement already satisfied: tqdm in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from cdsapi->atlite==0.0.4.dev330+g2fbdd9f) (4.46.0)
Requirement already satisfied: pyparsing>=2.1.6 in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from snuggs>=1.4.1->rasterio>=1.0->atlite==0.0.4.dev330+g2fbdd9f) (2.4.7)
Requirement already satisfied: munch in /home/pypsa/miniconda3/envs/atlite/lib/python3.7/site-packages (from fiona->geopandas->atlite==0.0.4.dev330+g2fbdd9f) (2.5.0)
Building wheels for collected packages: atlite
Building wheel for atlite (setup.py) ... done
Created wheel for atlite: filename=atlite-0.0.4.dev330+g2fbdd9f-py2.py3-none-any.whl size=63453 sha256=d0a94b784da2cb309c2b0913df8c9e52023788bf8eaf852cad4ee594e5c033e9
Stored in directory: /tmp/pip-ephem-wheel-cache-53fc5lbo/wheels/8b/16/63/70c28877db9b6c491a9550dbcc78dd21b3704cc10c6d9ca4f8
Successfully built atlite
Installing collected packages: atlite
Successfully installed atlite-0.0.4.dev330+g2fbdd9f
Is this intended behaviour?
I presume it is related to the changes with how the versioning works.
The coordinates in SARAH cutouts are messed up, i.e. mirrored, see the image below.
> albedo
<xarray.DataArray 'albedo' (y: 160, x: 270)>
array([[0.04827 , 0.048171, 0.048043, ..., 0.103032, 0.104033, 0.104248],
[0.047978, 0.047966, 0.047898, ..., 0.104392, 0.109037, 0.11042 ],
[0.047583, 0.04764 , 0.047696, ..., 0.105399, 0.109951, 0.111676],
...,
[0.03567 , 0.03552 , 0.035488, ..., 0.353771, 0.35935 , 0.36295 ],
[0.035606, 0.035525, 0.035507, ..., 0.356749, 0.357823, 0.359936],
[0.035569, 0.0355 , 0.035465, ..., 0.360018, 0.361162, 0.359274]],
dtype=float32)
Coordinates:
time datetime64[ns] 2013-06-01T12:00:00
* x (x) float64 -12.0 -11.8 -11.6 -11.4 -11.2 ... 41.2 41.4 41.6 41.8
* y (y) float64 33.0 33.2 33.4 33.6 33.8 ... 64.0 64.2 64.4 64.6 64.8
lon (x) float64 -12.0 -11.8 -11.6 -11.4 -11.2 ... 41.2 41.4 41.6 41.8
lat (y) float64 33.0 33.2 33.4 33.6 33.8 ... 64.0 64.2 64.4 64.6 64.8
Attributes:
units: (0 - 1)
long_name: Albedo
The cutout
> cutout
<Cutout europe-2013-sarah x=-12.00-41.80 y=33.00-64.80 time=2013-01-01-2013-12-31 prepared_features=['temperature', 'influx'] is_view=False>
with the messed up coordinates
> cutout.data
<xarray.Dataset>
Dimensions: (time: 8760, x: 270, y: 160)
Coordinates:
* time (time) datetime64[ns] 2013-01-01 ... 2013-12-31T23:00:00
* x (x) float64 -12.0 -11.8 -11.6 -11.4 ... 41.2 41.4 41.6 41.8
* y (y) float64 33.0 33.2 33.4 33.6 33.8 ... 64.2 64.4 64.6 64.8
lon (x) float64 -12.0 -11.8 -11.6 -11.4 ... 41.2 41.4 41.6 41.8
lat (y) float64 33.0 33.2 33.4 33.6 33.8 ... 64.2 64.4 64.6 64.8
Data variables:
temperature (time, y, x) float32 ...
influx_toa (time, y, x) float32 ...
albedo (time, y, x) float32 ...
influx_direct (time, y, x) float32 ...
influx_diffuse (time, y, x) float32 ...
Attributes:
module: sarah
prepared_features: ['temperature', 'influx']
creation_parameters: {'resolution': 0.2, 'x': slice(-12.0, 42.0, None), ...
Maybe an Australian PoV?
Add SPHINX extension embedding capability for jupyter notebooks.
Unfortunately the file handler _file_obj
we use in
Line 173 in b269238
was removed in the latest xarray version (was just released few days ago). This line crashed when updating xarray.
The cutout preparation fails on windows machines as netcdf files are handled differently and want to be closed manually. Unfortunately the file handling on the current branch v0.2 does not allow this in an easy way. I disabled the travis on windows machines. This should however be solved for v0.2
I am currently trying to produce a hourly capacity factor time series but am running into some issues. It seems that the only way that atlite processes hourly time series is for energy generation, and that for capacity factors it only produces the average. Is there a specification in the parameters that could allow for an hourly capacity factor generation instead? This is the code Im working with:
pv_CF = cutout.pv(pv_model, capacity_factor=True , orientation={'slope': cutout_lat, 'azimuth': 0})
I am using ERA5 runoff data to determine inflow time series for hydro power stations. The time series contain large negative values (-1e5) which look odd as they physically don't make sense.
I tried to trace them back to their origin. I am not exactly sure where and why this happens, but I am pretty sure it happens within cutout.prepare()
: I ran a comparison of data downloaded with atlite
and data downloaded directly from CDS
and found that those values deviate slightly and that runoff data is always >= 0 for raw CDS data, but can be <0 for atlite data.
The differences are tiny and very likely based on numerics. I am still reporting this here for three reasons: First, they may be tiny after download, but blow up to -1e5 in my case for inflow data. Second, because the data is not processed and because it should always be stored as float32, I don't exactly understand why the data changes. Third, this report may be of help for others running into the same issue.
Atlite version: 62897
Keeping this here as a reminder. Not a pressing issue, but something that has been bothering me for some time.
I have not yet observed that atlite
is using more than one processing core for calculations (e.g. wind speed to wind power conversion).
I would assume that this should be easy to implement (as in the backend we are using dask
and xarray
, but I haven't been able to identify the culprit.
Initially I was only monitoring the CPU util. using htop
:
There only one core seems to be active.
A better way is to use a dask
dashboard and a local client:
from dask.distributed import Client
import xarray as xr
client = Client()
client
I also do not see any dask activity during calculations there.
When checking cutout.data
this returns an xarray
Dataset using numpy arrays.
When loading the dataset (xarray.open_dataset(...)
) with the chunks
option or rechunking the dataset after the cutout was loaded using cutout.data.chunk(...)
then shows some dask activity during calculations (but slow, I guess a lot of overhead from spawning and orchestrating the different workers and threads).
With the new version #20 calculations have become significantly faster and caching of datasets obsolete (can now be done implicitly by xarray and dask).
So this is only interesting when doing conversion of large datasets or repeated conversions.
> from platform import python_version
> python_version()
3.7.3
> import xarray
> xarray.__version__
'0.12.3'
> import dask
> dask.__version__
'2.1.0'
As tackled in PR #69 the shapely strtree
is a better choice for filtering out shapes that are not overlapping. The change in v0.2
foresees to store the sindex of the grid cells in order to faster calculate the indicator matrix. This becomes obsolete when defining the new "sindex" via strtree, see here
len(cutout.grid_cells)
Out[7]: 20698
%time STRtree(cutout.grid_cells)
CPU times: user 70.7 ms, sys: 3.76 ms, total: 74.4 ms
Wall time: 72.3 ms
Out[8]: <shapely.strtree.STRtree at 0x7fdc3db0be50>
(the grid cells here are the standard ones for pypsa-eur)
Compare this to the previous sindex calculation of ~1.5 seconds.
Sooo, this raises the question, if the GridCells
class is really necessary. It seems the only purpose for its introduction was to store the sindex, which does not work properly with pickle.dump and which anyway becomes obsolete now.
@coroa I know you are buisy, but perhaps can just shortly give a hint whether there was another intention when you introduced the GridCells
class.
I would like to use atlite with the EFAS data recently released on the Copernicus Data Store. Do you have a tutorial or some docs on how to "add" another input dataset?
The current setup method requires
config.py
The problems I have with this:
a. It's rather cumbersome and prone to errors
b. Requires reinstallation whenever directories from config.py
change
c. Is not correctly reflected in the README.rst
It seems to me it might be better to integrate the few config parameters as function parameters.
Or am I missing something?
Different topic:
I failed to run cutout.pv(...)
for the last days and found the problem to be insufficient memory (the script ran slowly and crashed at one point in time without a tractable error).
I solved it with windows="D"
, which requires less data to be loaded into memory at the same time.
@coroa : What do you think about making windows="D"
default for the convert operations?
Originally posted by @euronion in #20 (comment)
Since we're still having some trouble to properly intergrate v0.2 (#20), it would be helpful to add a test script which canvasses major relations.
Atm I'm trying to figure out how this would look like. Some things I have in mind:
This list should continue. I'm starting to write down some test files. Feel free to add to the list
There is a small bug when loading the static variable only, i.e. 'height' with era5. This will have the time dimension of the first time parameters of the first request. I'll suggest a solution
@hailiangliu89 and me are currently running into quite a few problems with installed dependencies.
Until now I think we were able to solve all the problems by using a different source for installation (or switching OS).
I don't think the causes can be solved from within atlite
.
Is there a good way to cope with these problems?
The two obvious possibilities for me:
Approach 1. would be more convinient, but I have the feeling that most problems are caused by the different, individual configurations.
This leaves approach 2., which would also be more versatile (and wouldn't require regular updates).
Any ideas?
Example:
For me installing pyshp
from conda-forge
causes problems, installing via pip
works fine.
The same problem with geopandas
.
Cheers.
We do have (with the new version #20 ) a new configuration system in the pipeline.
It does however have a few downsides which became apparent during further developments, see e.g. the discussion here, where an observer can not trigger on changes of the module variables.
I don't think
__setattr__
is called on module level (but feel free to try, i am unsure).
We could package everything up into an object! :)Thought for the future:
I worked on it in https://github.com/euronion/atlite/blob/f1e039abf9c5295ac549d0c39152beff7697c418/atlite/config.py
and had a few things I could not figure out and dropped it.
(like not calling the config asatlite.config.config
and making sure the object is shared > among all modules when a new config-object is used to overwrite the existing one - in short:
The current solution is more robust).
Originally posted by @euronion in #43 (comment)
It would be nice to implement the config as a true object.
RLI built a list of 139 recent wind turbines collected from 11 different sources. They re-published them as a table on their oedb under an ODbL.
We should provide a small adaptor and instructions for querying this table and using it in atlite.
saio is a great help for that.
@coroa pointed out that the heat_demand might have problems when chunking (at least there were some issues with windows). A time shift in the time coordinates might lead to loosing values at the border of chunks. This seems to be no problem for dask arrays:
When looking at the results of chunked and non-chunked datasets, the result for heat_demand
is exactly the same.
Originally posted by @euronion in #20 (comment)
grid_cells
and grid_coordinates()
I am thinking more along the line of: Are people expecting the order in the lists returned by cutout.grid_cells
and cutout.grid_coordinates()
to be reliable or not.
Maybe it does cancel out in PyPSA-EUR
or one codes it independent of this order (I just did so).
But maybe people are expecting the order to be consistent between version, i.e. then we are changing the API behaviour in an unexpected way -> we should at lteast add this to the release notes.
The code with a more hideous and less obvious problem I am refering to is this one [1]:
profile, capacities = func(
# Redindex, stack index and conv. to CSR to be consistent with the format expected by atlite
matrix=layoutmatrix.reindex_like(
cutout.data).stack(spatial=('y', 'x')).data.tocsr(),
index=buses,
per_unit=True,
return_capacity=True,
show_progress='Compute profiles: ',
**resource)
More specifically how the matrix
passed to atlite
has to be structured. This matrix is 2D, where one dimension is reserved for the index=buses
and the other dimension is a .stack()
-ed version of a 2D array.
Now the behaviour of .stack()
depends on the order of the index, that's why the .reindex_like(...)
is relevant here. In the version before it looked like this [2]:
profile, capacities = func(matrix=layoutmatrix, index=buses, per_unit=True,
return_capacity=True, show_progress='Compute profiles: ',
**resource)
where the matrix=layoutmatrix
as implicitly assuming the 2D array of values refering to the coordinates in the cutout to be stacked with descending y
and ascending x
. That was the old atlite
structure.
If I use the same code with the new atlite=v0.2
the result is different, because we assume (inside atlite
!) that this matrix is stacked with ascending y
and x
.
It boils down to
cutout.sortby('y', ascending=True).stack(spatial=('y','x'))
not being the same as
cutout.sortby('y', ascending=False).stack(spatial=('y','x'))
but we use
cutout.stack(spatial=('y','x')
here
Line 31 in 7d60f68
The easiest solution could be to guarantee the old behaviour by fixing the index order of created cutouts during their creation
cutout.data = cutout.data.sortby('y', ascending=False)
cutout.data = cutout.data.sortby('x', ascending=True)
This would allow us to make the change in the .Cutout(...)
signature while ensuring backwards compatability.
[1] Source: New code in PyPSA-EUR
for the upcoming atlite
version
https://github.com/PyPSA/pypsa-eur/blob/283042d1dd3cd11b5313479c7ca03cbe58f778c9/scripts/build_renewable_profiles.py#L390-L398
[2] Source: https://github.com/PyPSA/pypsa-eur/blob/bb3477cd693d1c8e77e75e61a1a7e1a4647e6a3c/scripts/build_renewable_profiles.py#L303-L305
The original sarah datasets, SID and SIS, have both a band of nans along the earth where dawn and nightfall takes place. On one side you find mostly values > 0, on the other only values of 0.
Atm we are extensively interpolating those areas with a 1d interpolation, which for large cutouts takes a lot of time. The question raises, is the interpolation necessary.
In pypsa-eur the parallel processing of the availability matrix results in weird overlaps of the progessbars. Very likely the tqdm progressbar can handle this better. According to https://stackoverflow.com/questions/41920124/multiprocessing-use-tqdm-to-display-a-progress-bar something like this should work
from tqdm.contrib.concurrent import process_map # or thread_map
availability = process_map(_process_func, shapes.index, max_workers=nprocesses)
Allow for the back extension of ERA5 in upcoming v0.2 (#104)
Encountering the following error when creating a cutout (ERA5):
INFO:cdsapi:Download rate 3.8M/s
[ ] | 0% Completed | 0.0s
[########################################] | 100% Completed | 0.3s
---------------------------------------------------------------------------
PermissionError Traceback (most recent call last)
<ipython-input-5-3416e809141b> in <module>
8
9 # This is where all the work happens (this can take some time, for us it took ~15 minutes).
---> 10 cutout.prepare()
c:\users\j. hampp\documents\github\atlite\atlite\data.py in cutout_prepare(cutout, features, freq, tmpdir, overwrite)
226 finally:
227 if not keep_tmpdir:
--> 228 rmtree(tmpdir)
229
230 # Re-open
C:\anaconda\envs\atlite\lib\shutil.py in rmtree(path, ignore_errors, onerror)
511 # can't continue even if onerror hook returns
512 return
--> 513 return _rmtree_unsafe(path, onerror)
514
515 # Allow introspection of whether or not the hardening against symlink
C:\anaconda\envs\atlite\lib\shutil.py in _rmtree_unsafe(path, onerror)
395 os.unlink(fullname)
396 except OSError:
--> 397 onerror(os.unlink, fullname, sys.exc_info())
398 try:
399 os.rmdir(path)
C:\anaconda\envs\atlite\lib\shutil.py in _rmtree_unsafe(path, onerror)
393 else:
394 try:
--> 395 os.unlink(fullname)
396 except OSError:
397 onerror(os.unlink, fullname, sys.exc_info())
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\J705E~1.HAM\\AppData\\Local\\Temp\\tmplaut1buy\\tmp6jjdud34.nc'
cutout.data
data variables are available.Uh annoying, they worked wonders in Python 2.7. The default has been changed to ignore them in 3.2
unless you add -Wd
to your python interpreter or change the filter as you've shown. Hmm .. if I understand the manual correctly it's not intended for libraries to set captureWarnings
or change the filter. I suppose we should warn in the manual about that and change the examples accordingly.
Do you think logging.captureWarnings(True)
or warnings.simplefilter('always', DeprecationWarning)
should be used? I'm tending towards the former.
When preparing a new cutout with cutout.prepare()
, the above assertion error is thrown.
I use this example with the relevant line(s):
# Define the cutout; this will not yet trigger any major operations
cutout = atlite.Cutout(name="uk-2011-01-2",
module="era5",
bounds=UkIr.unary_union.bounds,
time="2011-01")
# This is where all the work happens (this can take some time, for us it took ~15 minutes).
cutout.prepare()
Full trace:
INFO:cdsapi:Request is completed
---------------------------------------------------------------------------
AssertionError Traceback (most recent call last)
<ipython-input-6-898cb53315e6> in <module>
6
7 # This is where all the work happens (this can take some time, for us it took ~15 minutes).
----> 8 cutout.prepare()
c:\users\j. hampp\documents\github\atlite\atlite\data.py in cutout_prepare(cutout, features, freq, tmpdir, overwrite)
201 return
202
--> 203 ds = get_missing_data(cutout, missing_features, freq, tmpdir=tmpdir)
204
205 # Merge with existing cutout
c:\users\j. hampp\documents\github\atlite\atlite\data.py in get_missing_data(cutout, features, freq, tmpdir)
163 datasets.append(feature_data)
164
--> 165 datasets, = dask.compute(datasets)
166
167 return xr.merge(datasets, compat='equals')
C:\anaconda\envs\atlite\lib\site-packages\dask\base.py in compute(*args, **kwargs)
444 keys = [x.__dask_keys__() for x in collections]
445 postcomputes = [x.__dask_postcompute__() for x in collections]
--> 446 results = schedule(dsk, keys, **kwargs)
447 return repack([f(r, *a) for r, (f, a) in zip(results, postcomputes)])
448
C:\anaconda\envs\atlite\lib\site-packages\dask\threaded.py in get(dsk, result, cache, num_workers, pool, **kwargs)
80 get_id=_thread_get_id,
81 pack_exception=pack_exception,
---> 82 **kwargs
83 )
84
C:\anaconda\envs\atlite\lib\site-packages\dask\local.py in get_async(apply_async, num_workers, dsk, result, cache, get_id, rerun_exceptions_locally, pack_exception, raise_exception, callbacks, dumps, loads, **kwargs)
489 _execute_task(task, data) # Re-execute locally
490 else:
--> 491 raise_exception(exc, tb)
492 res, worker_id = loads(res_info)
493 state["cache"][key] = res
C:\anaconda\envs\atlite\lib\site-packages\dask\compatibility.py in reraise(exc, tb)
128 if exc.__traceback__ is not tb:
129 raise exc.with_traceback(tb)
--> 130 raise exc
131
132 import pickle as cPickle
C:\anaconda\envs\atlite\lib\site-packages\dask\local.py in execute_task(key, task_info, dumps, loads, get_id, pack_exception)
231 try:
232 task, data = loads(task_info)
--> 233 result = _execute_task(task, data)
234 id = get_id()
235 result = dumps((result, id))
C:\anaconda\envs\atlite\lib\site-packages\dask\core.py in _execute_task(arg, cache, dsk)
117 func, args = arg[0], arg[1:]
118 args2 = [_execute_task(a, cache) for a in args]
--> 119 return func(*args2)
120 elif not ishashable(arg):
121 return arg
c:\users\j. hampp\documents\github\atlite\atlite\datasets\common.py in get_data_gebco_height(xs, ys, gebco_path)
91 '-r', 'average',
92 gebco_path, target])
---> 93 assert ret == 0, "gdalwarp was not able to resample gebco"
94 except OSError:
95 logger.warning("gdalwarp was not found for resampling gebco. "
AssertionError: gdalwarp was not able to resample gebco
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.