Pyspectral is a package to read and manipulate satellite sensor spectral responses and solar irradiance spectra
Home Page: https://pyspectral.readthedocs.org/
License: GNU General Public License v3.0
This is the sandbox area for the pytroll project, an international cooperation on a future distributed real-time processing system for Meteorological Satellite Data. ------------------------------------ December 2010 Lars Ørum Rasmussen, Esben Sigård Nielsen, Kristian Rune Larsen, Martin Raspaud, Anna Geidne, Adam Dybbroe. Danish Meteorological Institute (DMI) Swedish Meteorological and Hydrological Institute (SMHI)
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
![pre-commit-ci[bot] avatar](https://avatars.githubusercontent.com/in/68672?v=4 "pre-commit-ci[bot]")
Hello, I'm interested in adding the ability to process Hyperion data. Could you give point me towards any resources or explanations that would help in adding this? Thanks!
from satpy import Scene
fname = "hrpt_noaa18_20180731_0934_67992.l1b"
glbl = Scene(reader='avhrr_aapp_l1b', filenames=[fname,])
glbl.load(['day_microphysics'])Use of PySpectral via SatPy for AVHRR data fails. The above script works fine for MSG/SEVIRI data.
Traceback relevant to PySpectral:
[WARNING: 2018-08-09 12:04:56 : pyspectral.rsr_reader] Inconsistent instrument/satellite input - instrument set to avhrr/3
[DEBUG: 2018-08-09 12:04:56 : pyspectral.rsr_reader] Filename: /home/lahtinep/.local/share/pyspectral/rsr_avhrr3_NOAA-18.h5
[WARNING: 2018-08-09 12:04:56 : pyspectral.rsr_reader] No rsr file /home/lahtinep/.local/share/pyspectral/rsr_avhrr3_NOAA-18.h5 on disk
[INFO: 2018-08-09 12:04:56 : pyspectral.rsr_reader] Will download from internet...
[DEBUG: 2018-08-09 12:04:57 : urllib3.connectionpool] Starting new HTTPS connection (1): zenodo.org
[DEBUG: 2018-08-09 12:04:57 : urllib3.connectionpool] https://zenodo.org:443 "GET /record/1205138/files/pyspectral_rsr_data.tgz HTTP/1.1" 200 2798376
2798376it [00:02, 1097715.00it/s]
[WARNING: 2018-08-09 12:05:00 : pyspectral.rsr_reader] rsr data may not be up to date: /home/lahtinep/.local/share/pyspectral/rsr_avhrr3_NOAA-18.h5
[INFO: 2018-08-09 12:05:00 : pyspectral.rsr_reader] Will download from internet...
[DEBUG: 2018-08-09 12:05:00 : urllib3.connectionpool] Starting new HTTPS connection (1): zenodo.org
[DEBUG: 2018-08-09 12:05:01 : urllib3.connectionpool] https://zenodo.org:443 "GET /record/1205138/files/pyspectral_rsr_data.tgz HTTP/1.1" 200 2798376
2798376it [00:02, 1095147.97it/s]
[DEBUG: 2018-08-09 12:05:04 : pyspectral.rsr_reader] Filename: /home/lahtinep/.local/share/pyspectral/rsr_avhrr3_NOAA-18.h5
[DEBUG: 2018-08-09 12:05:04 : pyspectral.rsr_reader] No detectors found - assume only one...
[WARNING: 2018-08-09 12:05:04 : pyspectral.rsr_reader] Inconsistent instrument/satellite input - instrument set to avhrr/3
[DEBUG: 2018-08-09 12:05:04 : pyspectral.rsr_reader] Filename: /home/lahtinep/.local/share/pyspectral/rsr_avhrr3_NOAA-18.h5
[DEBUG: 2018-08-09 12:05:04 : pyspectral.rsr_reader] Filename: /home/lahtinep/.local/share/pyspectral/rsr_avhrr3_NOAA-18.h5
[DEBUG: 2018-08-09 12:05:04 : pyspectral.rsr_reader] No detectors found - assume only one...
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
<ipython-input-31-2c22589a44ad> in <module>()
----> 1 glbl.load(['day_microphysics'])
~/Software/pytroll/packages/satpy/satpy/scene.py in load(self, wishlist, calibration, resolution, polarization, level, generate, unload, **kwargs)
848 self.read(**kwargs)
849 if generate:
--> 850 keepables = self.generate_composites()
851 else:
852 # don't lose datasets we loaded to try to generate composites
~/Software/pytroll/packages/satpy/satpy/scene.py in generate_composites(self, nodes)
763 nodes = set(self.dep_tree.trunk(nodes=required_nodes)) - \
764 set(self.datasets.keys())
--> 765 return self._read_composites(nodes)
766
767 def _remove_failed_datasets(self, keepables):
~/Software/pytroll/packages/satpy/satpy/scene.py in _read_composites(self, compositor_nodes)
737 keepables = set()
738 for item in compositor_nodes:
--> 739 self._generate_composite(item, keepables)
740 return keepables
741
~/Software/pytroll/packages/satpy/satpy/scene.py in _generate_composite(self, comp_node, keepables)
710 composite = compositor(prereq_datasets,
711 optional_datasets=optional_datasets,
--> 712 **self.attrs)
713
714 cid = DatasetID.from_dict(composite.attrs)
~/Software/pytroll/packages/satpy/satpy/composites/__init__.py in __call__(self, projectables, optional_datasets, **info)
468 for wavelength outside [3, 4] µm.
469 """
--> 470 self._init_refl3x(projectables)
471 _nir, _ = projectables
472 refl = self._get_reflectance(projectables, optional_datasets) * 100
~/Software/pytroll/packages/satpy/satpy/composites/__init__.py in _init_refl3x(self, projectables)
489
490 _nir, _tb11 = projectables
--> 491 self._refl3x = Calculator(_nir.attrs['platform_name'], _nir.attrs['sensor'], _nir.attrs['name'])
492
493 def _get_reflectance(self, projectables, optional_datasets):
~/Software/miniconda3/lib/python3.6/site-packages/pyspectral/near_infrared_reflectance.py in __init__(self, platform_name, instrument, band, **kwargs)
59
60 def __init__(self, platform_name, instrument, band, **kwargs):
---> 61 super(Calculator, self).__init__(platform_name, instrument, band, **kwargs)
62
63 from numbers import Number
~/Software/miniconda3/lib/python3.6/site-packages/pyspectral/radiance_tb_conversion.py in __init__(self, platform_name, instrument, band, **options)
126 self.rsr_integral = 1.0
127
--> 128 self._get_rsr()
129
130 def _get_rsr(self):
~/Software/miniconda3/lib/python3.6/site-packages/pyspectral/radiance_tb_conversion.py in _get_rsr(self)
152 self.bandname = get_bandname_from_wavelength(self.instrument, self.band, self.rsr)
153
--> 154 self.wavelength_or_wavenumber = (self.rsr[self.bandname][self.detector][self.wavespace] *
155 self._wave_si_scale)
156 self.response = self.rsr[self.bandname][self.detector]['response']
KeyError: '3b'Here '3b' is the channel name SatPy uses for 3.8 um AVHRR channel. Looking at the HDF5 file in ~/.local/share/pyspectral/ the group ("folder") names are ch1, ... ch3b, ... ch5, so there is an inconsistency in the channel naming between SatPy and PySpectral.
Conda, Python 3.6, current PySpectral master branch, current SatPy master branch.
The radiance to tb LUT is bound between 150K and 360 K (hard coded in radiance_tb_conversion.py) and no check is done in case the brightness temperature is outside this range. So an IndexError is raised.
This was seen to be a problem with Himawari-8 data from Chris
himawari branch.
from satpy import Scene, find_files_and_readers
from satpy.utils import debug_on
debug_on()
files = find_files_and_readers(base_dir='/home/a000680/data/hrit/20180102',
reader='seviri_l1b_hrit')
scn = Scene(filenames=files)
composite = 'day_microphysics'
scn.load([composite])
scn.show(composite)
Reported on pyspectral channel on pytroll-slack March 28th:
"trying to use the LUT's in the training and I get this error:
Directory /path/to/radiance/tb/lut/data does not exist! Check config file
I downloaded the rsr and the aerosol LUTs"
An RGB image - the day_microphysics RGB
Directory /path/to/radiance/tb/lut/data does not exist! Check config file
...but an image is generated as it should.
Using conda installation, pyspectral 0.8.6
Thank you for reporting an issue !
import glob
from satpy import Scene
fnames = glob.glob("/path/to/msg/data/H*__")
glbl = glbl.load(["day_microphysics"])
lcl = glbl.resample("euro4")
lcl.show("day_microphysics")The above code using Satpy uses Pyspectral to determine the NIR reflectance and uses it as one channel in the RGB. Current version of Pyspectral masks the data with a single strict limit of Sun zenith angle (85.0 degrees by default). The same limit is used to limit the range of SZA correction. These two should be separated. When this composite is used together with a night-side composite, there's a big gap between the composites.
Adjustable limit for masking invalid data by SZA.
Separate the SZA limit (sunz_threshold kwarg) and the masking limit of invalid data. So I would add another keyword argument to pyspectral.near_infrared_reflectance.Calculator.__init__. Maybe masking_limit. By default these both can have the same limit (TERMINATOR_LIMIT), but for Day/Night composites the masking limit should be adaptable.
Current master branch of Pyspectral.
I will push a PR momentarily.
Need to update the platform names to follow the WMO Oscar naming style, which is now standard in Pytroll. So EOS-Terra, EOS-Aqua, Meteosat-10, etc
#
from satpy import Scene, find_files_and_readers
from satpy.resample import get_area_def
from pyresample.utils import get_area_def
area_id = 'Sudamerica'
x_size = 3000
y_size = 1200
area_extent = (-5570248.4773392612, -4263473.5610361192, -384719.90821206354, 1339786.2707295895)
projection = '+proj=geos +a=6378137. +b=6356752.31414 +lon_0=-75 +f=.00335281068119356027 +h=35786023 +sweep=x'
description = "Sud geos"
proj_id = 'geos0'
areadef = get_area_def(area_id, description, proj_id, projection,x_size, y_size, area_extent)
scn = Scene(sensor='abi', filenames=['OR_ABI-L1b-RadF-M3C01_G16_s20182211830459_e20182211841226_c20182211841271.nc','OR_ABI-L1b-RadF-M3C02_G16_s20182211830459_e20182211841226_c20182211841260.nc','OR_ABI-L1b-RadF-M3C03_G16_s20182211830459_e20182211841226_c20182211841272.nc'])
rgbname = 'true_color'
local_scene = scn.resample(areadef)
local_scene.show(rgbname)I'm trying to generate a corrected true color images from GOES-16 netcdf files available en AWS.
I tried the example gave for Himawari-8 (https://github.com/pytroll/pytroll-examples/blob/master/satpy/ahi_true_color_pyspectral.ipynb). First, I couldn't use find_files_and_readers, so I define the filenames by myself. No problem on that.
Second, I couldn't process full disk image because of memory lack, I'm running the code in a 16GB RAM and 12 core server. Then I tried to process just South America. I used area definition from this web https://github.com/pytroll/satpy/blob/master/satpy/etc/areas.def
I got the following error:
Traceback (most recent call last):
File "true_pytroll.py", line 28, in
local_scene.show(rgbname)
File "/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/scene.py", line 986, in show
img = get_enhanced_image(self[dataset_id].squeeze(), overlay=overlay)
File "/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/scene.py", line 573, in getitem
return self.datasets[key]
File "/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/readers/init.py", line 299, in getitem
key = self.get_key(item)
File "/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/readers/init.py", line 288, in get_key
best=best, **dfilter)
File "/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/readers/init.py", line 239, in get_key
raise KeyError("No dataset matching '{}' found".format(str(key)))
KeyError: "No dataset matching 'DatasetID(name='true_color', wavelength=None, resolution=None, polarization=None, calibration=None, level=None, modifiers=None)' found"
Thank you very much.
There is no land-sea mask in Pyspectral, and no way yet to apply a land/sea dependent correction in Satpy using Pyspectral.
How much of this should be handled/supported in Pyspectral can be debated, and needs to be agreed.
But, that there is a need to different correction over land and sea is clear!
Perhaps Pycoast can be of use?
When I run my code with red band included I get the traceback below, however, if I call the function excluding the red band it runs as expected. I noticed at the bottom of the 'rayleigh.py' script the test function doesn't include the red band.
I have checked that all of my arrays going into the function are of the same shape, so I am rather confused why it doesn't like the shape of the red band.
Traceback (most recent call last):
File "/Volumes/GoogleDrive/My Drive/GOES_PROJECT/merge_full_disk_for_color_with_gamma.py", line 107, in
blue_cor = ray.get_reflectance(sun_alt, satz, ssadiff, 'C01', red)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pyspectral/rayleigh.py", line 228, in get_reflectance
redband = from_array(redband, chunks=redband.shape)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/dask/array/core.py", line 2327, in from_array
token = tokenize(x, chunks)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/dask/base.py", line 606, in tokenize
return md5(str(tuple(map(normalize_token, args))).encode()).hexdigest()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/dask/utils.py", line 415, in call
return meth(arg)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/dask/base.py", line 745, in normalize_array
data = hash_buffer_hex(x.copy().ravel(order='K').view('i1'))
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/ma/core.py", line 3046, in view
output = ndarray.view(self, dtype)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/ma/core.py", line 3271, in setattr
self._mask.shape = self.shape
ValueError: total size of new array must be unchanged
Python 2.7.15, Pyspectral 0.8.2
Thank you for reporting an issue !
Since Dask version 2021.5.1 the unittests fails. The searchsorted function is not supported by Dask anymore.
# Your code here
import numpy as np
from dask.array import from_array
wvl = 444.01239999999996
wvl_coord = np.array([440., 445.], dtype=np.float32)
dawvl_coord = from_array(wvl_coord)
idx = np.searchsorted(wvl_coord, wvl)
daidx = np.searchsorted(dawvl_coord, wvl)This simple code fails on the last line with the error message:
--> 773 meta = np.searchsorted(a._meta, v._meta)
774 out = blockwise(
775 _searchsorted_block,
AttributeError: 'float' object has no attribute '_meta'
The unittests should pass!
dask 2021.12.0 pyhd8ed1ab_0 conda-forge
h5py 3.6.0 nompi_py39h7e08c79_100 conda-forge
hdf5 1.12.1 nompi_h7f166f4_103 conda-forge
python 3.9.7 hf930737_3_cpython conda-forge
numpy 1.21.4 py39hdbf815f_0 conda-forge
Thank you for reporting an issue !
I'm using pyspectral.utils.download_luts() and pyspectral.utils.download_rsr() to pre-download the ancillary data needed by pyspectral. I noticed that it seems to re-download the data every time I run them. Is this expected? It is not a big deal for me since they should only be run once, but I was trying to run them to make sure I had the newest versions. If I have the newest versions I was hoping to not redownload.
# Your code here
from pyspectral.rsr_reader import RelativeSpectralResponse
avhrr3 = RelativeSpectralResponse('NOAA-19', 'avhrr-3')The above code sample gives a warning. The accepted naming of the sensor is avhrr/3 and not avhrr-3. The former is how it is displayed on WMO Oscar, hence this is the accepted naming in Pyspectral.
It would be fair though to also accept avhrr-3. This is also what Satpy uses, as / can be problematic to handle in a yaml encoding for instance.
This means Satpy is always raising a warning here. Pyspectral anyhow fixes it so it reads the correct RSR data anyhow, so the warning in Satpy is most disturbing and confusing the user.
Python 3.8
Latest Pyspectral release: 0.10.2
Thank you for reporting an issue !
Currently PySpectral uses masked arrays from Numpy to handle the data arrays. For better integration and SatPy support, these should be converted to dask arrays.
No code sample, sorry.
Trying to process GOES-18 simulated data with Satpy produces an exception in pyspectral when it tries to read the RSR data. The SRFs are available at:
https://ncc.nesdis.noaa.gov/GOESR/ABI.php
FM3 is G18 and I guess FM4 is G19.
Success!
File "/home/shared/bin/geo2grid_v_1_0_2/lib/python3.7/site-packages/satpy/scene.py", line 843, in _generate_composite
**self.attrs)
File "/home/shared/bin/geo2grid_v_1_0_2/lib/python3.7/site-packages/satpy/composites/__init__.py", line 575, in __call__
red.data)
File "/home/shared/bin/geo2grid_v_1_0_2/lib/python3.7/site-packages/pyspectral/rayleigh.py", line 248, in get_reflectance
wvl = self.get_effective_wavelength(bandname)
File "/home/shared/bin/geo2grid_v_1_0_2/lib/python3.7/site-packages/pyspectral/rayleigh.py", line 187, in get_effective_wavelength
rsr = RelativeSpectralResponse(self.platform_name, self.sensor)
File "/home/shared/bin/geo2grid_v_1_0_2/lib/python3.7/site-packages/pyspectral/rsr_reader.py", line 104, in __init__
"platform name and sensor or filename must be specified")
Thank you for reporting an issue !
I installed pyspectral locally via my GitHub fork (to fix the MPL warning)
# Your code here
from pyspectral.solar import (SolarIrradianceSpectrum,
TOTAL_IRRADIANCE_SPECTRUM_2000ASTM,
)
solar_irr = SolarIrradianceSpectrum(TOTAL_IRRADIANCE_SPECTRUM_2000ASTM,
dlambda=0.1)I installed my fork into my conda environment like I always do:
pip install -e .but the data file is not being copied at this point.
I know from myself that setting this up correctly in setup.py was a pain, but I currently don't remember what was the trick.
Data should be copied correctly.
---------------------------------------------------------------------------
OSError Traceback (most recent call last)
<ipython-input-93-630e5c929426> in <module>()
1 solar_irr = SolarIrradianceSpectrum(TOTAL_IRRADIANCE_SPECTRUM_2000ASTM,
----> 2 dlambda=0.1)
~/miniconda3/envs/stable/lib/python3.6/site-packages/pyspectral/solar.py in __init__(self, filename, **options)
~/miniconda3/envs/stable/lib/python3.6/site-packages/pyspectral/solar.py in _load(self)
~/miniconda3/envs/stable/lib/python3.6/site-packages/numpy/lib/npyio.py in genfromtxt(fname, dtype, comments, delimiter, skip_header, skip_footer, converters, missing_values, filling_values, usecols, names, excludelist, deletechars, replace_space, autostrip, case_sensitive, defaultfmt, unpack, usemask, loose, invalid_raise, max_rows, encoding)
1687 fname = str(fname)
1688 if isinstance(fname, basestring):
-> 1689 fhd = iter(np.lib._datasource.open(fname, 'rt', encoding=encoding))
1690 own_fhd = True
1691 else:
~/miniconda3/envs/stable/lib/python3.6/site-packages/numpy/lib/_datasource.py in open(path, mode, destpath, encoding, newline)
258
259 ds = DataSource(destpath)
--> 260 return ds.open(path, mode, encoding=encoding, newline=newline)
261
262
~/miniconda3/envs/stable/lib/python3.6/site-packages/numpy/lib/_datasource.py in open(self, path, mode, encoding, newline)
614 encoding=encoding, newline=newline)
615 else:
--> 616 raise IOError("%s not found." % path)
617
618
OSError: /Users/klay6683/miniconda3/envs/stable/lib/python3.6/site-packages/pyspectral/data/e490_00a.dat not found.Using conda Python 3.6 on macOS 10.12.6, all up-to-date.
# Your code here[this should also explain why the current behaviour is a problem and why the
expected output is a better solution.]
Thank you for reporting an issue !
While it is possible to apply Pyspectral to correct for Rayleigh scattering of a full disk Himawari/GOES-16 image, the correction is not strong enough at large viewing angles. This is because the RTM simulations and LUT tables actually only cover the 0-70 degree interval. We need to extend the LUTs down to 90 degrees.
There are a lot of functions in pyspectral.blackbody that call np.array(some_input_argument) which can unnecessarily convert dask arrays to numpy arrays. I/we should update this so it allows for dask arrays to pass through. One option:
try:
from dask.array import asanyarray
except ImportError:
from numpy import asanyarray
Thoughts?
Add Himawari-9 spectral response functions
Could I obtain Correction reflectance by Rayleigh in numpy array? I would like to obtain channel 2,3,5.
We found this example but it is not so clear, how can I implement every 10 minutes in the netcdf file.
https://github.com/pytroll/pytroll-examples/blob/master/pyspectral/pyspectral_rayleigh_abi.ipynb
import numpy as np
from pyspectral.rayleigh import Rayleigh
from pyspectral.utils import debug_on
debug_on()
msi = Rayleigh('Sentinel-2C', 'msi')
sunz = np.array([[32., 40.], [31., 41.]])
satz = np.array([[45., 20.], [46., 21.]])
ssadiff = np.array([[110, 170], [120, 180]])
refl_cor_red = msi.get_reflectance(sunz, satz, ssadiff, 'B02')In the case that a new satellite not yet supported with the spectral responses in PySpectral, a download will be attempted when a band name is provided to the get_reflectance method of the Rayleigh object.
To avoid trying to download RSR data in vain (cases where even after download the rsr data for the requested sensor and platform will anyway not be there) one needs to add a check if the downloaded RSR data are the latest. This kind of checking is already implemented for the atm correction LUTs.
So, currently with the code example above a download is attempted, even if RSR data are the most recent. As seen in the IOError below it attempts to download the data since there is no file for S2C. But, if you repeat the run it will do the same again.
Instead, it should detect that the latest RSR data have been downloaded and are available, and if the S2C RSR data are not part of the data, it can throw an IOError as is done.
[INFO: 2018-06-01 13:43:43 : pyspectral.rayleigh] Atmosphere chosen: us-standard
[DEBUG: 2018-06-01 13:43:43 : pyspectral.rayleigh] LUT filename: /home/a000680/.local/share/pyspectral/marine_clean_aerosol/rayleigh_lut_us-standard.h5
[DEBUG: 2018-06-01 13:43:43 : pyspectral.rsr_reader] Filename: /home/a000680/.local/share/pyspectral/rsr_msi_Sentinel-2C.h5
[WARNING: 2018-06-01 13:43:43 : pyspectral.rsr_reader] No rsr file /home/a000680/.local/share/pyspectral/rsr_msi_Sentinel-2C.h5 on disk
[INFO: 2018-06-01 13:43:43 : pyspectral.rsr_reader] Will download from internet...
[DEBUG: 2018-06-01 13:43:43 : urllib3.connectionpool] Starting new HTTPS connection (1): zenodo.org
[DEBUG: 2018-06-01 13:43:44 : urllib3.connectionpool] https://zenodo.org:443 "GET /record/1205138/files/pyspectral_rsr_data.tgz HTTP/1.1" 200 2798376
2798376it [00:01, 1678973.88it/s]
[ERROR: 2018-06-01 13:43:46 : pyspectral.rayleigh] No spectral responses for this platform and sensor: Sentinel-2C msi
Traceback (most recent call last):
File "/home/a000680/usr/src/pyspectral/pyspectral/rayleigh.py", line 138, in get_effective_wavelength
rsr = RelativeSpectralResponse(self.platform_name, self.sensor)
File "/home/a000680/usr/src/pyspectral/pyspectral/rsr_reader.py", line 94, in __init__
raise IOError(errmsg)
IOError: pyspectral RSR file does not exist! Filename = /home/a000680/.local/share/pyspectral/rsr_msi_Sentinel-2C.h5
Files matching instrument and satellite platform: []
Python 2.7.5
Pyspectral 0.7.2 - latest master - 3937586
Thank you for reporting an issue !
It's both in Calculator (near_infrared_reflectance) and rayleigh
# requires AMI PR from Satpy
from satpy import Scene
from glob import glob
scn = Scene(reader='ami_l1b', filenames=glob('/data/satellite/ami/20190930_0300/*.nc'))
scn.load(['day_microphysics_eum'])AHI is able to make the day_microphysics_eum composite, but AMI is not. Tracking it down I found that the nir_reflectance modifier in Satpy was failing during some calls to pyspectral. Specifically the piece of the solar.py code that uses the wvl from the RSR files (I think). If I print out the values of wvl for AMI I see:
[3.9797828 3.979624 3.979466 ... 3.6831057 3.68297 3.6828344]
This shows that the values are decreasing instead of increasing. The code is failing (see traceback below) when we try to tell np.linspace to give us -593 elements in our array. This number is calculated by doing end - start where end and start are the last and first wavelength in the above array. This shows that AMI's RSR data is stored backwards (90% sure).
If I add this right before the linspace call then it works:
if start > end:
start, end = end, start
wvl = wvl[::-1]
But this is an ugly hack and I feel like pyspectral should probably require that RSR files have increasing wavelengths in its file.
No errors and some new data returned.
~/repos/git/satpy/satpy/composites/__init__.py in _init_refl3x(self, projectables)
613 raise ImportError("No module named pyspectral.near_infrared_reflectance")
614 _nir, _tb11 = projectables
--> 615 self._refl3x = Calculator(_nir.attrs['platform_name'], _nir.attrs['sensor'], _nir.attrs['name'])
616
617 def _get_reflectance(self, projectables, optional_datasets):
~/repos/git/pyspectral/pyspectral/near_infrared_reflectance.py in __init__(self, platform_name, instrument, band, **kwargs)
87 self.solar_flux = kwargs.get('solar_flux', None)
88 if self.solar_flux is None:
---> 89 self._get_solarflux()
90
91 self._rad3x = None
~/repos/git/pyspectral/pyspectral/near_infrared_reflectance.py in _get_solarflux(self)
162 dlambda=0.0005,
163 wavespace=self.wavespace)
--> 164 self.solar_flux = solar_spectrum.inband_solarflux(self.rsr[self.bandname])
165
166 def emissive_part_3x(self, tb=True):
~/repos/git/pyspectral/pyspectral/solar.py in inband_solarflux(self, rsr, scale, **options)
119 spectral response valid for an earth-sun distance of one AU.
120 """
--> 121 return self._band_calculations(rsr, True, scale, **options)
122
123 def inband_solarirradiance(self, rsr, scale=1.0, **options):
~/repos/git/pyspectral/pyspectral/solar.py in _band_calculations(self, rsr, flux, scale, **options)
166 LOG.debug("Begin and end wavelength/wavenumber: %f %f ", start, end)
167 dlambda = self._dlambda
--> 168 xspl = np.linspace(start, end, round((end - start) / self._dlambda) + 1)
169
170 ius = InterpolatedUnivariateSpline(wvl, resp)
<__array_function__ internals> in linspace(*args, **kwargs)
~/miniconda3/envs/satpy_py37/lib/python3.7/site-packages/numpy/core/function_base.py in linspace(start, stop, num, endpoint, retstep, dtype, axis)
128 num = _index_deprecate(num)
129 if num < 0:
--> 130 raise ValueError("Number of samples, %s, must be non-negative." % num)
131 div = (num - 1) if endpoint else num
132
ValueError: Number of samples, -593, must be non-negative.
Current master of everything.
Thank you for reporting an issue !
Importing pyspectral's config.py module results in:
/Users/davidh/repos/git/pyspectral/pyspectral/config.py:75: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
config = recursive_dict_update(config, yaml.load(fp_))
This is because of a deprecation in pyyaml 5.1+ (see here).
This should be updated to use:
try:
from yaml import UnsafeLoader
except ImportError:
from yaml import Loader as UnsafeLoader
res = yaml.load(file, Loader=UnsafeLoader)from pyspectral.rsr_reader import RelativeSpectralResponse
olci = RelativeSpectralResponse('Sentinel-3A', 'olci')
olci.rsr['ch21']['det-1'].keys()The channel 21 of the Sentinel-3A OLCI is missing
Thank you for reporting an issue !
from glob import glob
from satpy import Scene
from pyresample.geometry import AreaDefinition
from pyresample.utils import proj4_str_to_dict
from pyproj import Proj
from satpy.utils import debug_on
import dask as da
from multiprocessing.pool import ThreadPool
from datetime import datetime
def EPSG_4326_definition(ll_ur=None, resolution=2.0):
# ll_ur is (lon, lat, lon, lat)
area_id = 'epsg4326'
description = 'EPSG:4326'
proj_id = 'epsg4326'
#projection = '+proj=eqc +lat_ts=0 +lat_0=0 +lon_0=0 +x_0=0 +y_0=0 +ellps=WGS84 +datum=WGS84 +units=m'
projection = "+proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs"
proj4_dict = proj4_str_to_dict(projection)
#resolution = 2.0
y_size = 20480 / resolution # Divide by two for 2km resolution
x_size = 40960 / resolution # ditto!
ll_ur_ref = (-180.0, -90.0, 180.0, 90.0)
area_extent = ll_ur
y_size = int((ll_ur[3] - ll_ur[1]) / (ll_ur_ref[3] - ll_ur_ref[1]) * y_size)
x_size = int((ll_ur[2] - ll_ur[0]) / (ll_ur_ref[2] - ll_ur_ref[0]) * x_size)
print(y_size, x_size)
return(AreaDefinition(area_id, description, proj_id, proj4_dict, x_size, y_size, area_extent))
debug_on()
da.config.set(pool=ThreadPool(4))
FILES = glob("/data/kuehn/AHI_rt/DAT/two/*DAT")
scn = Scene(filenames=FILES, reader='ahi_hsd')
scn.load(['true_color'])
#scn['true_color'].data.chunks
areadef = EPSG_4326_definition(ll_ur=(80.0, -10, 180.0, 30))
local_scene = scn.resample(areadef, cache_dir='/data/kuehn/AHI_rt/cache')
local_scene.save_dataset('true_color', filename='./local_true_color_crefl.tif', GDAL_OPTIONS=['COMPRESS=JPEG', 'PHOTOMETRIC=YCBCR', 'TILED=YES'])
Problem description
The rsr data is downloaded every time I run the script. The data is indeed downloaded and stored in a local directory with write permission. It should not be downloaded every time.
Note that I've modified /satpy/etc/enhancements/generic.yaml such that crefl_scaling is the default true_color enhancement.
Actual Result, Traceback if applicable
[INFO: 2018-10-10 21:29:45 : pyspectral.rayleigh] Atmosphere chosen: us-standard
[DEBUG: 2018-10-10 21:29:45 : pyspectral.rayleigh] LUT filename: /home/kuehn/.local/share/pyspectral/marine_clean_aerosol/rayleigh_lut_us-standard.h5
[DEBUG: 2018-10-10 21:29:45 : pyspectral.rsr_reader] Filename: /home/kuehn/.local/share/pyspectral/rsr_ahi_Himawari-8.h5
[WARNING: 2018-10-10 21:29:45 : pyspectral.rsr_reader] rsr data may not be up to date: /home/kuehn/.local/share/pyspectral/rsr_ahi_Himawari-8.h5
[INFO: 2018-10-10 21:29:45 : pyspectral.rsr_reader] Will download from internet...
[INFO: 2018-10-10 21:29:45 : pyspectral.utils] Download RSR files and store in directory /home/kuehn/.local/share/pyspectral
[DEBUG: 2018-10-10 21:29:45 : pyspectral.utils] Get data. URL: https://zenodo.org/record/1409621/files/pyspectral_rsr_data.tgz
[DEBUG: 2018-10-10 21:29:45 : pyspectral.utils] Destination = /home/kuehn/.local/share/pyspectral
[DEBUG: 2018-10-10 21:29:45 : urllib3.connectionpool] Starting new HTTPS connection (1): zenodo.org:443
[DEBUG: 2018-10-10 21:29:46 : urllib3.connectionpool] https://zenodo.org:443 "GET /record/1409621/files/pyspectral_rsr_data.tgz HTTP/1.1" 200 2949478
2949478it [00:04, 611429.75it/s]
[DEBUG: 2018-10-10 21:29:53 : pyspectral.rsr_reader] Filename: /home/kuehn/.local/share/pyspectral/rsr_ahi_Himawari-8.h5
Versions of Python, package at hand and relevant dependencies
packages in environment at /home/kuehn/miniconda3/envs/satpy3:
Name Version Build Channel
affine 2.2.1 py_0 conda-forge
appdirs 1.4.3 py_1 conda-forge
asn1crypto 0.24.0 py36_1003 conda-forge
attrs 18.2.0 py_0 conda-forge
blas 1.0 mkl
bokeh 0.13.0 py36_0 conda-forge
boost-cpp 1.67.0 h3a22d5f_0 conda-forge
boto3 1.9.16 py_0 conda-forge
botocore 1.12.17 py_0 conda-forge
bottleneck 1.2.1 py36h7eb728f_1 conda-forge
bzip2 1.0.6 h470a237_2 conda-forge
ca-certificates 2018.8.24 ha4d7672_0 conda-forge
cairo 1.14.12 he6fea26_5 conda-forge
certifi 2018.8.24 py36_1001 conda-forge
cffi 1.11.5 py36h5e8e0c9_1 conda-forge
cftime 1.0.1 py36h7eb728f_0 conda-forge
chardet 3.0.4 py36_1003 conda-forge
click 7.0 py_0 conda-forge
click-plugins 1.0.4 py_0 conda-forge
cligj 0.5.0 py_0 conda-forge
cloudpickle 0.5.6 py_0 conda-forge
configobj 5.0.6 py_0 conda-forge
cryptography 2.3.1 py36hdffb7b8_0 conda-forge
cryptography-vectors 2.3.1 py36_1000 conda-forge
curl 7.61.0 h93b3f91_2 conda-forge
cytoolz 0.9.0.1 py36h470a237_1 conda-forge
dask 0.19.2 py_0 conda-forge
dask-core 0.19.2 py_0 conda-forge
distributed 1.23.2 py36_1 conda-forge
docutils 0.14 py36_1001 conda-forge
expat 2.2.5 hfc679d8_2 conda-forge
fontconfig 2.13.1 h65d0f4c_0 conda-forge
freetype 2.9.1 h6debe1e_4 conda-forge
freexl 1.0.5 h470a237_2 conda-forge
gdal 2.2.4 py36hb00a9d7_9 conda-forge
geos 3.6.2 hfc679d8_3 conda-forge
geotiff 1.4.2 h700e5ad_4 conda-forge
gettext 0.19.8.1 h5e8e0c9_1 conda-forge
giflib 5.1.4 h470a237_1 conda-forge
glib 2.55.0 h464dc38_2 conda-forge
h5netcdf 0.6.2 py_0 conda-forge
h5py 2.8.0 py36h7eb728f_3 conda-forge
hdf4 4.2.13 h951d187_2 conda-forge
hdf5 1.10.2 hc401514_2 conda-forge
heapdict 1.0.0 py36_1000 conda-forge
icu 58.2 hfc679d8_0 conda-forge
idna 2.7 py36_1002 conda-forge
intel-openmp 2019.0 118
jinja2 2.10 py_1 conda-forge
jmespath 0.9.3 py_1 conda-forge
jpeg 9c h470a237_1 conda-forge
json-c 0.12.1 h470a237_1 conda-forge
kealib 1.4.9 h0bee7d0_2 conda-forge
krb5 1.14.6 0 conda-forge
libdap4 3.19.1 h8fe5423_1 conda-forge
libffi 3.2.1 hfc679d8_5 conda-forge
libgcc-ng 7.2.0 hdf63c60_3 conda-forge
libgdal 2.2.4 hbd6f514_9 conda-forge
libgfortran 3.0.0 1 conda-forge
libgfortran-ng 7.2.0 hdf63c60_3 conda-forge
libiconv 1.15 h470a237_3 conda-forge
libkml 1.3.0 hccc92b1_8 conda-forge
libnetcdf 4.6.1 he6cff42_8 conda-forge
libpng 1.6.35 ha92aebf_2 conda-forge
libpq 9.6.3 0 conda-forge
libspatialite 4.3.0a hdfcc80b_23 conda-forge
libssh2 1.8.0 h5b517e9_2 conda-forge
libstdcxx-ng 7.2.0 hdf63c60_3 conda-forge
libtiff 4.0.9 he6b73bb_2 conda-forge
libuuid 2.32.1 h470a237_2 conda-forge
libxcb 1.13 h470a237_2 conda-forge
libxml2 2.9.8 h422b904_5 conda-forge
locket 0.2.0 py_2 conda-forge
markupsafe 1.0 py36h470a237_1 conda-forge
mkl 2019.0 118
mkl_fft 1.0.6 py36_0 conda-forge
mkl_random 1.0.1 py36_0 conda-forge
msgpack-python 0.5.6 py36h2d50403_3 conda-forge
ncurses 6.1 hfc679d8_1 conda-forge
netcdf4 1.4.1 py36h62672b6_0 conda-forge
numpy 1.15.0 py36h1b885b7_0
numpy-base 1.15.0 py36h3dfced4_0
olefile 0.46 py_0 conda-forge
openjpeg 2.3.0 h0e734dc_3 conda-forge
openssl 1.0.2p h470a237_0 conda-forge
packaging 18.0 py_0 conda-forge
pandas 0.23.4 py36hf8a1672_0 conda-forge
partd 0.3.8 py_1 conda-forge
pcre 8.41 hfc679d8_3 conda-forge
pillow 5.3.0 py36hc736899_0 conda-forge
pip 18.0 py36_1001 conda-forge
pixman 0.34.0 h470a237_3 conda-forge
poppler 0.67.0 h4d7e492_3 conda-forge
poppler-data 0.4.9 0 conda-forge
postgresql 9.6.3 4 conda-forge
proj4 4.9.3 h470a237_8 conda-forge
psutil 5.4.7 py36h470a237_1 conda-forge
pthread-stubs 0.4 h470a237_1 conda-forge
pycparser 2.19 py_0 conda-forge
pykdtree 1.3.1 py36h7eb728f_2 conda-forge
pyopenssl 18.0.0 py36_0 conda-forge
pyorbital 1.3.1 py_0 conda-forge
pyparsing 2.2.2 py_0 conda-forge
pyproj 1.9.5.1 py36h508ed2a_5 conda-forge
pyresample 1.10.2 py36hf8a1672_0 conda-forge
pysocks 1.6.8 py36_1002 conda-forge
pyspectral 0.8.3 py_0 conda-forge
python 3.6.6 h5001a0f_0 conda-forge
python-dateutil 2.7.3 py_0 conda-forge
python-geotiepoints 1.1.6 py36h24bf2e0_0 conda-forge
pytz 2018.5 py_0 conda-forge
pyyaml 3.13 py36h470a237_1 conda-forge
rasterio 1.0.8 py36h1b5fcde_0 conda-forge
readline 7.0 haf1bffa_1 conda-forge
requests 2.19.1 py36_1 conda-forge
s3transfer 0.1.13 py36_1001 conda-forge
satpy 0.9.1+307.g75e8cb86.dirty <pip>
scipy 1.1.0 py36hc49cb51_0
setuptools 40.4.3 py36_0 conda-forge
six 1.11.0 py36_1001 conda-forge
snuggs 1.4.1 py_1 conda-forge
sortedcontainers 2.0.5 py_0 conda-forge
sqlite 3.25.2 hb1c47c0_0 conda-forge
tblib 1.3.2 py_1 conda-forge
tk 8.6.8 ha92aebf_0 conda-forge
toolz 0.9.0 py_1 conda-forge
tornado 5.1.1 py36h470a237_0 conda-forge
tqdm 4.26.0 py_0 conda-forge
trollimage 1.5.7 py_0 conda-forge
trollsift 0.3.0 py_0 conda-forge
urllib3 1.23 py36_1 conda-forge
wheel 0.32.1 py36_0 conda-forge
xarray 0.10.9 py36_0 conda-forge
xerces-c 3.2.0 h5d6a6da_2 conda-forge
xorg-kbproto 1.0.7 h470a237_2 conda-forge
xorg-libice 1.0.9 h470a237_4 conda-forge
xorg-libsm 1.2.2 h8c8a85c_6 conda-forge
xorg-libx11 1.6.6 h470a237_0 conda-forge
xorg-libxau 1.0.8 h470a237_6 conda-forge
xorg-libxdmcp 1.1.2 h470a237_7 conda-forge
xorg-libxext 1.3.3 h470a237_4 conda-forge
xorg-libxrender 0.9.10 h470a237_2 conda-forge
xorg-renderproto 0.11.1 h470a237_2 conda-forge
xorg-xextproto 7.3.0 h470a237_2 conda-forge
xorg-xproto 7.0.31 h470a237_7 conda-forge
xz 5.2.4 h470a237_1 conda-forge
yaml 0.1.7 h470a237_1 conda-forge
zict 0.1.3 py_0 conda-forge
zlib 1.2.11 h470a237_3 conda-forge
Thank you for reporting an issue !
At the benefit of development/testing time the original python versions of the crefl algorithms for rayleigh correction were added as modifiers to satpy. These should really be in pyspectral. There is one algorithm for VIIRS/MODIS and one for ABI.
https://github.com/pytroll/satpy/blob/master/satpy/composites/crefl_utils.py
# Your code hereTravis just started failing this week without any changes being done to the repo. It appears we have reached a limit on the git-lfs quota. We need to find a way to handle this!
See here
Thank you for reporting an issue !
I think this is a pyspectral error?
File "/packages/pytroll/local/lib/python2.7/site-packages/satpy/scene.py", line 564, in load
keepables = self.compute()
File "/packages/pytroll/local/lib/python2.7/site-packages/satpy/scene.py", line 511, in compute
return self.read_composites(nodes)
File "/packages/pytroll/local/lib/python2.7/site-packages/satpy/scene.py", line 485, in read_composites
self._generate_composite(item, keepables)
File "/packages/pytroll/local/lib/python2.7/site-packages/satpy/scene.py", line 445, in _generate_composite
keepables,
File "/packages/pytroll/local/lib/python2.7/site-packages/satpy/scene.py", line 405, in _get_prereq_datasets
self._generate_composite(prereq_node, keepables)
File "/packages/pytroll/local/lib/python2.7/site-packages/satpy/scene.py", line 460, in _generate_composite
**self.info)
File "/packages/pytroll/local/lib/python2.7/site-packages/satpy/composites/__init__.py", line 374, in __call__
refl_cor_band = corrector.get_reflectance(sunz, satz, ssadiff, vis.id.name, red)
File "/packages/pytroll/local/lib/python2.7/site-packages/pyspectral/rayleigh.py", line 235, in get_reflectance
(1 - (blueband - 20) / 80) * res)
File "/packages/pytroll/local/lib/python2.7/site-packages/satpy/dataset.py", line 99, in wrapper
res = func(self, other, *args, **kwargs)
File "/packages/pytroll/local/lib/python2.7/site-packages/satpy/dataset.py", line 311, in __mul__
return super(Dataset, self).__mul__(other)
File "/packages/pytroll/local/lib/python2.7/site-packages/numpy/ma/core.py", line 4086, in __mul__
return multiply(self, other)
File "/packages/pytroll/local/lib/python2.7/site-packages/numpy/ma/core.py", line 1011, in __call__
result = self.f(da, db, *args, **kwargs)
ValueError: operands could not be broadcast together with shapes (21440,5416) (5360,1354)
Trying to load a MODIS true_color with satpy 0.8.0, pyspectral 0.5.0
On a fresh Ubuntu 16.04 box, after 'pip install pyspectral', this error occurs when using Rayleigh corrections in the true_color satpy images:
File "/usr/local/lib/python2.7/dist-packages/satpy-0.3.0-py2.7.egg/satpy/scene.py", line 656, in load
**kwargs)
File "/usr/local/lib/python2.7/dist-packages/satpy-0.3.0-py2.7.egg/satpy/scene.py", line 541, in read
return self.read_from_deptree(dataset_keys, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/satpy-0.3.0-py2.7.egg/satpy/scene.py", line 538, in read_from_deptree
composites = self.read_composites(tree.trunk(), **kwargs)
File "/usr/local/lib/python2.7/dist-packages/satpy-0.3.0-py2.7.egg/satpy/scene.py", line 506, in read_composites
**self.info)
File "/usr/local/lib/python2.7/dist-packages/satpy-0.3.0-py2.7.egg/satpy/composites/__init__.py", line 324, in __call__
from pyspectral.rayleigh import Rayleigh
File "/home/balt/.local/lib/python2.7/site-packages/pyspectral/rayleigh.py", line 35, in <module>
from pyspectral.rsr_reader import RelativeSpectralResponse
File "/home/balt/.local/lib/python2.7/site-packages/pyspectral/rsr_reader.py", line 39, in <module>
from pyspectral.utils import (INSTRUMENTS, download_rsr)
File "/home/balt/.local/lib/python2.7/site-packages/pyspectral/utils.py", line 72, in <module>
for option, value in CONF.items('general', raw=True):
File "/usr/lib/python2.7/ConfigParser.py", line 642, in items
raise NoSectionError(section)
ConfigParser.NoSectionError: No section: 'general'
Workaround is to export the location of the config file explicitly like so:
os.environ["PSP_CONFIG_FILE"] = "/home/user/.local/etc/pyspectral.cfg"
There is no RSR data available for Metop/AVHRR. Or at least they are not defined in the built-in configuration file.
Hi,
Would it be possible to have some examples how to create Rayleigh corrected true color for AHI? Thank you very much.
After #51 the crefl algorithms should be rewritten in cython for the best performance. Currently they use numpy/dask arrays, but these are wasteful with intermediate/temporary arrays. Doing things by pixel should theoretically perform better and reduce the memory usage.
@ralphk11 and @wroberts4 have the most experience with these algorithms probably. @mraspaud also wrote the original viirs/modis crefl python code that is in satpy although @ralphk11 had one in the past too.
Work has been conducted on a "limb-cooling" correction algorithm. This is work in progress but results look quite good, so it would be adequate adding some info about it
# Your code here
In [1]: from pyspectral.radiance_tb_conversion import RadTbConverter
In [2]: viirs = RadTbConverter('Suomi-NPP', 'viirs', 'I5')
In [3]: viirs.radiance2tb(viirs.tb2radiance(290)['radiance'])
Out[3]: 289.600620823891
The radiance-tb converter looks like it has two functions going from tb to radiance and back again, making the user think one is the inverse of the other. However the radiance-to-tb method radiance2tb uses the central wavelength and does not make a convolution with the spectral response function, thus one does not arrive back to the exact same Tb as shown in the above example.
If the two functions were inverse of each other, one should expect smaller deviations in the calculations as the above.
Python 3.7.3
pyspectral 0.9.3
Thank you for reporting an issue !
The equation for converting from SI to mW/m^2 sr^-1 (cm^-1)^-1 given here:
https://pyspectral.readthedocs.io/en/master/api.html#pyspectral.blackbody.blackbody_wn
is currently:
1.0 W/m^2 sr^-1 (m^-1)^-1 = 0.1 mW/m^2 sr^-1 (cm^-1)^-1
But it probably should be:
1.0 W/m^2 sr^-1 (m^-1)^-1 = 1e5 * mW/m^2 sr^-1 (cm^-1)^-1
Many thanks!
Getting the near infrared reflectance of a viirs scene crashes.
I'll be filling this bug report properly when I'm in a well connected place again.
This is happening in a couple of places in solar.py
np.linspace(start, end, round((end - start) / self._dlambda) + 1)This is not working with numpy > 1.18 as round doesn't return an int (see numpy/numpy#11557 )
I expect this to run without crashing.
File "/san1/opt/ears_viirs_imagery_processor/releases/ears_viirs_imagery_processor-0.1.9/lib/python3.6/site-packages/pyspectral/solar.py", line 182, in _band_calculations
xspl = np.linspace(start, end, round((end - start) / self._dlambda) + 1)
File "<__array_function__ internals>", line 6, in linspace
File "/san1/opt/ears_viirs_imagery_processor/releases/ears_viirs_imagery_processor-0.1.9/lib/python3.6/site-packages/numpy/core/function_base.py", line 121, in linspace
.format(type(num)))
TypeError: object of type <class 'numpy.float64'> cannot be safely interpreted as an integer.A simple solution would be to replace round with int(round)
Numpy 1.18.1
Hi,
It would be very nice to have the database (RSR) updated to the spectral response functions of SLSTR on Sentinel 3B. Functions for SLSTR on Sentinel 3A are already in.
Best
Rolle
Thank you for reporting an issue !
The atmospheric correction in the visible range currently requires access to the satellite sensor RSR functions. Many sensors are already available in PySPectral. However, it would be nice/helpful if one could derive the Rayleigh correction without having the RSR implemented in PySpectral. It would just require the user to provide a central or rayleigh scattering effective wavelength, supposing the user could calculate this herself.
Potentially the code could look like this:
from pyspectral.rayleigh import Rayleigh
viirs = Rayleigh('Suomi-NPP', 'viirs')
corr = viirs.get_reflectance(sun_zenith, sat_zenith, azidiff, effective_wavelength=0.486, refl_m05)
Currently the last call as to llok like this:
corr = viirs.get_reflectance(sun_zenith, sat_zenith, azidiff, 'M03', refl_m05)
Thank you for reporting an issue !
The Landsat-8 TIRS instrument RSR is available here: https://landsat.gsfc.nasa.gov/landsat-8/spectral-response-thermal-sensor
I suggest adding to the RSR lookups, and would be happy to try to make this change however I am new to HDF5 format.
I've been playing around with dask-distributed for using satpy/dask on a cluster. While learning more about how to submit tasks/data to workers I learned about the idea of sending data to the workers that stays at the workers for future calculations. This sounds like it could be a good idea for rayleigh correction where pyspectral/satpy pushes the LUTs to the workers and then as rayleigh correction tasks are submitted for processing they would be sent to workers that had the LUTs already in memory. I'm not sure this is a perfect use case for this since LUTs, by their nature, require the entire LUT to be available (you can't chunk them and do a look up in only one chunk).
I just wanted to write this down for future consideration.
from pyspectral.radiance_tb_conversion import RadTbConverter
from pyspectral.utils import debug_on
debug_on()
modis = RadTbConverter('EOS-Aqua', 'modis', '20')
res = modis.tb2radiance(281., '20', lut=False)
When the above code is run, the spectral radiance for the MODIS band 20 is derived.
One issue is that it should actually not be needed to pass the band to the method. It should
actually not be allowed to derive the radiance for another band.
But, my main issue, is that it should also be possible to derive the band integrated radiance.
This can be derived by not dividing with the integral of the relative spectral response for the given
band. So something like this:
modis = RadTbConverter('EOS-Aqua', 'modis', '20')
res = modis.tb2radiance(281., lut=False, normalized=False)
Thank you for reporting an issue !
Looks like the dropbox links for the Rayleigh look-up tables are broken: https://dl.dropboxusercontent.com/u/37482654/pyspectral/rayleigh_only/pyspectral_rayleigh_correction_luts.tgz
This is most likely happening due to Dropbox disabling Public Links - https://www.dropbox.com/help/files-folders/public-folder
It would be awesome if you are able to share the files on another location or if I can pull them from FTP.
Thanks for the awesome work with this project!
I'm rather new to pyspectral. (A colleague downloaded it around the first of this year.) I'm using MODIS HDF files from CSPP, I've used data from two sources: granules from NASA/LANCE (near-realtime server) and swaths from a direct broadcast system at U. of Alaska. With both sources, I'm running them through the U. of Wisconsin 'polar2grid' package, creating binary files. Most files work beautifully - creating very nice Daytime Microphysics RGBs. However, occasionally, while in 'reflectance_from_tbs', it will fail. And, in a rare instance, I will get the message "Killed" - and the program will not recover - even in an 'except' clause. Can you provide any assistance as to the possible reason? The arrays passed in seem fine - legitimate values for solarZenithAngle, limbCorrectedBand20 (3.9micron), and limbCorrectedBand31(~11micron).
# Your code here
>>> from pyspectral.rsr_reader import RelativeSpectralResponse
>>> rsr = RelativeSpectralResponse('NOAA-20', 'viirs')The code fails for latest versions of h5py (3.1.0 released Nov 6, 2020), see below!
Code should run without exceptions and return the RSR data for N20/VIIRS
rsr data may not be up to date: /data/proj/safutv/.local/share/pyspectral/rsr_viirs_NOAA-20.h5
4494378it [00:02, 1923809.28it/s]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/san1/opt/viirs_imagery_processor/releases/viirs_imagery_processor-0.2.65/lib/python3.8/site-packages/pyspectral/rsr_reader.py", line 125, in __init__
self.load()
File "/san1/opt/viirs_imagery_processor/releases/viirs_imagery_processor-0.2.65/lib/python3.8/site-packages/pyspectral/rsr_reader.py", line 174, in load
self.description = self.description.decode('utf-8')
AttributeError: 'str' object has no attribute 'decode'
h5py=3.1.0
pyspectral=0.10.1 and latest master as of Nov 10, 2020
Thank you for reporting an issue !
In the solar.py code you have this for interpolation in the method interpolate:
xspl = np.linspace(start, end, (end - start) / self._dlambda)Could it be that you wanted:
xspl = np.linspace(start, end, round((end - start) / self._dlambda)+1)because then, for a dlambda=0.001, for example, you would actually get 1 nm wide bins.
The round is required because linspace takes an integer as input there, to avoid floating point issues?
With your method, interpolate(dlambda=0.001, ival_wavelength=(0.200, 1.2))
gets me this:
array([0.2 , 0.201001 , 0.202002 , 0.203003 , 0.204004 ,
0.20500501, 0.20600601, 0.20700701, 0.20800801, 0.20900901,while I think the user expects bins of 1 nm in size at this point?
My solution would do exactly that.
Let me know, I can push another PR (but that, potentially, could change some numbers on your side).
When I am creating a product (i.e. cloudtop_daytime) on a server with no internet connection, the rsr-data couldn't be downloaded/refreshed automatically. This rises an error message.
No error output
rsr data may not be up to date: /home/schitter/.local/share/pyspectral/rsr_seviri_Meteosat-11.h5
Traceback (most recent call last):
File "/usr/lib64/python3.6/site-packages/urllib3/connection.py", line 157, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw
File "/usr/lib64/python3.6/site-packages/urllib3/util/connection.py", line 61, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/usr/lib64/python3.6/socket.py", line 745, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 672, in urlopen
chunked=chunked,
File "/usr/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 376, in _make_request
self._validate_conn(conn)
File "/usr/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 994, in _validate_conn
conn.connect()
File "/usr/lib64/python3.6/site-packages/urllib3/connection.py", line 300, in connect
conn = self._new_conn()
File "/usr/lib64/python3.6/site-packages/urllib3/connection.py", line 169, in _new_conn
self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7f67d69eed30>: Failed to establish a new connection: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib64/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/usr/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 720, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/usr/lib64/python3.6/site-packages/urllib3/util/retry.py", line 436, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='zenodo.org', port=443): Max retries exceeded with url: /record/3461164/files/pyspectral_rsr_data.tgz (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f67d69eed30>: Failed to establish a new connection: [Errno -2] Name or service not known',))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/pytroll/scripts/create_msg.py", line 176, in
euro_scene.load([product])
File "/usr/lib/python3.6/site-packages/satpy/scene.py", line 978, in load
keepables = self.generate_composites()
File "/usr/lib/python3.6/site-packages/satpy/scene.py", line 893, in generate_composites
return self._read_composites(nodes)
File "/usr/lib/python3.6/site-packages/satpy/scene.py", line 868, in _read_composites
self._generate_composite(item, keepables)
File "/usr/lib/python3.6/site-packages/satpy/scene.py", line 810, in _generate_composite
keepables,
File "/usr/lib/python3.6/site-packages/satpy/scene.py", line 757, in _get_prereq_datasets
self._generate_composite(prereq_node, keepables)
File "/usr/lib/python3.6/site-packages/satpy/scene.py", line 843, in _generate_composite
**self.attrs)
File "/usr/lib/python3.6/site-packages/satpy/composites/init.py", line 653, in call
self._init_refl3x(projectables)
File "/usr/lib/python3.6/site-packages/satpy/composites/init.py", line 614, in _init_refl3x
self._refl3x = Calculator(_nir.attrs['platform_name'], _nir.attrs['sensor'], _nir.attrs['name'])
File "/usr/lib/python3.6/site-packages/pyspectral/near_infrared_reflectance.py", line 70, in init
super(Calculator, self).init(platform_name, instrument, band, **kwargs)
File "/usr/lib/python3.6/site-packages/pyspectral/radiance_tb_conversion.py", line 134, in init
self._get_rsr()
File "/usr/lib/python3.6/site-packages/pyspectral/radiance_tb_conversion.py", line 143, in _get_rsr
sensor = RelativeSpectralResponse(self.platform_name, self.instrument)
File "/usr/lib/python3.6/site-packages/pyspectral/rsr_reader.py", line 109, in init
self._get_filename()
File "/usr/lib/python3.6/site-packages/pyspectral/rsr_reader.py", line 161, in _get_filename
download_rsr()
File "/usr/lib/python3.6/site-packages/pyspectral/utils.py", line 400, in download_rsr
response = requests.get(HTTP_PYSPECTRAL_RSR)
File "/usr/lib64/python3.6/site-packages/requests/api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "/usr/lib64/python3.6/site-packages/requests/api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/lib64/python3.6/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib64/python3.6/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/usr/lib64/python3.6/site-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='zenodo.org', port=443): Max retries exceeded with url: /record/3461164/files/pyspectral_rsr_data.tgz (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f67d69eed30>: Failed to establish a new connection: [Errno -2] Name or service not known',))
python 3.6, satpy 0.20, pyspectral 0.9.5
In /usr/lib/site-packages/pyspectral/etc/pysprectral.yaml the option for disabling the download from internet. When I set this to "False" the product is created without "actual" rsr-data.
After an update of pyspectral this setting would the file overwritten? So I have to remember, that I have changed this file and restore it after each update.
A workaround from Adam is the following:
Using a customized pyspectral.yaml file and pointing to it with the environment variable PSP_CONFIG_FILE
Today we use the spectral response function and Planck function and make a convolution and create LUT, and thus radiance to Tb conversion can be made fast.
There are other alternatives to solve this, e.g. a brute force or iterative approach as utilised in Libradtran. Check out of this is a better way to solve the conversion, than creating a LUT.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.