[SatPy] Can't create ABI full disk true color about pyspectral HOT 23 CLOSED

Hi @djhoese, I just realice that Python version is really important. The performance of the process is incredibly better when I changed to python 3. Since I use python3 the performance allow me to process full disk in 4 minutes and do not take more than 8GB ram. Thanks for all your help

from pyspectral.

Thanks for the bug report. This seems like it is a satpy error and in the future this is probably better on the satpy repository. Anyway...

Could you try running at the top of your code from satpy.utils import debug_on; debug_on() and running your code again and pasting the output here. The error message you got (which I thought I had cleaned up) is saying that the true_color composite wasn't created. The debug messages should show you why.

It's possible satpy is looking for more than just C01, C02, and C03 data, but I can't think of a reason why.

from pyspectral.

[DEBUG: 2018-08-10 11:43:08 : satpy.scene] Setting 'PPP_CONFIG_DIR' to '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc'
[DEBUG: 2018-08-10 11:43:08 : satpy.readers] Reading ['/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/avhrr_eps_l1b.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/avhrr_eps_l1b.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/avhrr_eps_l1b.yaml']
[WARNING: 2018-08-10 11:43:08 : satpy.readers.yaml_reader] No filenames found for reader: avhrr_eps_l1b
[DEBUG: 2018-08-10 11:43:08 : satpy.readers] Reading ['/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/grib.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/grib.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/grib.yaml']
[INFO: 2018-08-10 11:43:08 : satpy.readers] Cannot use ['/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/grib.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/grib.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/grib.yaml']
[DEBUG: 2018-08-10 11:43:08 : satpy.readers] while constructing a Python object
cannot find module 'satpy.readers.grib' (No module named pygrib)
  in "<string>", line 9, column 18:
        file_reader: !!python/name:satpy.readers.grib ... 
                     ^
[DEBUG: 2018-08-10 11:43:08 : satpy.readers] Reading ['/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/hdfeos_l1b.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/hdfeos_l1b.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/hdfeos_l1b.yaml']
[INFO: 2018-08-10 11:43:08 : satpy.readers] Cannot use ['/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/hdfeos_l1b.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/hdfeos_l1b.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/hdfeos_l1b.yaml']
[DEBUG: 2018-08-10 11:43:08 : satpy.readers] while constructing a Python object
cannot find module 'satpy.readers.hdfeos_l1b' (No module named pyhdf.error)
  in "<string>", line 438, column 18:
        file_reader: !!python/name:satpy.readers.hdfe ... 
                     ^
[DEBUG: 2018-08-10 11:43:08 : satpy.readers] Reading ['/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/nc_olci_l1b.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/nc_olci_l1b.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/nc_olci_l1b.yaml']
[WARNING: 2018-08-10 11:43:09 : satpy.readers.yaml_reader] No filenames found for reader: nc_olci_l1b
[DEBUG: 2018-08-10 11:43:09 : satpy.readers] Reading ['/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/mipp_xrit.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/mipp_xrit.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/mipp_xrit.yaml']
[INFO: 2018-08-10 11:43:09 : satpy.readers] Cannot use ['/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/mipp_xrit.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/mipp_xrit.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/mipp_xrit.yaml']
[DEBUG: 2018-08-10 11:43:09 : satpy.readers] while constructing a Python object
cannot find module 'satpy.readers.mipp_xrit' (cannot import name area_defs_to_extent)
  in "<string>", line 6, column 11:
      reader: !!python/name:satpy.readers.mipp ... 
              ^
[DEBUG: 2018-08-10 11:43:09 : satpy.readers] Reading ['/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/nc_olci_l2.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/nc_olci_l2.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/nc_olci_l2.yaml']
[WARNING: 2018-08-10 11:43:09 : satpy.readers.yaml_reader] No filenames found for reader: nc_olci_l2
[DEBUG: 2018-08-10 11:43:09 : satpy.readers] Reading ['/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/viirs_sdr.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/viirs_sdr.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/viirs_sdr.yaml']
[WARNING: 2018-08-10 11:43:10 : satpy.readers.yaml_reader] No filenames found for reader: viirs_sdr
[DEBUG: 2018-08-10 11:43:10 : satpy.readers] Reading ['/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/ghrsst_osisaf.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/ghrsst_osisaf.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/ghrsst_osisaf.yaml']
[WARNING: 2018-08-10 11:43:11 : satpy.readers.yaml_reader] No filenames found for reader: ghrsst_osisaf
[DEBUG: 2018-08-10 11:43:11 : satpy.readers] Reading ['/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/hrit_electrol.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/hrit_electrol.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/hrit_electrol.yaml']
[WARNING: 2018-08-10 11:43:11 : satpy.readers.yaml_reader] No filenames found for reader: hrit_electrol
[DEBUG: 2018-08-10 11:43:11 : satpy.readers] Reading ['/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/mipp_h8.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/mipp_h8.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/mipp_h8.yaml']
[INFO: 2018-08-10 11:43:11 : satpy.readers] Cannot use ['/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/mipp_h8.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/mipp_h8.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/mipp_h8.yaml']
[DEBUG: 2018-08-10 11:43:11 : satpy.readers] while constructing a Python object
cannot find module 'satpy.readers.mipp_xrit' (cannot import name area_defs_to_extent)
  in "<string>", line 7, column 11:
      reader: !!python/name:satpy.readers.mipp ... 
              ^
[DEBUG: 2018-08-10 11:43:11 : satpy.readers] Reading ['/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/aapp_l1b.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/aapp_l1b.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/aapp_l1b.yaml']
[WARNING: 2018-08-10 11:43:11 : satpy.readers.yaml_reader] No filenames found for reader: avhrr_aapp_l1b
[DEBUG: 2018-08-10 11:43:11 : satpy.readers] Reading ['/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/olci.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/olci.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/olci.yaml']
[WARNING: 2018-08-10 11:43:11 : satpy.readers.yaml_reader] No filenames found for reader: nc_olci
[DEBUG: 2018-08-10 11:43:11 : satpy.readers] Reading ['/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/viirs_l1b.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/viirs_l1b.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/viirs_l1b.yaml']
[WARNING: 2018-08-10 11:43:12 : satpy.readers.yaml_reader] No filenames found for reader: viirs_l1b
[DEBUG: 2018-08-10 11:43:12 : satpy.readers] Reading ['/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/abi_l1b.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/abi_l1b.yaml', '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/abi_l1b.yaml']
[DEBUG: 2018-08-10 11:43:12 : satpy.readers.yaml_reader] Assigning to abi_l1b: ['OR_ABI-L1b-RadF-M3C01_G16_s20182211830459_e20182211841226_c20182211841271.nc', 'OR_ABI-L1b-RadF-M3C02_G16_s20182211830459_e20182211841226_c20182211841260.nc', 'OR_ABI-L1b-RadF-M3C03_G16_s20182211830459_e20182211841226_c20182211841272.nc']
[DEBUG: 2018-08-10 11:43:13 : satpy.composites] Looking for composites config file abi.yaml
[DEBUG: 2018-08-10 11:43:13 : satpy.composites] Looking for composites config file visir.yaml
[DEBUG: 2018-08-10 11:43:13 : satpy.scene] Setting 'PPP_CONFIG_DIR' to '/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/etc'
Traceback (most recent call last):
  File "true_pytroll.py", line 29, in <module>
    local_scene.show(rgbname)
  File "/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/scene.py", line 986, in show
    img = get_enhanced_image(self[dataset_id].squeeze(), overlay=overlay)
  File "/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/scene.py", line 573, in __getitem__
    return self.datasets[key]
  File "/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/readers/__init__.py", line 299, in __getitem__
    key = self.get_key(item)
  File "/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/readers/__init__.py", line 288, in get_key
    best=best, **dfilter)
  File "/Users/adioses/anaconda2/lib/python2.7/site-packages/satpy/readers/__init__.py", line 239, in get_key
    raise KeyError("No dataset matching '{}' found".format(str(key)))
KeyError: "No dataset matching 'DatasetID(name='true_color', wavelength=None, resolution=None, polarization=None, calibration=None, level=None, modifiers=None)' found"

from pyspectral.

Ah I should have realized this earlier. You need to actually load the data scn.load(['true_color']).

Edit: Note this should happen right after the Scene creation and before resampling.

from pyspectral.

I got the same error Memory. It's weird because I process full disk image in the same server using GDAL.

[DEBUG: 2018-08-10 12:06:16 : satpy.scene] Setting 'PPP_CONFIG_DIR' to '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc'
[DEBUG: 2018-08-10 12:06:16 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/viirs_compact.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/viirs_compact.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/viirs_compact.yaml']
[WARNING: 2018-08-10 12:06:19 : satpy.readers.yaml_reader] No filenames found for reader: viirs_compact
[DEBUG: 2018-08-10 12:06:19 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/nucaps.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/nucaps.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/nucaps.yaml']
[WARNING: 2018-08-10 12:06:19 : satpy.readers.yaml_reader] No filenames found for reader: nucaps
[DEBUG: 2018-08-10 12:06:19 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/clavrx.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/clavrx.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/clavrx.yaml']
[INFO: 2018-08-10 12:06:19 : satpy.readers] Cannot use ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/clavrx.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/clavrx.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/clavrx.yaml']
[DEBUG: 2018-08-10 12:06:19 : satpy.readers] while constructing a Python object
cannot find module 'satpy.readers.clavrx' (No module named pyhdf.SD)
in "", line 9, column 18:
file_reader: !!python/name:satpy.readers.clav ...
^
[DEBUG: 2018-08-10 12:06:19 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/fci_fdhsi.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/fci_fdhsi.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/fci_fdhsi.yaml']
[WARNING: 2018-08-10 12:06:19 : satpy.readers.yaml_reader] No filenames found for reader: fci_fdhsi
[DEBUG: 2018-08-10 12:06:19 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/grib.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/grib.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/grib.yaml']
[INFO: 2018-08-10 12:06:19 : satpy.readers] Cannot use ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/grib.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/grib.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/grib.yaml']
[DEBUG: 2018-08-10 12:06:19 : satpy.readers] while constructing a Python object
cannot find module 'satpy.readers.grib' (No module named pygrib)
in "", line 9, column 18:
file_reader: !!python/name:satpy.readers.grib ...
^
[DEBUG: 2018-08-10 12:06:19 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/avhrr_hrpt_l1b.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/avhrr_hrpt_l1b.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/avhrr_hrpt_l1b.yaml']
[INFO: 2018-08-10 12:06:19 : satpy.readers] Cannot use ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/avhrr_hrpt_l1b.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/avhrr_hrpt_l1b.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/avhrr_hrpt_l1b.yaml']
[DEBUG: 2018-08-10 12:06:19 : satpy.readers] while constructing a Python object
cannot find module 'satpy.readers.hrpt' (No module named pygac.gac_calibration)
in "", line 93, column 22:
file_reader: !!python/name:satpy.readers.hrpt ...
^
[DEBUG: 2018-08-10 12:06:19 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/omps_edr.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/omps_edr.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/omps_edr.yaml']
[WARNING: 2018-08-10 12:06:20 : satpy.readers.yaml_reader] No filenames found for reader: omps_edr
[DEBUG: 2018-08-10 12:06:20 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/nc_slstr.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/nc_slstr.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/nc_slstr.yaml']
[WARNING: 2018-08-10 12:06:20 : satpy.readers.yaml_reader] No filenames found for reader: nc_slstr
[DEBUG: 2018-08-10 12:06:20 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/safe_sar_c.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/safe_sar_c.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/safe_sar_c.yaml']
[WARNING: 2018-08-10 12:06:20 : satpy.readers.yaml_reader] No filenames found for reader: safe_sar_c
[DEBUG: 2018-08-10 12:06:20 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/hrit_jma.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/hrit_jma.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/hrit_jma.yaml']
[WARNING: 2018-08-10 12:06:21 : satpy.readers.yaml_reader] No filenames found for reader: hrit_jma
[DEBUG: 2018-08-10 12:06:21 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/ghrsst_osisaf.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/ghrsst_osisaf.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/ghrsst_osisaf.yaml']
[WARNING: 2018-08-10 12:06:21 : satpy.readers.yaml_reader] No filenames found for reader: ghrsst_osisaf
[DEBUG: 2018-08-10 12:06:21 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/scatsat1_l2b.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/scatsat1_l2b.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/scatsat1_l2b.yaml']
[WARNING: 2018-08-10 12:06:21 : satpy.readers.yaml_reader] No filenames found for reader: scatsat1_l2b
[DEBUG: 2018-08-10 12:06:21 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/acspo.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/acspo.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/acspo.yaml']
[WARNING: 2018-08-10 12:06:21 : satpy.readers.yaml_reader] No filenames found for reader: acspo
[DEBUG: 2018-08-10 12:06:21 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/nc_nwcsaf_msg.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/nc_nwcsaf_msg.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/nc_nwcsaf_msg.yaml']
[WARNING: 2018-08-10 12:06:21 : satpy.readers.yaml_reader] No filenames found for reader: nc_nwcsaf_msg
[DEBUG: 2018-08-10 12:06:21 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/ahi_hsd.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/ahi_hsd.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/ahi_hsd.yaml']
[WARNING: 2018-08-10 12:06:22 : satpy.readers.yaml_reader] No filenames found for reader: ahi_hsd
[DEBUG: 2018-08-10 12:06:22 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/nc_nwcsaf_pps.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/nc_nwcsaf_pps.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/nc_nwcsaf_pps.yaml']
[WARNING: 2018-08-10 12:06:22 : satpy.readers.yaml_reader] No filenames found for reader: nc_nwcsaf_pps
[DEBUG: 2018-08-10 12:06:22 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/generic_image.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/generic_image.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/generic_image.yaml']
[WARNING: 2018-08-10 12:06:22 : satpy.readers.yaml_reader] No filenames found for reader: generic_image
[DEBUG: 2018-08-10 12:06:22 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/geocat.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/geocat.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/geocat.yaml']
[WARNING: 2018-08-10 12:06:22 : satpy.readers.yaml_reader] No filenames found for reader: geocat
[DEBUG: 2018-08-10 12:06:22 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/nc_olci_l1b.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/nc_olci_l1b.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/nc_olci_l1b.yaml']
[WARNING: 2018-08-10 12:06:23 : satpy.readers.yaml_reader] No filenames found for reader: nc_olci_l1b
[DEBUG: 2018-08-10 12:06:23 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/avhrr_eps_l1b.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/avhrr_eps_l1b.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/avhrr_eps_l1b.yaml']
[WARNING: 2018-08-10 12:06:23 : satpy.readers.yaml_reader] No filenames found for reader: avhrr_eps_l1b
[DEBUG: 2018-08-10 12:06:23 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/hdf4_caliopv3.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/hdf4_caliopv3.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/hdf4_caliopv3.yaml']
[INFO: 2018-08-10 12:06:23 : satpy.readers] Cannot use ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/hdf4_caliopv3.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/hdf4_caliopv3.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/hdf4_caliopv3.yaml']
[DEBUG: 2018-08-10 12:06:23 : satpy.readers] while constructing a Python object
cannot find module 'satpy.readers.hdf4_caliopv3' (No module named pyhdf.SD)
in "", line 40, column 18:
file_reader: !!python/name:satpy.readers.hdf4 ...
^
[DEBUG: 2018-08-10 12:06:23 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/maia.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/maia.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/maia.yaml']
[WARNING: 2018-08-10 12:06:23 : satpy.readers.yaml_reader] No filenames found for reader: maia
[DEBUG: 2018-08-10 12:06:23 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/iasi_l2.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/iasi_l2.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/iasi_l2.yaml']
[WARNING: 2018-08-10 12:06:23 : satpy.readers.yaml_reader] No filenames found for reader: iasi_l2
[DEBUG: 2018-08-10 12:06:23 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/gac_lac_l1.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/gac_lac_l1.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/gac_lac_l1.yaml']
[INFO: 2018-08-10 12:06:23 : satpy.readers] Cannot use ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/gac_lac_l1.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/gac_lac_l1.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/gac_lac_l1.yaml']
[DEBUG: 2018-08-10 12:06:23 : satpy.readers] while constructing a Python object
cannot find module 'satpy.readers.gac_lac_l1' (No module named pygac.gac_calibration)
in "", line 138, column 22:
file_reader: !!python/name:satpy.readers.gac_ ...
^
[DEBUG: 2018-08-10 12:06:23 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/amsr2_l1b.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/amsr2_l1b.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/amsr2_l1b.yaml']
[WARNING: 2018-08-10 12:06:24 : satpy.readers.yaml_reader] No filenames found for reader: amsr2_l1b
[DEBUG: 2018-08-10 12:06:24 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/viirs_l1b.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/viirs_l1b.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/viirs_l1b.yaml']
[WARNING: 2018-08-10 12:06:24 : satpy.readers.yaml_reader] No filenames found for reader: viirs_l1b
[DEBUG: 2018-08-10 12:06:24 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/native_msg.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/native_msg.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/native_msg.yaml']
[WARNING: 2018-08-10 12:06:25 : satpy.readers.yaml_reader] No filenames found for reader: native_msg
[DEBUG: 2018-08-10 12:06:25 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/hrit_goes.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/hrit_goes.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/hrit_goes.yaml']
[WARNING: 2018-08-10 12:06:25 : satpy.readers.yaml_reader] No filenames found for reader: hrit_goes
[DEBUG: 2018-08-10 12:06:25 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/avhrr_aapp_l1b.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/avhrr_aapp_l1b.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/avhrr_aapp_l1b.yaml']
[WARNING: 2018-08-10 12:06:25 : satpy.readers.yaml_reader] No filenames found for reader: avhrr_aapp_l1b
[DEBUG: 2018-08-10 12:06:25 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/nc_olci_l2.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/nc_olci_l2.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/nc_olci_l2.yaml']
[WARNING: 2018-08-10 12:06:25 : satpy.readers.yaml_reader] No filenames found for reader: nc_olci_l2
[DEBUG: 2018-08-10 12:06:25 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/safe_msi.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/safe_msi.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/safe_msi.yaml']
[INFO: 2018-08-10 12:06:25 : satpy.readers] Cannot use ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/safe_msi.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/safe_msi.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/safe_msi.yaml']
[DEBUG: 2018-08-10 12:06:25 : satpy.readers] while constructing a Python object
cannot find module 'satpy.readers.safe_msi' (No module named glymur)
in "", line 10, column 22:
file_reader: !!python/name:satpy.readers.safe ...
^
[DEBUG: 2018-08-10 12:06:25 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/hrit_msg.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/hrit_msg.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/hrit_msg.yaml']
[WARNING: 2018-08-10 12:06:26 : satpy.readers.yaml_reader] No filenames found for reader: hrit_msg
[DEBUG: 2018-08-10 12:06:26 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/abi_l1b.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/abi_l1b.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/abi_l1b.yaml']
[DEBUG: 2018-08-10 12:06:26 : satpy.readers.yaml_reader] Assigning to abi_l1b: ['OR_ABI-L1b-RadF-M3C01_G16_s20182211830459_e20182211841226_c20182211841271.nc', 'OR_ABI-L1b-RadF-M3C02_G16_s20182211830459_e20182211841226_c20182211841260.nc', 'OR_ABI-L1b-RadF-M3C03_G16_s20182211830459_e20182211841226_c20182211841272.nc']
[DEBUG: 2018-08-10 12:06:27 : satpy.composites] Looking for composites config file abi.yaml
[DEBUG: 2018-08-10 12:06:27 : satpy.composites] Looking for composites config file visir.yaml
[DEBUG: 2018-08-10 12:06:27 : satpy.node] Skipping optional satellite_zenith_angle: Unknown dataset satellite_zenith_angle
[DEBUG: 2018-08-10 12:06:27 : satpy.node] Skipping optional satellite_azimuth_angle: Unknown dataset satellite_azimuth_angle
[DEBUG: 2018-08-10 12:06:27 : satpy.node] Skipping optional satellite_zenith_angle: Unknown dataset satellite_zenith_angle
[DEBUG: 2018-08-10 12:06:27 : satpy.node] Skipping optional solar_azimuth_angle: Unknown dataset solar_azimuth_angle
[DEBUG: 2018-08-10 12:06:27 : satpy.node] Skipping optional solar_zenith_angle: Unknown dataset solar_zenith_angle
[DEBUG: 2018-08-10 12:06:27 : satpy.node] Skipping optional satellite_zenith_angle: Unknown dataset satellite_zenith_angle
[DEBUG: 2018-08-10 12:06:27 : satpy.node] Skipping optional satellite_azimuth_angle: Unknown dataset satellite_azimuth_angle
[DEBUG: 2018-08-10 12:06:27 : satpy.node] Skipping optional satellite_zenith_angle: Unknown dataset satellite_zenith_angle
[DEBUG: 2018-08-10 12:06:27 : satpy.node] Skipping optional solar_azimuth_angle: Unknown dataset solar_azimuth_angle
[DEBUG: 2018-08-10 12:06:27 : satpy.node] Skipping optional solar_zenith_angle: Unknown dataset solar_zenith_angle
[DEBUG: 2018-08-10 12:06:27 : satpy.node] Skipping optional satellite_zenith_angle: Unknown dataset satellite_zenith_angle
[DEBUG: 2018-08-10 12:06:27 : satpy.readers.abi_l1b] Reading in get_dataset C03.
[DEBUG: 2018-08-10 12:06:27 : satpy.readers.abi_l1b] Calibrating to reflectances
[DEBUG: 2018-08-10 12:06:27 : satpy.readers.abi_l1b] Reading in get_dataset C01.
[DEBUG: 2018-08-10 12:06:27 : satpy.readers.abi_l1b] Calibrating to reflectances
[DEBUG: 2018-08-10 12:06:27 : satpy.readers.abi_l1b] Reading in get_dataset C02.
[DEBUG: 2018-08-10 12:06:27 : satpy.readers.abi_l1b] Calibrating to reflectances
[WARNING: 2018-08-10 12:06:27 : satpy.readers.yaml_reader] Can't load ancillary dataset DQF
[WARNING: 2018-08-10 12:06:27 : satpy.readers.yaml_reader] Can't load ancillary dataset DQF
[WARNING: 2018-08-10 12:06:27 : satpy.readers.yaml_reader] Can't load ancillary dataset DQF
[DEBUG: 2018-08-10 12:06:27 : satpy.composites] Applying sun zen correction
[DEBUG: 2018-08-10 12:06:27 : satpy.composites] Computing sun zenith angles.
Traceback (most recent call last):
File "true_pytroll.py", line 27, in
scn.load([rgbname])
File "/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/scene.py", line 850, in load
keepables = self.generate_composites()
File "/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/scene.py", line 765, in generate_composites
return self._read_composites(nodes)
File "/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/scene.py", line 739, in _read_composites
self.generate_composite(item, keepables)
File "/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/scene.py", line 712, in generate_composite
**self.attrs)
File "/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/composites/init.py", line 341, in call
lons, lats),
File "/home/satelite/anaconda2/lib/python2.7/site-packages/pyorbital-1.3.1-py2.7.egg/pyorbital/astronomy.py", line 144, in cos_zen
return (np.sin(lat) * np.sin(dec) + np.cos(lat) * np.cos(dec) * np.cos(h))
MemoryError

from pyspectral.

A couple things that may be a problem:

1. Instead of specifying sensor when you are creating your Scene, use reader='abi_l1b'. This will skip searching for a reader and will speed up the initial loading time of your script.
2. What version of satpy are you using (satpy.__version__)? If you aren't on 0.9 then you will probably not be able to create a full disk true color image. Version 0.9 was satpy's first version using dask and xarray which allowed us major improvements in processing time and memory usage. We provide satpy 0.9 in the conda-forge channel for conda if you aren't already using it, but pip should be fine too.
3. Do you need to be on python 2? This isn't really an issue for performance, but if this is a new project you are starting I highly recommend moving to python 3.

Lastly, satpy isn't simply putting the bands together. It is configured to perform multiple corrections to the data to produce the best true color image we know how to make. This includes a solar-zenith correction and a rayleigh scattering correction. The rayleigh correction requires a lot of time and memory compared to not using it. This is why gdal might not have memory issues (among other things) when creating full disk imagery. You could try loading true_color_raw which won't do the rayleigh correction.

I'm curious, what full disk imagery were you making with GDAL? Was it an ABI true color? How did you make the green band?

from pyspectral.

Thank, I'll make the changes you just suggest.
I use Satpy 0.9, I'll change to python 3 and sensor to reader.

I completely understand that, this are not simply correction and take a lot of resources. I would read more about this correction. I'm trying to create corrected true color images operational, so I need to speed up the process.

I process ABI true color. In order to produce Green Band I use a equation I found in this web (http://edc.occ-data.org/goes16/python/)
GreenB = 0.48358168 * ch2 + 0.45706946 * ch1 + 0.06038137 * ch3.
Then I make some brightness adjustment, There is not physical sense on that is just to improve visualization.

from pyspectral.

I ran the script with CONUS images and the output is great. Thank you so much.

Also, I found an error: Platform file /home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/platforms.txt not found. how could I correct this?

from pyspectral.

Don't worry about that message. It is more of a warning than an error and is actually expected in satpy right now. @adybbroe is there a solution for this platforms.txt being missing? Can't we stop the message from appearing in satpy?

from pyspectral.

Also FYI @adioses I run satpy on my laptop (macbook with 16GB memory) and can produce an ABI true color full disk image in ~8 minutes (maybe less, can't remember). Another thing I just noticed is that you said you have 12 cores. This may be the reason you are running out of memory since by default dask (the lower level array library we're using) will create a worker thread for each logical core on your system. You may want to configure it to have less workers, something like 8, following the directions using ThreadPool here: http://dask.pydata.org/en/latest/scheduler-overview.html#configuring-the-schedulers

from pyspectral.

Thanks. I’ll try it. When I tried to run South America took like an hour and got out off memory.

Also I’ll try to reduce my processing area. I want to focus on South America and not in CONUS sector

from pyspectral.

Hhhhmmm it should never take an hour with satpy 0.9+. Make sure you have semi-recent versions of xarray, dask, pyresample, and trollimage.

from pyspectral.

I updated all the libraries you mentioned. But I still have problems with FullDisk Images.
I couldn't configure dask. It's not really clear for me I'll keep trying.
Otherwise, I would like to know if there is a way to read just part of the file so it process less information or I should modify directly the NetCDF file?

from pyspectral.

Configuring the number of workers is the easiest way to limit the amount of memory used since you will only have so many workers working at a single moment. As for configuring dask, right after the imports in your script do:

import dask
from multiprocessing.pool import ThreadPool
with dask.config.set(pool=ThreadPool(4)):
    # all of your other satpy-based code goes in this block of code including the .show call

SatPy is already attempting to minimize the amount of data being processed when you resampled the data. There is another method but it currently has a bug so I'm not going to suggest it right now.

Lastly, you can configure the "chunks" that dask creates from the data arrays. Each chunk is processed separately on an individual dask worker when possible. You can configure this right now by specifying an environment variable although we hope to make it easier in the future. You can set PYTROLL_CHUNK_SIZE to a value between 1024 and 4096 (default) on the command line before running your script. For bash that would be:

export PYTROLL_CHUNK_SIZE=2048
python your_script.py

Lastly...again, you could also provide cache_dir='/path/where/to/put/cache/files/' to your resample call. This won't fix your memory errors but it will make processing of the same geostationary area much faster on future executions.

from pyspectral.

My code now is this:

from satpy.utils import debug_on; debug_on()
from satpy import Scene, find_files_and_readers
from satpy.resample import get_area_def
from pyresample.utils import get_area_def
import dask
from multiprocessing.pool import ThreadPool
with dask.config.set(pool=ThreadPool(4)):

	scn = Scene(reader='abi_l1b', filenames=['OR_ABI-L1b-RadC-M3C01_G16_s20182211832268_e20182211835041_c20182211835090.nc','OR_ABI-L1b-RadC-M3C02_G16_s20182211832268_e20182211835041_c20182211835086.nc','OR_ABI-L1b-RadC-M3C03_G16_s20182211832268_e20182211835041_c20182211835089.nc'])
	rgbname = 'true_color'
	scn.load([rgbname])
	new_scn = scn.resample(scn.min_area(), resampler='native')
	new_scn.save_dataset(rgbname, filename='GOES_{datetime}.png'.format(datetime=scn.start_time.strftime('%Y%m%d%H%M')))

Now I have this error:
[DEBUG: 2018-08-10 16:15:59 : satpy.scene] Setting 'PPP_CONFIG_DIR' to '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc'
[DEBUG: 2018-08-10 16:15:59 : satpy.readers] Reading ['/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/abi_l1b.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/abi_l1b.yaml', '/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/etc/readers/abi_l1b.yaml']
[DEBUG: 2018-08-10 16:15:59 : satpy.readers.yaml_reader] Assigning to abi_l1b: ['OR_ABI-L1b-RadC-M3C01_G16_s20182211832268_e20182211835041_c20182211835090.nc', 'OR_ABI-L1b-RadC-M3C02_G16_s20182211832268_e20182211835041_c20182211835086.nc', 'OR_ABI-L1b-RadC-M3C03_G16_s20182211832268_e20182211835041_c20182211835089.nc']
Traceback (most recent call last):
File "true_pytroll.py", line 9, in
scn = Scene(reader='abi_l1b', filenames=['OR_ABI-L1b-RadC-M3C01_G16_s20182211832268_e20182211835041_c20182211835090.nc','OR_ABI-L1b-RadC-M3C02_G16_s20182211832268_e20182211835041_c20182211835086.nc','OR_ABI-L1b-RadC-M3C03_G16_s20182211832268_e20182211835041_c20182211835089.nc'])
File "/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/scene.py", line 151, in init
reader_kwargs=reader_kwargs)
File "/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/scene.py", line 196, in create_reader_instances
ppp_config_dir=self.ppp_config_dir)
File "/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/readers/init.py", line 599, in load_readers
reader_instance.create_filehandlers(loadables)
File "/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/readers/yaml_reader.py", line 493, in create_filehandlers
filename_set)
File "/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/readers/yaml_reader.py", line 480, in new_filehandlers_for_filetype
return list(filtered_iter)
File "/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/readers/yaml_reader.py", line 453, in filter_fh_by_metadata
for filehandler in filehandlers:
File "/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/readers/yaml_reader.py", line 396, in new_filehandler_instances
yield filetype_cls(filename, filename_info, filetype_info, *req_fh)
File "/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/readers/abi_l1b.py", line 57, in init
chunks={'x': CHUNK_SIZE, 'y': CHUNK_SIZE})
File "/home/satelite/anaconda2/lib/python2.7/site-packages/xarray/backends/api.py", line 320, in open_dataset
**backend_kwargs)
File "/home/satelite/anaconda2/lib/python2.7/site-packages/xarray/backends/netCDF4_.py", line 332, in open
ds = opener()
File "/home/satelite/anaconda2/lib/python2.7/site-packages/xarray/backends/netCDF4_.py", line 231, in _open_netcdf4_group
ds = nc4.Dataset(filename, mode=mode, **kwargs)
File "netCDF4/_netCDF4.pyx", line 2015, in netCDF4._netCDF4.Dataset.init
File "netCDF4/_netCDF4.pyx", line 1636, in netCDF4._netCDF4._ensure_nc_success
IOError: [Errno 2] No such file or directory: '/home/satelite/Documents/GOES16/AB1_L1B/OR_ABI-L1b-RadC-M3C01_G16_s20182211832268_e20182211835041_c20182211835090.nc'
Exception AttributeError: AttributeError("'NC_ABI_L1B' object has no attribute 'nc'",) in <bound method NC_ABI_L1B.del of <NC_ABI_L1B: 'OR_ABI-L1b-RadC-M3C01_G16_s20182211832268_e20182211835041_c20182211835090.nc'>> ignored

The file and the code are in the same folder and the files exist

from pyspectral.

What does ls /home/satelite/Documents/GOES16/AB1_L1B/OR_ABI-L1b-RadC-M3C01_G16_s20182211832268_e20182211835041_c20182211835090.nc give you?

What about ls /home/satelite/Documents/GOES16/AB1_L1B/true_pytroll.py?

Edit: And are you running this from '/home/satelite/Documents/GOES16/AB1_L1B/?

from pyspectral.

Sorry. Omit the error, I changed the files that caused the error.
One question, in this line

with dask.config.set(pool=ThreadPool(4)):

4 is the number of cores to use?

Also, If I check the process with top command the script don't use more than one core.

from pyspectral.

Correct, my laptop creates 8 workers. Technically this is number of threads not cores where to me "cores" has the assumption that we're talking about multiple processes. I think you have to specify extra flags in top to get it to show you threads for a process, but that may depend on the operating system/version of top.

from pyspectral.

Same error

[DEBUG: 2018-08-10 17:03:48 : satpy.composites] Computing sun zenith angles.
Traceback (most recent call last):
File "true_pytroll.py", line 11, in
scn.load([rgbname])
File "/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/scene.py", line 850, in load
keepables = self.generate_composites()
File "/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/scene.py", line 765, in generate_composites
return self._read_composites(nodes)
File "/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/scene.py", line 739, in _read_composites
self.generate_composite(item, keepables)
File "/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/scene.py", line 712, in generate_composite
**self.attrs)
File "/home/satelite/anaconda2/lib/python2.7/site-packages/satpy/composites/init.py", line 341, in call
lons, lats),
File "/home/satelite/anaconda2/lib/python2.7/site-packages/pyorbital-1.3.1-py2.7.egg/pyorbital/astronomy.py", line 144, in cos_zen
return (np.sin(lat) * np.sin(dec) + np.cos(lat) * np.cos(dec) * np.cos(h))
MemoryError

I tried using ThreadPool(4) and ThreadPool(8). Also I define export PYTROLL_CHUNK_SIZE=2048

from pyspectral.

@djhoese Can we conclude that this is rather a Satpy issue than a Pyspectral one?

from pyspectral.

Sorry @adioses, i lost track of this issue. Adam yes it is a satpy or pyorbital issue but im not sure how much more I can offer as help. The newest versions of every library don't produce memory issues for most people.

from pyspectral.

Okay, I close it here then. Hope the memory error is no more an issue for you @adioses

from pyspectral.

@adioses Feel free to comment here or create a new issue on the satpy repository if you still have issues.

from pyspectral.