<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/us

MRMS GageCorr Data only available before 15 Oct 2020 about data-retrieval-scripts HOT 2 CLOSED

danhamill commented on July 18, 2024

MRMS GageCorr Data only available before 15 Oct 2020

from data-retrieval-scripts.

Comments (2)

FHanbali commented on July 18, 2024

Hi Daniel, yes, we encountered this on Cumulus. Your impressions are the same as ours. GageCorr transition to latest MRMS products, with Pass2 incorporating more corrections than Pass1, but Pass2 lags Pass1 by a few hours in near realtime availability

from data-retrieval-scripts.

danhamill commented on July 18, 2024

@FHanbali Thanks for your input.

As of now, I have split the MRMS QPE download scripts into two scripts:

pre MRMS v12.0

data-retrieval-scripts/MRMS_GageCorr_qpe_async_download.py

Lines 36 to 56 in 6a930e6

 start = datetime(2018, 12, 1, 0, 0) 

 end = datetime(2019,1, 1, 0, 0) 

 hour = timedelta(hours=1) 

 destination = r"C:\workspace\ririe\HMS\data\precip" 

 assert start < datetime(2020,10,15), "GageCorr MRMS data before 2020-10-15 does not exist, consider looking for MultiSensor qpe grids" 

 assert end < datetime(2020,10,15), "GageCorr MRMS data before 2020-10-15 does not exist, consider looking for MultiSensor qpe grids" 

 #loop through and see if you already have the file locally 

 date = start 

 urls = [] 

 opath = [] 

 while date < end: 

 url = "http://mtarchive.geol.iastate.edu/{:04d}/{:02d}/{:02d}/mrms/ncep/GaugeCorr_QPE_01H/GaugeCorr_QPE_01H_00.00_{:04d}{:02d}{:02d}-{:02d}0000.grib2.gz".format( 

 date.year, date.month, date.day, date.year, date.month, date.day, date.hour) 

 filename = url.split("/")[-1] 

 if not os.path.isfile(destination + os.sep + filename): 

 urls.append(url) 

 opath.append(destination + os.sep + filename) 

 date += hour

post MRMS v12.0

data-retrieval-scripts/MRMS_multiSensor_qpe_async_download.py

Lines 36 to 70 in 6a930e6

 start = datetime(2021, 10, 1, 0, 0) 

 end = datetime(2021, 10, 2, 0, 0) 

 destination = r"C:\workspace\ririe\HMS\data\precip" 

 assert start >= datetime(2020,10,15), "MultiSensor MRMS data before 2020-10-15 does not exist, consider looking for GageCorr qpe grids" 

 assert end >= datetime(2020,10,15), "MultiSensor MRMS data before 2020-10-15 does not exist, consider looking for GageCorr qpe grids" 

 """ 

  https://inside.nssl.noaa.gov/mrms/past-code-updates/ 

  In the MRMS v12.0 update to NCO section, the third bullet from the bottom: 

  - Multi-sensor QPE scheme using gauges and model QPFs to fill radar coverage gaps 

  I think that suggests the MultiSensor_QPE is gage corrected. So the data might have just changed names in the MRMS migration to 12.0. 

  """ 

 hour = timedelta(hours=1) 

 os.makedirs(destination, exist_ok=True) 

 #loop through and see if you already have the file locally 

 date = start 

 urls = [] 

 opath = [] 

 while date < end: 

 url = 'https://mtarchive.geol.iastate.edu/{:04d}/{:02d}/{:02d}/mrms/ncep/MultiSensor_QPE_01H_Pass2/MultiSensor_QPE_01H_Pass2_00.00_{:04d}{:02d}{:02d}-{:02d}0000.grib2.gz'.format( 

 date.year, date.month, date.day, date.year, date.month, date.day, date.hour) 

 filename = url.split("/")[-1] 

 if not os.path.isfile(destination + os.sep + filename): 

 urls.append(url) 

 opath.append(destination + os.sep + filename) 

 date += hour

These could easily be combined with some additional conditional logic when building the download urls.

from data-retrieval-scripts.

MRMS GageCorr Data only available before 15 Oct 2020 about data-retrieval-scripts HOT 2 CLOSED

Comments (2)

Related Issues (3)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent

	start = datetime(2018, 12, 1, 0, 0)
	end = datetime(2019,1, 1, 0, 0)
	hour = timedelta(hours=1)
	destination = r"C:\workspace\ririe\HMS\data\precip"

	assert start < datetime(2020,10,15), "GageCorr MRMS data before 2020-10-15 does not exist, consider looking for MultiSensor qpe grids"
	assert end < datetime(2020,10,15), "GageCorr MRMS data before 2020-10-15 does not exist, consider looking for MultiSensor qpe grids"

	#loop through and see if you already have the file locally
	date = start
	urls = []
	opath = []
	while date < end:
	url = "http://mtarchive.geol.iastate.edu/{:04d}/{:02d}/{:02d}/mrms/ncep/GaugeCorr_QPE_01H/GaugeCorr_QPE_01H_00.00_{:04d}{:02d}{:02d}-{:02d}0000.grib2.gz".format(
	date.year, date.month, date.day, date.year, date.month, date.day, date.hour)

	filename = url.split("/")[-1]
	if not os.path.isfile(destination + os.sep + filename):
	urls.append(url)
	opath.append(destination + os.sep + filename)
	date += hour

	start = datetime(2021, 10, 1, 0, 0)
	end = datetime(2021, 10, 2, 0, 0)
	destination = r"C:\workspace\ririe\HMS\data\precip"


	assert start >= datetime(2020,10,15), "MultiSensor MRMS data before 2020-10-15 does not exist, consider looking for GageCorr qpe grids"
	assert end >= datetime(2020,10,15), "MultiSensor MRMS data before 2020-10-15 does not exist, consider looking for GageCorr qpe grids"


	"""
	https://inside.nssl.noaa.gov/mrms/past-code-updates/

	In the MRMS v12.0 update to NCO section, the third bullet from the bottom:
	- Multi-sensor QPE scheme using gauges and model QPFs to fill radar coverage gaps
	I think that suggests the MultiSensor_QPE is gage corrected. So the data might have just changed names in the MRMS migration to 12.0.
	"""

	hour = timedelta(hours=1)
	os.makedirs(destination, exist_ok=True)

	#loop through and see if you already have the file locally
	date = start
	urls = []
	opath = []
	while date < end:

	url = 'https://mtarchive.geol.iastate.edu/{:04d}/{:02d}/{:02d}/mrms/ncep/MultiSensor_QPE_01H_Pass2/MultiSensor_QPE_01H_Pass2_00.00_{:04d}{:02d}{:02d}-{:02d}0000.grib2.gz'.format(
	date.year, date.month, date.day, date.year, date.month, date.day, date.hour)

	filename = url.split("/")[-1]
	if not os.path.isfile(destination + os.sep + filename):
	urls.append(url)
	opath.append(destination + os.sep + filename)
	date += hour