data-retrieval-scripts's Issues

Exception in RTMA asyncio download script

@danhamill the asyncio scripts are great! So much speed!

I hit an exception running the asyncio script for RTMA Temperature. I ran for the dates 2016-10-01 00:00 to 2016-10-15 00:00. It's very likely there could be a missing grid in there causing the crash. Can you try reproducing, and see if there is a workaround that doesn't bring the script down? For reference, we added some missing URL logic in https://github.com/HydrologicEngineeringCenter/data-retrieval-scripts/blob/master/retrieve_qpe_gagecorr_01h.py

  File "C:/Projects/data-retrieval-scripts/RTMA_Temp_async_download.py", line 63, in <module>
    results = loop.run_until_complete(main())
  File "C:\Projects\data-retrieval-scripts\venv\lib\site-packages\nest_asyncio.py", line 95, in run_until_complete
    return f.result()
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1729.0_x64__qbz5n2kfra8p0\lib\asyncio\futures.py", line 178, in result
    raise self._exception
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1729.0_x64__qbz5n2kfra8p0\lib\asyncio\tasks.py", line 282, in __step
    result = coro.throw(exc)
  File "C:/Projects/data-retrieval-scripts/RTMA_Temp_async_download.py", line 57, in main
    return await asyncio.gather(*tasks)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1729.0_x64__qbz5n2kfra8p0\lib\asyncio\tasks.py", line 349, in __wakeup
    future.result()
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1729.0_x64__qbz5n2kfra8p0\lib\asyncio\tasks.py", line 282, in __step
    result = coro.throw(exc)
  File "C:/Projects/data-retrieval-scripts/RTMA_Temp_async_download.py", line 26, in download_coroutine
    chunk = await response.content.read(1024)
  File "C:\Projects\data-retrieval-scripts\venv\lib\site-packages\aiohttp\streams.py", line 368, in read
    await self._wait('read')
  File "C:\Projects\data-retrieval-scripts\venv\lib\site-packages\aiohttp\streams.py", line 296, in _wait
    await waiter
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1729.0_x64__qbz5n2kfra8p0\lib\asyncio\futures.py", line 260, in __await__
    yield self  # This tells Task to wait for completion.
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1729.0_x64__qbz5n2kfra8p0\lib\asyncio\tasks.py", line 349, in __wakeup
    future.result()
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1729.0_x64__qbz5n2kfra8p0\lib\asyncio\futures.py", line 178, in result
    raise self._exception
aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed

MRMS GageCorr Data only available before 15 Oct 2020

@tombrauer @mikebartles @ejgumbel @joshuarwillis @FHanbali

I have been looking for recent MRMS data and found the GaugeCorr_QPE grids seem are no longer served on the Iowa State Mesonet servers after 15 Oct 2020.

When MRMS v 12.0 was released on 15 Oct 2020, they started to provide a new data product MultiSensor_QPE_24H_Pass2 grids.

https://inside.nssl.noaa.gov/mrms/past-code-updates/
In the MRMS v12.0 update to NCO section, the third bullet from the bottom:
- Multi-sensor QPE scheme using gauges and model QPFs to fill radar coverage gaps
I think that suggests the MultiSensor_QPE is gage corrected. So the data might have just changed names in the MRMS migration to 12.0.

I am looking for another set of eyes to confirm my assumption the GageCorr_QPE grids have just been renamed to MultiSensor_QPE. I am working a study where we have calibration events pre/post MRMS v 12.0, but I am a bit uneasy mixing the QPE grids.

Have any of you come across this before?

Thanks!

asyncio data retrieval examples

Are you guys interested in posting download scripts that use asyncio?

from datetime import datetime
from datetime import timedelta
import os
import nest_asyncio
nest_asyncio.apply()
import asyncio
import uuid
import aiohttp
import async_timeout


async def download_coroutine(url, session):
    with async_timeout.timeout(1200):
        async with session.get(url) as response:
            if response.status == 200:
                fp = r"C:\workspace\ririe\HMS\data\precip" + os.sep + os.path.basename(url)
                with open(fp, 'wb') as f_handle:
                    while True:
                        chunk = await response.content.read(1024)
                        if not chunk:
                            break
                        f_handle.write(chunk)
            else:
                print(url)
            return await response.release()

async def main(loop):
    
    start = datetime(2015, 1, 22, 0, 0)
    end = datetime(2020, 8, 1, 0, 0)
    hour = timedelta(hours=1)
    
    date = start
    urls = []
    opath = []
    while date < end:
        url = "http://mtarchive.geol.iastate.edu/{:04d}/{:02d}/{:02d}/mrms/ncep/GaugeCorr_QPE_01H/GaugeCorr_QPE_01H_00.00_{:04d}{:02d}{:02d}-{:02d}0000.grib2.gz".format(
        date.year, date.month, date.day, date.year, date.month, date.day, date.hour)
        #print(url)
        destination = r"C:\workspace\ririe\HMS\data\temp"
        
        filename = url.split("/")[-1]
        if not os.path.isfile(destination + os.sep + filename):
            urls.append(url)
            opath.append(destination + os.sep + filename)
        date += hour
        
    async with aiohttp.ClientSession() as session:
        tasks = [download_coroutine(url, session) for url in urls]
        return await asyncio.gather(*tasks)




if __name__ == '__main__':

    loop = asyncio.get_event_loop()
    
    results = loop.run_until_complete(main(loop))

hydrologicengineeringcenter / data-retrieval-scripts Goto Github PK

data-retrieval-scripts's Issues

Exception in RTMA asyncio download script

MRMS GageCorr Data only available before 15 Oct 2020

asyncio data retrieval examples

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent