Giter Club home page Giter Club logo

data-retrieval-scripts's People

Contributors

danhamill avatar ejgumbel avatar mikebartles avatar tombrauer avatar

Stargazers

avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar

Forkers

danhamill

data-retrieval-scripts's Issues

asyncio data retrieval examples

Are you guys interested in posting download scripts that use asyncio?

from datetime import datetime
from datetime import timedelta
import os
import nest_asyncio
nest_asyncio.apply()
import asyncio
import uuid
import aiohttp
import async_timeout


async def download_coroutine(url, session):
    with async_timeout.timeout(1200):
        async with session.get(url) as response:
            if response.status == 200:
                fp = r"C:\workspace\ririe\HMS\data\precip" + os.sep + os.path.basename(url)
                with open(fp, 'wb') as f_handle:
                    while True:
                        chunk = await response.content.read(1024)
                        if not chunk:
                            break
                        f_handle.write(chunk)
            else:
                print(url)
            return await response.release()

async def main(loop):
    
    start = datetime(2015, 1, 22, 0, 0)
    end = datetime(2020, 8, 1, 0, 0)
    hour = timedelta(hours=1)
    
    date = start
    urls = []
    opath = []
    while date < end:
        url = "http://mtarchive.geol.iastate.edu/{:04d}/{:02d}/{:02d}/mrms/ncep/GaugeCorr_QPE_01H/GaugeCorr_QPE_01H_00.00_{:04d}{:02d}{:02d}-{:02d}0000.grib2.gz".format(
        date.year, date.month, date.day, date.year, date.month, date.day, date.hour)
        #print(url)
        destination = r"C:\workspace\ririe\HMS\data\temp"
        
        filename = url.split("/")[-1]
        if not os.path.isfile(destination + os.sep + filename):
            urls.append(url)
            opath.append(destination + os.sep + filename)
        date += hour
        
    async with aiohttp.ClientSession() as session:
        tasks = [download_coroutine(url, session) for url in urls]
        return await asyncio.gather(*tasks)




if __name__ == '__main__':

    loop = asyncio.get_event_loop()
    
    results = loop.run_until_complete(main(loop))

Exception in RTMA asyncio download script

@danhamill the asyncio scripts are great! So much speed!

I hit an exception running the asyncio script for RTMA Temperature. I ran for the dates 2016-10-01 00:00 to 2016-10-15 00:00. It's very likely there could be a missing grid in there causing the crash. Can you try reproducing, and see if there is a workaround that doesn't bring the script down? For reference, we added some missing URL logic in https://github.com/HydrologicEngineeringCenter/data-retrieval-scripts/blob/master/retrieve_qpe_gagecorr_01h.py

  File "C:/Projects/data-retrieval-scripts/RTMA_Temp_async_download.py", line 63, in <module>
    results = loop.run_until_complete(main())
  File "C:\Projects\data-retrieval-scripts\venv\lib\site-packages\nest_asyncio.py", line 95, in run_until_complete
    return f.result()
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1729.0_x64__qbz5n2kfra8p0\lib\asyncio\futures.py", line 178, in result
    raise self._exception
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1729.0_x64__qbz5n2kfra8p0\lib\asyncio\tasks.py", line 282, in __step
    result = coro.throw(exc)
  File "C:/Projects/data-retrieval-scripts/RTMA_Temp_async_download.py", line 57, in main
    return await asyncio.gather(*tasks)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1729.0_x64__qbz5n2kfra8p0\lib\asyncio\tasks.py", line 349, in __wakeup
    future.result()
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1729.0_x64__qbz5n2kfra8p0\lib\asyncio\tasks.py", line 282, in __step
    result = coro.throw(exc)
  File "C:/Projects/data-retrieval-scripts/RTMA_Temp_async_download.py", line 26, in download_coroutine
    chunk = await response.content.read(1024)
  File "C:\Projects\data-retrieval-scripts\venv\lib\site-packages\aiohttp\streams.py", line 368, in read
    await self._wait('read')
  File "C:\Projects\data-retrieval-scripts\venv\lib\site-packages\aiohttp\streams.py", line 296, in _wait
    await waiter
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1729.0_x64__qbz5n2kfra8p0\lib\asyncio\futures.py", line 260, in __await__
    yield self  # This tells Task to wait for completion.
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1729.0_x64__qbz5n2kfra8p0\lib\asyncio\tasks.py", line 349, in __wakeup
    future.result()
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1729.0_x64__qbz5n2kfra8p0\lib\asyncio\futures.py", line 178, in result
    raise self._exception
aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed

MRMS GageCorr Data only available before 15 Oct 2020

@tombrauer @mikebartles @ejgumbel @joshuarwillis @FHanbali

I have been looking for recent MRMS data and found the GaugeCorr_QPE grids seem are no longer served on the Iowa State Mesonet servers after 15 Oct 2020.

When MRMS v 12.0 was released on 15 Oct 2020, they started to provide a new data product MultiSensor_QPE_24H_Pass2 grids.

https://inside.nssl.noaa.gov/mrms/past-code-updates/
In the MRMS v12.0 update to NCO section, the third bullet from the bottom:
- Multi-sensor QPE scheme using gauges and model QPFs to fill radar coverage gaps
I think that suggests the MultiSensor_QPE is gage corrected. So the data might have just changed names in the MRMS migration to 12.0.

I am looking for another set of eyes to confirm my assumption the GageCorr_QPE grids have just been renamed to MultiSensor_QPE. I am working a study where we have calibration events pre/post MRMS v 12.0, but I am a bit uneasy mixing the QPE grids.

Have any of you come across this before?

Thanks!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.