Giter Club home page Giter Club logo

irv-autopkg-client's Introduction

irv-autopkg-client

PyPI

A client library for accessing the irv-autopkg API, currently hosted at global.infrastructureresilience.org.

The irv-autopkg service allows users to extract portions of global datasets pertaining to climate risk and resilience. This Python package is a client for communicating with the irv-autopkg API.

Installation

Install from PyPI with:

pip install irv-autopkg-client

Usage

Create a client object to establish a session:

import irv_autopkg_client
client = irv_autopkg_client.Client()

For a list of available methods, try:

help(client)

Quick start

Is the API responding?

client.server_readiness()

Which boundaries can we create extracts for?

client.boundary_list()

Which datasets are available?

client.dataset_list()

Get information on a specific dataset:

client.dataset("wri_aqueduct.version_2")

To submit an extract job:

job_id = client.job_submit(
    country_iso,
    [
        "gri_osm.roads_and_rail_version_1",
        "wri_aqueduct.version_2"
    ]
)

We can then check if the job is complete:

client.job_complete(job_id)

Get the boundary of a territory:

boundary = client.boundary_geometry("bgd")

Download some extracted data:

client.extract_download(
    "bgd",
    "data",
    # there may be other datasets available, but only download the following
    dataset_filter=[
        "gri_osm.roads_and_rail_version_1",
        "wri_aqueduct.version_2"
    ],
    overwrite=True
)

Development

First clone this repository (currently hosted at nismod/irv-autopkg-client).

To install the package in editable mode, run:

pip install -e .
pip install vcrpy~=4.0

Alternatively, if you have poetry, run:

poetry install

Testing

To run the bundled tests, try:

python -m unittest

With poetry, either work within poetry shell, or run single commands in the virtual environment:

poetry run python -m unittest

Acknowledgments

This research received funding from the FCDO Climate Compatible Growth Programme. The views expressed here do not necessarily reflect the UK government's official policies.

irv-autopkg-client's People

Contributors

thomas-fred avatar tomalrussell avatar

Stargazers

 avatar

Watchers

 avatar  avatar

irv-autopkg-client's Issues

Check memory usage/write speed in file downloads

We can likely improve memory usage and/or write speed in file downloads:

with open(file_path, "wb") as file:
response = requests.get(url)
file.write(response.content)

The usual pattern in requests with large files is to stream the response and write from r.iter_content, to avoid reading the whole response into memory:

with requests.get(url, stream=True) as r:
    with open(filename, 'wb') as f:
        for chunk in r.iter_content(chunk_size=8192):
            f.write(chunk)

It might also be worth looking at writing directly from the raw response, which wouldn't handle transfer encoding, but should hand off streaming and writing - see https://requests.readthedocs.io/en/latest/user/quickstart/#raw-response-content and discussion under SO answer:

with requests.get(url, stream=True) as r:
    with open(filename, 'wb') as f:
        shutil.copyfileobj(r.raw, f)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.