Giter Club home page Giter Club logo

gridpp's Introduction

Gridded post-processor

"Latest release" C/C++ CI

Gridpp a is post-processing tool for gridded weather forecasts. It consists of a library of commonly-used methods and a command-line tool that applies these methods to forecast fields in NetCDF files.

Gridpp is written in C++ but offers python bindings to the functions in the library. The tool is used at MET Norway to produce operational weather forecasts for Yr (https://www.yr.no).

Gridpp is currently under active development and the current version is a prototype for testing. We welcome suggesions for improvements.

Features

  • Methods for downscaling a forecast from a coarse grid to a fine grid
  • Methods for calibrating a downscaled grid, such as quantile mapping
  • Computationally efficient neighbourhood methods to compute neighbourhood min, mean, max, and any quantile.
  • Data assimilation using optimal interpolation (OI) to merge observations and gridded forecasts (deterministic or ensemble)
  • Efficient data structures for nearest location lookup in a vector or grid of locations
  • Command-line client with support for Netcdf files with flexibility in how variables and dimensions are configured

Resources

For information on how to use gridpp, check out the wiki at https://github.com/metno/gridpp/wiki. The API reference is found at https://metno.github.io/gridpp/. Found a bug? Please report it in the issue tracker at https://github.com/metno/gridpp/issues. Reach out to the gridpp community in the discussions at https://github.com/metno/gridpp/discussions. Or contact Thomas Nipen ([email protected]) directly.

Getting started

The easiest way to get started is to install gridpp for python using:

pip install gridpp

Let's say you have a gridded background field and want to merge this with a set of point observations. We can use gridpp.optimal_interpolation() for this! Just run the following:

import gridpp
import numpy as np
import scipy.ndimage.filters

# Create a nice background field and define its grid
noise = np.random.randn(200, 200) * 3
field = scipy.ndimage.filters.gaussian_filter(noise, sigma=5)
grid = gridpp.Grid(*np.meshgrid(np.linspace(0, 1, 200), np.linspace(0, 1, 200)))

# Next, create some trustworthy observations
points = gridpp.Points(np.random.rand(10), np.random.rand(10))
obs = np.random.randn(10) / 2
pobs = gridpp.nearest(grid, points, field)
obs_to_background_variance_ratio = 0.5*np.ones(10)

# Run optimal interpolation with a Barnes structure function (10km e-folding distance)
structure = gridpp.BarnesStructure(10000)
max_points = 10
output = gridpp.optimal_interpolation(grid, field, points, obs,
                                      obs_to_background_variance_ratio,
                                      pobs, structure, max_points)

Example

Full gridpp installation from source

1) Install dependencies

On Ubuntu Bionic, these can be installed like this:

sudo apt-get update
sudo apt-get install libboost-all-dev
sudo apt-get install libgsl0-dev libblas-dev
sudo apt-get install netcdf-bin libnetcdf-dev
sudo apt-get install libarmadillo6 libarmadillo-dev
sudo apt install swig cmake

Note that Ubuntu Xenial only has Armadillo 6.5 in its apt repository. In that case you need to install Armadillo 6.6 or later manually.

2) Download source code

Either download the source code from the latest release, unzip the file and navigate into the extracted folder; or clone the repo from github.

3) Set up cmake installation

mkdir build
cd build
cmake ..

4) Install the C++ library

sudo make install

This will install the library in /usr/local/lib/libgridpp.so and the gridpp command-line client in /usr/local/bin/gridpp. To specify a custom installation path, use the following in step 3:

cmake .. -DCMAKE_INSTALL_PREFIX=<custom path>

5) Install the python bindings

make install-python-user

This installs the python bindings in ~/local/lib/python3.6/site-packages/gridpp.py. To install the python bindings system-wide, use sudo make install-python instead.

If you only want to build the package, and want to install it in a custom location instead, run:

make build-python

and copy extras/SWIG/python/gridpp.py and extras/SWIG/python/_gridpp.so to the desired location.

6) Install the R bindings

make build-r

Currently, the R package is not installed centrally, but instead is placed in extras/SWIG/R/gridpp.R in the build directory.

gridpp client installation from source

  1. Extra dependencies The gridpp commando-line client requires the following extra dependencies:

On Ubuntu Bionic, these can be installed like this:

sudo apt-get install libgsl0-dev libblas-dev
sudo apt-get install netcdf-bin libnetcdf-dev
  1. Install the client
make gridpp-client

Copyright and license

Copyright © 2014-2024 Norwegian Meteorological Institute. Gridpp is licensed under the GNU LEsser General Public License (LGPL). See LICENSE file.

gridpp's People

Contributors

cskarby avatar e-benoit avatar jmnipen avatar joewkr avatar khintz avatar knedlsepp avatar lonnekeb avatar mpejcoch avatar tnipen avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

gridpp's Issues

Crippled Travis setup

Current version of Travis CI is configured in such a way that just leaves an impression that gridpp is being tested over different versions of Ubuntu but in fact it always uses a default container.

Xenial isn't supported by Travis yet (travis-ci/travis-ci#5821) and selection of environment by the DIST variable seems to have no effect; thus, according to 'Worker information' from one of the last builds, a trusty container is used for whatever DIST was set:

instance: testing-gce-9d0b1023-0d41-4555-9d32-1c250c9669fa:travis-ci-garnet-trusty-1503972833 (via amqp) #DIST=precise
instance: testing-gce-e0f22bcf-9085-4ef6-a9f7-bd905dbd1fbb:travis-ci-garnet-trusty-1503972833 (via amqp) #DIST=trusty
instance: testing-gce-72317033-7f2d-4e9e-9ab0-2e3d7f15729e:travis-ci-garnet-trusty-1503972833 (via amqp) #DIST=xenial

Add -d upscale

This "downscaler" could be used to upscale a high-resolution field to a low-resolution one. It would use a nearest neighbour tree going from output to input and take the average of all high-res points that belonged to a given low-res point.

GRIB support

Investigate if conversion to NetCDF and then back to GRIB is a possibility.

Handle missing leadtimes in gridpp_train

If a leadtime in the observation files is always missing (such as precipitation for leadtime 0), then an alternative leadtime should be used when creating the calibration (such as leadtime 24) after appropriate time adjustments have been made.

Memory error on out of bounds time-slice access

File::getField() gives a memory error when a field (which exists in the file) is access with an out of bounds time index on the first access.

The following causes a memory error, instead of aborting with an error message:
FileFake f0(Options("nLat=3 nLon=3 nEns=1 nTime=3"));
f0.getField(Variable::T, 4);

However, the following works fine:
FileFake f0(Options("nLat=3 nLon=3 nEns=1 nTime=3"));
f0.getField(Variable::T, 0);
f0.getField(Variable::T, 4);

Use fast nearest neighbour also when getting parameters

Implement Yurii's fast nearest neighbour approach in ParameterFile::getNearestNeighbour(), since this uses a slow brute force method. This results in slow performance when the parameter grid is different than the forecast grid.

Add --useGeopotential flag

Allow altitude to be calculated from the geopotential. This will be more appropriate when reading surface pressure in AROME for example, where waves in the field are seen.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.