airfuse

AirFuse provides several data fusion techniques designed to work with AirNow, PurpleAir, NOAA's Air Quality Forecast and NASA's Composition Forecast.

Overview

The standard driver applies Nearest Neighbor Averaging, Voronoi Neighbor Averaging (VNA), extended VNA (eVNA), and additive VNA (aVNA). eVNA corrects the model surface multiplying the ratio of obs:mod. aVNA is like eVNA, except it corrects teh model surface by subtracting the bias. For both eVNA and aVNA, the ratio or bias is interpolated from Voronoi neighbors using inverse distance weights.

This is currently a research product and is provided as-is with no warranty expressed or implied. Users should be cautious.

AirFuse Examples

from airfuse.drivers import fuse

date = '2023-08-24T18Z'
pmpaths = fuse(
    obssource='airnow', species='pm25', startdate=date, model='naqfc'
)
o3paths = fuse(
    obssource='airnow', species='o3', startdate=date, model='naqfc'
)

Install

Using pip

airfuse currently requires the nna_methods package, which is another github repository. So, installing requires two calls to pip.

pip install git+https://github.com/barronh/nna_methods.git
pip install git+https://github.com/barronh/airfuse.git

From Downloaded Source

airfuse can also be installed by downloading the source code.

wget https://github.com/barronh/airfuse/archive/refs/heads/main.zip
unzip main
cd airfuse-main
pip install -r requirements.txt
pip install .

Feedback

If you have feedback about airfuse, please open an issue.

System fails when NCEP OpenDAP server is not working.

Summary

System fails when NCEP OpenDAP server is not working -- even if it is using https servers that are working for its data.

Background

AirFuse uses NCEI OpenDAP as the defining grid coordinate. This is the result of potentially receiving alternate coordinates when they are derived from the grib2 meta-data. For example, cfgrib can be used by xarray to open grib2 files. When that happens, it constructs the x and y coordinates from the file. In that case, the resulting x and y are numerically different than those from the OpenDAP server.

The differences in coordinates are very small. I had selected the NCEI because it is the only file that explicitly contains coordinates and it is distributed by NOAA. The coordinates are mostly the same. Distortion is largest at the corners.

j	i	Source	LONGITUDE	LATITUDE
0	0	OpenDAP	-133.45901408	12.19000009
0	0	GRIB2	-133.45900000	12.19000000
0	1472	OpenDAP	-65.09702233	14.33589514
0	1472	GRIB2	-65.09700984	14.33588836
1024	1472	OpenDAP	-49.39552159	57.28784477
1024	1472	GRIB2	-49.39550324	57.28783924
1024	0	OpenDAP	-152.85301655	54.53265860
1024	0	GRIB2	-152.85299695	54.53266156

Having different coordinates depending on how the data was acquired is more of an annoyance than a real issue. The differences are very small, but cause potential inconsistencies. For example, f = xr.concat(paths, dim='time') will lead to missing data if paths include data from both systems. Which ever is first will have data -- the second will not.

To avoid this type of issue, I have setup a separate grid file that is downloaded from NCEI OpenDAP. It is reused, so it only needs to be downloaded once. However, if the OpenDAP server is not working (e.g., 2024-03-18), the whole system will fail because it cannot get the coordinates.

Potential Resolutions

Wait for the OpenDAP to provide coordinates and archive them (currently already happens)
Disable this "feature" and instead deal with it when coordinates vary.
Create a "hard-coded" off-line grid definition NCEP 227
Defer to GRIB coordinates as the definitive source

barronh / airfuse Goto Github PK

airfuse's Introduction

airfuse

Overview

AirFuse Examples

Install

Using pip

From Downloaded Source

Feedback

airfuse's People

Contributors

Stargazers

Watchers

Forkers

airfuse's Issues

Summary

Background

Potential Resolutions

Recommend Projects

Recommend Topics

Recommend Org