kylebarron / landsat-mosaic-latest Goto Github PK

Auto-updating global Landsat 8 mosaic of Cloud-Optimized GeoTIFFs from SNS notifications

License: MIT License

Python 90.57% Dockerfile 2.82% Makefile 2.07% Shell 4.54%

landsat landsat-8 landsat-data serverless aerial-imagery lambda satellite-imagery aws-lambda cloud-optimized-geotiff python

landsat-mosaic-latest's Issues

Outline

SNS message comes in
~~Load metadata from DynamoDB to know quadkey zoom, or potentially provide it as an environment variable~~. Don't need to know this, because it's specified in the pre-generated pathrow-quadkey index.

Find new asset(s)

with open('example.json') as f:
    body = json.load(f)

message = json.loads(body['Message'])

records = message['Records']
record = records[0]

For each asset:
- Retrieve metadata. The metadata is in a *_MTL.json file in the same directory as the index.html referenced in the SNS message. So you'll have to do an S3 List, and then retrieve the JSON file. The structure of the JSON file is listed in #4.
- If cloud cover is above designated level, return
- Find overlapping quadkeys from path-row to quadkey index
- For each overlapping mercator tile:
  - Find existing dynamoDB payload
  - Add new image
  - Find out which of the existing images can be removed with the addition of the new image, while still covering the entire tile. Remove these.
  - Write back to DynamoDB

To do

Create path-row to quadkey index

Script to create overview tiles

Creating overview tiles might be better placed in a separate repository...

But it would be cool to have a script to ~weekly or biweekly update tiles on S3 for zooms 0-7.

Use stale while revalidate to allow long caching

I think if you put a month long cache control with a day long swr, cloudflare would check each tile once a day?

Script to create initial recent mosaic

Maybe this could be an update to landsat-cogeo-mosaic...

Take all images in the last x amount of time, or even since 2013. For each path-row, find the most recent cloudless scene. Then you have a collection of cloudless scenes to make a mosaic from.

Possible updating bug

http://localhost:8000/#7.08/38.168/-111.423

The mosaic was initially created with landsat-cogeo-mosaic, which seems to be working fine, so this is most likely a bug in the updating.

Call from SNS in serverless.yml

https://serverless.com/framework/docs/providers/aws/events/sns#using-a-pre-existing-topic

functions:
  dispatcher:
    handler: dispatcher.dispatch
    events:
      - sns: arn:xxx

Update two tables, a latest and a latest cloudless

Add metadata record when creating table

There should be a metadata record with quadkey -1. That never gets created

SNS Body Example

https://lists.osgeo.org/pipermail/landsat-pds/2016-January/000124.html

Optimize index further

At relatively high latitudes, you have overlapping scenes such that you don't need all of them to cover a tile. So you don't need to include all of them.

Use only RT or only T1 scenes

https://www.usgs.gov/land-resources/nli/landsat/landsat-collection-1?qt-science_support_page_related_con=1#qt-science_support_page_related_con

Support bounding box option

I.e. if someone wants to use this to continually update landsat imagery only for a given bounding box

JSON Scene metadata

{
    "L1_METADATA_FILE": {
        "IMAGE_ATTRIBUTES": {
            "EARTH_SUN_DISTANCE": 0.9916119,
            "SUN_AZIMUTH": 134.09453276,
            "TIRS_SSM_POSITION_STATUS": "ESTIMATED",
            "GROUND_CONTROL_POINTS_MODEL": 254,
            "GEOMETRIC_RMSE_VERIFY": 3.891,
            "TIRS_STRAY_LIGHT_CORRECTION_SOURCE": "TIRS",
            "GEOMETRIC_RMSE_MODEL": 7.39,
            "CLOUD_COVER": 1.28,
            "IMAGE_QUALITY_TIRS": 9,
            "GROUND_CONTROL_POINTS_VERSION": 4,
            "TRUNCATION_OLI": "UPPER",
            "SUN_ELEVATION": 51.69540297,
            "CLOUD_COVER_LAND": 0.81,
            "GEOMETRIC_RMSE_MODEL_X": 5.676,
            "GEOMETRIC_RMSE_MODEL_Y": 4.732,
            "IMAGE_QUALITY_OLI": 9,
            "GROUND_CONTROL_POINTS_VERIFY": 54,
            "SATURATION_BAND_8": "N",
            "SATURATION_BAND_9": "N",
            "SATURATION_BAND_2": "N",
            "SATURATION_BAND_3": "N",
            "SATURATION_BAND_1": "N",
            "SATURATION_BAND_6": "Y",
            "SATURATION_BAND_7": "Y",
            "SATURATION_BAND_4": "N",
            "SATURATION_BAND_5": "N",
            "ROLL_ANGLE": -0.001,
            "TIRS_SSM_MODEL": "FINAL"
        },
        "TIRS_THERMAL_CONSTANTS": {
            "K1_CONSTANT_BAND_11": 480.8883,
            "K2_CONSTANT_BAND_11": 1201.1442,
            "K1_CONSTANT_BAND_10": 774.8853,
            "K2_CONSTANT_BAND_10": 1321.0789
        },
        "RADIOMETRIC_RESCALING": {
            "RADIANCE_MULT_BAND_7": 0.00052118,
            "RADIANCE_MULT_BAND_6": 0.0015463,
            "RADIANCE_MULT_BAND_5": 0.0062177,
            "RADIANCE_MULT_BAND_4": 0.010161,
            "RADIANCE_MULT_BAND_3": 0.012049,
            "RADIANCE_MULT_BAND_2": 0.013076,
            "RADIANCE_MULT_BAND_1": 0.012769,
            "RADIANCE_MULT_BAND_9": 0.00243,
            "RADIANCE_MULT_BAND_8": 0.011499,
            "RADIANCE_ADD_BAND_9": -12.15014,
            "RADIANCE_ADD_BAND_8": -57.49449,
            "RADIANCE_MULT_BAND_11": 0.0003342,
            "RADIANCE_MULT_BAND_10": 0.0003342,
            "REFLECTANCE_ADD_BAND_9": -0.1,
            "REFLECTANCE_ADD_BAND_8": -0.1,
            "RADIANCE_ADD_BAND_1": -63.84536,
            "REFLECTANCE_ADD_BAND_6": -0.1,
            "RADIANCE_ADD_BAND_3": -60.24566,
            "RADIANCE_ADD_BAND_2": -65.37843,
            "RADIANCE_ADD_BAND_5": -31.08862,
            "RADIANCE_ADD_BAND_4": -50.80254,
            "RADIANCE_ADD_BAND_7": -2.60591,
            "RADIANCE_ADD_BAND_6": -7.73146,
            "REFLECTANCE_MULT_BAND_9": 2e-05,
            "REFLECTANCE_MULT_BAND_8": 2e-05,
            "REFLECTANCE_MULT_BAND_1": 2e-05,
            "REFLECTANCE_MULT_BAND_3": 2e-05,
            "REFLECTANCE_MULT_BAND_2": 2e-05,
            "REFLECTANCE_MULT_BAND_5": 2e-05,
            "REFLECTANCE_MULT_BAND_4": 2e-05,
            "REFLECTANCE_MULT_BAND_7": 2e-05,
            "REFLECTANCE_MULT_BAND_6": 2e-05,
            "REFLECTANCE_ADD_BAND_7": -0.1,
            "REFLECTANCE_ADD_BAND_5": -0.1,
            "REFLECTANCE_ADD_BAND_4": -0.1,
            "RADIANCE_ADD_BAND_11": 0.1,
            "RADIANCE_ADD_BAND_10": 0.1,
            "REFLECTANCE_ADD_BAND_3": -0.1,
            "REFLECTANCE_ADD_BAND_2": -0.1,
            "REFLECTANCE_ADD_BAND_1": -0.1
        },
        "PRODUCT_METADATA": {
            "ANGLE_COEFFICIENT_FILE_NAME": "LC08_L1TP_139045_20170304_20170316_01_T1_ANG.txt",
            "BPF_NAME_OLI": "LO8BPF20170304042138_20170304055856.01",
            "BPF_NAME_TIRS": "LT8BPF20170225090252_20170313050703.01",
            "COLLECTION_CATEGORY": "T1",
            "CORNER_LL_LAT_PRODUCT": 20.61004,
            "CORNER_LL_LON_PRODUCT": 85.8695,
            "CORNER_LL_PROJECTION_X_PRODUCT": 382200.0,
            "CORNER_LL_PROJECTION_Y_PRODUCT": 2279400.0,
            "CORNER_LR_LAT_PRODUCT": 20.61049,
            "CORNER_LR_LON_PRODUCT": 88.06046,
            "CORNER_LR_PROJECTION_X_PRODUCT": 610500.0,
            "CORNER_LR_PROJECTION_Y_PRODUCT": 2279400.0,
            "CORNER_UL_LAT_PRODUCT": 22.71567,
            "CORNER_UL_LON_PRODUCT": 85.85297,
            "CORNER_UL_PROJECTION_X_PRODUCT": 382200.0,
            "CORNER_UL_PROJECTION_Y_PRODUCT": 2512500.0,
            "CORNER_UR_LAT_PRODUCT": 22.71616,
            "CORNER_UR_LON_PRODUCT": 88.07596,
            "CORNER_UR_PROJECTION_X_PRODUCT": 610500.0,
            "CORNER_UR_PROJECTION_Y_PRODUCT": 2512500.0,
            "CPF_NAME": "LC08CPF_20170101_20170331_01.02",
            "DATA_TYPE": "L1TP",
            "DATE_ACQUIRED": "2017-03-04",
            "ELEVATION_SOURCE": "GLS2000",
            "FILE_NAME_BAND_1": "LC08_L1TP_139045_20170304_20170316_01_T1_B1.TIF",
            "FILE_NAME_BAND_2": "LC08_L1TP_139045_20170304_20170316_01_T1_B2.TIF",
            "FILE_NAME_BAND_3": "LC08_L1TP_139045_20170304_20170316_01_T1_B3.TIF",
            "FILE_NAME_BAND_4": "LC08_L1TP_139045_20170304_20170316_01_T1_B4.TIF",
            "FILE_NAME_BAND_5": "LC08_L1TP_139045_20170304_20170316_01_T1_B5.TIF",
            "FILE_NAME_BAND_6": "LC08_L1TP_139045_20170304_20170316_01_T1_B6.TIF",
            "FILE_NAME_BAND_7": "LC08_L1TP_139045_20170304_20170316_01_T1_B7.TIF",
            "FILE_NAME_BAND_8": "LC08_L1TP_139045_20170304_20170316_01_T1_B8.TIF",
            "FILE_NAME_BAND_9": "LC08_L1TP_139045_20170304_20170316_01_T1_B9.TIF",
            "FILE_NAME_BAND_10": "LC08_L1TP_139045_20170304_20170316_01_T1_B10.TIF",
            "FILE_NAME_BAND_11": "LC08_L1TP_139045_20170304_20170316_01_T1_B11.TIF",
            "FILE_NAME_BAND_QUALITY": "LC08_L1TP_139045_20170304_20170316_01_T1_BQA.TIF",
            "METADATA_FILE_NAME": "LC08_L1TP_139045_20170304_20170316_01_T1_MTL.txt",
            "NADIR_OFFNADIR": "NADIR",
            "OUTPUT_FORMAT": "GEOTIFF",
            "PANCHROMATIC_LINES": 15541,
            "PANCHROMATIC_SAMPLES": 15221,
            "REFLECTIVE_LINES": 7771,
            "REFLECTIVE_SAMPLES": 7611,
            "RLUT_FILE_NAME": "LC08RLUT_20150303_20431231_01_12.h5",
            "SCENE_CENTER_TIME": "04:37:24.6864810Z",
            "SENSOR_ID": "OLI_TIRS",
            "SPACECRAFT_ID": "LANDSAT_8",
            "TARGET_WRS_PATH": 139,
            "TARGET_WRS_ROW": 45,
            "THERMAL_LINES": 7771,
            "THERMAL_SAMPLES": 7611,
            "WRS_PATH": 139,
            "WRS_ROW": 45
        },
        "PROJECTION_PARAMETERS": {
            "UTM_ZONE": 45,
            "GRID_CELL_SIZE_REFLECTIVE": 30.0,
            "MAP_PROJECTION": "UTM",
            "ORIENTATION": "NORTH_UP",
            "ELLIPSOID": "WGS84",
            "GRID_CELL_SIZE_THERMAL": 30.0,
            "DATUM": "WGS84",
            "GRID_CELL_SIZE_PANCHROMATIC": 15.0,
            "RESAMPLING_OPTION": "CUBIC_CONVOLUTION"
        },
        "METADATA_FILE_INFO": {
            "ORIGIN": "Image courtesy of the U.S. Geological Survey",
            "LANDSAT_PRODUCT_ID": "LC08_L1TP_139045_20170304_20170316_01_T1",
            "LANDSAT_SCENE_ID": "LC81390452017063LGN00",
            "PROCESSING_SOFTWARE_VERSION": "LPGS_2.7.0",
            "FILE_DATE": "2017-03-16T19:32:39Z",
            "COLLECTION_NUMBER": 1,
            "STATION_ID": "LGN",
            "REQUEST_ID": "0701703168044_00038"
        },
        "MIN_MAX_PIXEL_VALUE": {
            "QUANTIZE_CAL_MAX_BAND_5": 65535,
            "QUANTIZE_CAL_MAX_BAND_4": 65535,
            "QUANTIZE_CAL_MAX_BAND_7": 65535,
            "QUANTIZE_CAL_MAX_BAND_6": 65535,
            "QUANTIZE_CAL_MAX_BAND_1": 65535,
            "QUANTIZE_CAL_MAX_BAND_3": 65535,
            "QUANTIZE_CAL_MAX_BAND_2": 65535,
            "QUANTIZE_CAL_MAX_BAND_9": 65535,
            "QUANTIZE_CAL_MAX_BAND_8": 65535,
            "QUANTIZE_CAL_MIN_BAND_9": 1,
            "QUANTIZE_CAL_MIN_BAND_8": 1,
            "QUANTIZE_CAL_MIN_BAND_7": 1,
            "QUANTIZE_CAL_MIN_BAND_6": 1,
            "QUANTIZE_CAL_MIN_BAND_5": 1,
            "QUANTIZE_CAL_MIN_BAND_4": 1,
            "QUANTIZE_CAL_MIN_BAND_3": 1,
            "QUANTIZE_CAL_MIN_BAND_2": 1,
            "QUANTIZE_CAL_MIN_BAND_1": 1,
            "QUANTIZE_CAL_MIN_BAND_11": 1,
            "QUANTIZE_CAL_MIN_BAND_10": 1,
            "QUANTIZE_CAL_MAX_BAND_11": 65535,
            "QUANTIZE_CAL_MAX_BAND_10": 65535
        },
        "MIN_MAX_RADIANCE": {
            "RADIANCE_MINIMUM_BAND_6": -7.72991,
            "RADIANCE_MINIMUM_BAND_7": -2.60539,
            "RADIANCE_MINIMUM_BAND_4": -50.79238,
            "RADIANCE_MINIMUM_BAND_5": -31.0824,
            "RADIANCE_MINIMUM_BAND_2": -65.36536,
            "RADIANCE_MINIMUM_BAND_3": -60.23362,
            "RADIANCE_MINIMUM_BAND_1": -63.83259,
            "RADIANCE_MINIMUM_BAND_8": -57.48299,
            "RADIANCE_MINIMUM_BAND_9": -12.14771,
            "RADIANCE_MINIMUM_BAND_10": 0.10033,
            "RADIANCE_MINIMUM_BAND_11": 0.10033,
            "RADIANCE_MAXIMUM_BAND_10": 22.0018,
            "RADIANCE_MAXIMUM_BAND_11": 22.0018,
            "RADIANCE_MAXIMUM_BAND_8": 696.08582,
            "RADIANCE_MAXIMUM_BAND_9": 147.10172,
            "RADIANCE_MAXIMUM_BAND_1": 772.97577,
            "RADIANCE_MAXIMUM_BAND_2": 791.53662,
            "RADIANCE_MAXIMUM_BAND_3": 729.39423,
            "RADIANCE_MAXIMUM_BAND_4": 615.06635,
            "RADIANCE_MAXIMUM_BAND_5": 376.38992,
            "RADIANCE_MAXIMUM_BAND_6": 93.60475,
            "RADIANCE_MAXIMUM_BAND_7": 31.54981
        },
        "MIN_MAX_REFLECTANCE": {
            "REFLECTANCE_MAXIMUM_BAND_8": 1.2107,
            "REFLECTANCE_MINIMUM_BAND_8": -0.09998,
            "REFLECTANCE_MAXIMUM_BAND_9": 1.2107,
            "REFLECTANCE_MINIMUM_BAND_9": -0.09998,
            "REFLECTANCE_MINIMUM_BAND_4": -0.09998,
            "REFLECTANCE_MINIMUM_BAND_5": -0.09998,
            "REFLECTANCE_MINIMUM_BAND_6": -0.09998,
            "REFLECTANCE_MINIMUM_BAND_7": -0.09998,
            "REFLECTANCE_MINIMUM_BAND_1": -0.09998,
            "REFLECTANCE_MINIMUM_BAND_2": -0.09998,
            "REFLECTANCE_MINIMUM_BAND_3": -0.09998,
            "REFLECTANCE_MAXIMUM_BAND_6": 1.2107,
            "REFLECTANCE_MAXIMUM_BAND_7": 1.2107,
            "REFLECTANCE_MAXIMUM_BAND_4": 1.2107,
            "REFLECTANCE_MAXIMUM_BAND_5": 1.2107,
            "REFLECTANCE_MAXIMUM_BAND_2": 1.2107,
            "REFLECTANCE_MAXIMUM_BAND_3": 1.2107,
            "REFLECTANCE_MAXIMUM_BAND_1": 1.2107
        }
    }
}

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.