Giter Club home page Giter Club logo

geoscienceaustralia / dea-coastlines Goto Github PK

View Code? Open in Web Editor NEW
48.0 9.0 16.0 35 MB

Extracting tidally-constrained annual shorelines and robust rates of coastal change from freely available Earth observation data at continental scale

Home Page: https://knowledge.dea.ga.gov.au/data/product/dea-coastlines/?tab=overview

License: Apache License 2.0

Jupyter Notebook 9.38% Python 3.01% Shell 0.02% QML 0.94% Dockerfile 0.02% PLpgSQL 86.63%
digitalearthaustralia opendatacube coastal-change remote-sensing erosion coastal-dynamics dea coastlines satellite-imagery python

dea-coastlines's Introduction

๐ŸŒ Note: This branch contains the Australia-specific implementation of DEA Coastlines. For a more flexible implementation adapted to globally available USGS Collection 2 Level 2 Landsat data, we recommend adapting the Digital Earth Africa Coastlines implementation instead.

Digital Earth Australia Coastlines

Digital Earth Australia Coastlines

DOI License codecov example workflow

License: The code in this repository is licensed under the Apache License, Version 2.0. Digital Earth Australia data is licensed under the Creative Commons by Attribution 4.0 license.

Contact: For assistance with any of the Python code or Jupyter Notebooks in this repository, please post a Github issue. For questions or more information about this workflow, email [email protected].

To cite:

Bishop-Taylor, R., Nanson, R., Sagar, S., Lymburner, L. (2021). Mapping Australia's dynamic coastline at mean sea level using three decades of Landsat imagery. Remote Sensing of Environment, 267, 112734. Available: https://doi.org/10.1016/j.rse.2021.112734

Nanson, R., Bishop-Taylor, R., Sagar, S., Lymburner, L., (2022). Geomorphic insights into Australia's coastal change using a national dataset derived from the multi-decadal Landsat archive. Estuarine, Coastal and Shelf Science, 265, p.107712. Available: https://doi.org/10.1016/j.ecss.2021.107712

Bishop-Taylor, R., Sagar, S., Lymburner, L., Alam, I., Sixsmith, J. (2019). Sub-pixel waterline extraction: characterising accuracy and sensitivity to indices and spectra. Remote Sensing, 11 (24):2984. Available: https://doi.org/10.3390/rs11242984


Digital Earth Australia Coastlines is a continental dataset that includes annual shorelines and rates of coastal change along the entire Australian coastline from 1988 to the present.

The product combines satellite data from Geoscience Australia's Digital Earth Australia program with tidal modelling to map the typical location of the coastline at mean sea level for each year. The product enables trends of coastal erosion and growth to be examined annually at both a local and continental scale, and for patterns of coastal change to be mapped historically and updated regularly as data continues to be acquired. This allows current rates of coastal change to be compared with that observed in previous years or decades.

The ability to map shoreline positions for each year provides valuable insights into whether changes to our coastline are the result of particular events or actions, or a process of more gradual change over time. This information can enable scientists, managers and policy makers to assess impacts from the range of drivers impacting our coastlines and potentially assist planning and forecasting for future scenarios.

Applications

  • Monitoring and mapping rates of coastal erosion along the Australian coastline
  • Prioritise and evaluate the impacts of local and regional coastal management based on historical coastline change
  • Modelling how coastlines respond to drivers of change, including extreme weather events, sea level rise or human development
  • Supporting geomorphological studies of how and why coastlines have changed across time

Table of contents


Repository code

The code in this repository is built on the Digital Earth Australia implementation of the Open Data Cube software for accessing, managing, and analyzing large quantities of Earth observation (EO) data. The code currently runs on the Digital Earth Australia Sandbox infrastructure.

Getting started

Clone the dea-coastlines repository and checkout the develop branch:

git clone https://github.com/GeoscienceAustralia/dea-coastlines.git
git checkout --track origin/develop
FES2014 tidal model

DEA Coastlines uses the FES2014 tidal model to account for the influence of tide on shoreline positions. To install this tidal model, follow the Setting up tidal models for DEA Coastlines guide on the Wiki.

Python modules

Code in this repository is included in the coastlines Python package which contains three main modules. These are intended to be run in the following order:

  1. coastlines.raster: This module conducts raster generation for DEA Coastlines. This analysis is processed on individual study area tiles to minimise peak memory usage.

    • Load stack of all available Landsat 5, 7 and 8 satellite imagery for a location using ODC Virtual Products
    • Convert each satellite image into a remote sensing water index (e.g. MNDWI)
    • For each satellite image, model ocean tides into a tidal modelling grid based on exact time of image acquisition
    • Interpolate tide heights into spatial extent of image stack
    • Mask out high and low tide pixels by removing all observations acquired outside of 50 percent of the observed tidal range centered over mean sea level
    • Combine tidally-masked data into annual median composites representing the most representative position of the shoreline at approximately mean sea level each year
  2. coastlines.vector: This module conducts vector subpixel coastline extraction and rates of change statistics calculation. This analysis is processed on individual study area tiles to minimise peak memory usage.

    • Apply morphological extraction algorithms to mask annual median composite rasters to a valid coastal region
    • Extract shoreline vectors using subpixel waterline extraction (Bishop-Taylor et al. 2019b)
    • Compute rates of coastal change at every 30 m along the coastline using linear regression
  3. coastlines.continental: This module combines tiled layers into seamless continental-scale vector files:

    • Combines multiple output shoreline and rates of change statistics point vectors into single continental datasets
    • Aggregates this data to produce moving window coastal change hotspot datasets that summarise coastal change at regional and continental scale.

Jupyter notebooks

An interactive walk-through of each step of the tiled raster and vector DEA Coastlines workflow and the continental layer generation is provided in the following Jupyter Notebooks. These notebooks can be run on the DEA Sandbox to assist in prototyping or troubleshooting:

Running a DEA Coastlines analysis using the command-line interface (CLI)

These three modules have a command-line interface that can be used to automate each stage of the analysis. An example of using these tools is provided in the following Jupyter Notebook:

For help using these command line tools, run:

python -m coastlines.raster --help
python -m coastlines.vector --help
python -m coastlines.continental --help

Analysis outputs

Files generated by DEA Coastlines are exported to the data directory.

Temporary raster and vector outputs produced by coastlines.raster and coastlines.vector for each study area grid cell are exported to:

data/interim/raster/{unique_analysis_name}/{unique_analysis_name}_{study_area_name}
data/interim/vector/{unique_analysis_name}/{unique_analysis_name}_{study_area_name}

Once all study area grid cells have been processed, these are combined into a continental-scale output GeoPackage vector file and zipped ESRI Shapefiles using coastlines.continental. These final outputs are exported to:

data/processed/{unique_analysis_name}/coastlines_{continental_version}.gpkg
data/processed/{unique_analysis_name}/coastlines_{continental_version}.shp.zip

Data access

Data download

To download DEA Coastlines data for the entire Australian coastline, visit the "Access" tab of the Geoscience Australia DEA Coastlines product description and follow the instructions under "Access notes". Data is available in two formats:

  • GeoPackage (recommended): suitable for QGIS; includes built-in symbology for easier interpretation
  • ESRI Shapefiles: suitable for ArcMap and QGIS

Interactive map

To explore DEA Coastlines on an interactive map, visit the Digital Earth Australia Maps platform.

Zooming to annual rates of change and plotting chart in DEA Maps

Loading DEA Coastlines data from the Web Feature Service (WFS) using Python

DEA Coastlines data can be loaded directly in a Python script or Jupyter Notebook using the DEA Coastlines Web Feature Service (WFS) and geopandas:

import geopandas as gpd

# Specify bounding box
ymax, xmin = -33.65, 115.28
ymin, xmax = -33.66, 115.30

# Set up WFS requests for annual shorelines & rates of change points
deacl_annualshorelines_wfs = f'https://geoserver.dea.ga.gov.au/geoserver/wfs?' \
                       f'service=WFS&version=1.1.0&request=GetFeature' \
                       f'&typeName=dea:shorelines_annual&maxFeatures=1000' \
                       f'&bbox={ymin},{xmin},{ymax},{xmax},' \
                       f'urn:ogc:def:crs:EPSG:4326'
deacl_ratesofchange_wfs = f'https://geoserver.dea.ga.gov.au/geoserver/wfs?' \
                       f'service=WFS&version=1.1.0&request=GetFeature' \
                       f'&typeName=dea:rates_of_change&maxFeatures=1000' \
                       f'&bbox={ymin},{xmin},{ymax},{xmax},' \
                       f'urn:ogc:def:crs:EPSG:4326'

# Load DEA Coastlines data from WFS using geopandas
deacl_annualshorelines_gdf = gpd.read_file(deacl_annualshorelines_wfs)
deacl_ratesofchange_gdf = gpd.read_file(deacl_ratesofchange_wfs)

# Ensure CRSs are set correctly
deacl_annualshorelines_gdf.crs = 'EPSG:3577'
deacl_ratesofchange_gdf.crs = 'EPSG:3577'

# Optional: Keep only rates of change points with "good" certainty 
# (i.e. no poor quality flags)
deacl_ratesofchange_gdf = deacl_ratesofchange_gdf.query("certainty == 'good'")

Loading DEA Coastlines data from the Web Feature Service (WFS) using R

DEA Coastlines data can be loaded directly into R using the DEA Coastlines Web Feature Service (WFS) and sf:

library(magrittr)
library(glue)
library(sf)

# Specify bounding box
xmin = 115.28
xmax = 115.30
ymin = -33.66
ymax = -33.65

# Read in DEA Coastlines annual shoreline data, using `glue` to insert our bounding 
# box into the string, and `sf` to  load the spatial data from the Web Feature Service 
# and set the Coordinate Reference System to Australian Albers (EPSG:3577)
deacl_annualshorelines = "https://geoserver.dea.ga.gov.au/geoserver/wfs?service=WFS&version=1.1.0&request=GetFeature&typeName=dea:shorelines_annual&maxFeatures=1000&bbox={ymin},{xmin},{ymax},{xmax},urn:ogc:def:crs:EPSG:4326" %>% 
  glue::glue() %>%
  sf::read_sf() %>% 
  sf::st_set_crs(3577)

# Read in DEA Coastlines rates of change points
deacl_ratesofchange = "https://geoserver.dea.ga.gov.au/geoserver/wfs?service=WFS&version=1.1.0&request=GetFeature&typeName=dea:rates_of_change&maxFeatures=1000&bbox={ymin},{xmin},{ymax},{xmax},urn:ogc:def:crs:EPSG:4326" %>% 
  glue::glue() %>%
  sf::read_sf() %>% 
  sf::st_set_crs(3577)

Jupyter Notebook

An Introduction to DEA Coastlines Jupyter notebook providing additional useful tools for loading and analysing DEA Coastlines data can be found on the DEA Notebooks repository. This notebook is available on the interactive DEA Sandbox learning and analysis environment for easy access via a web browser.


Credits

Tidal modelling is provided by the FES2014 global tidal model, implemented using the pyTMD Python package. FES2014 was produced by NOVELTIS, LEGOS, CLS Space Oceanography Division and CNES. It is distributed by AVISO, with support from CNES (http://www.aviso.altimetry.fr/).

References

Bishop-Taylor, R., Nanson, R., Sagar, S., Lymburner, L. (2021). Mapping Australia's dynamic coastline at mean sea level using three decades of Landsat imagery. Remote Sensing of Environment, 267, 112734. Available: https://doi.org/10.1016/j.rse.2021.112734

Bishop-Taylor, R., Sagar, S., Lymburner, L., & Beaman, R. J. (2019a). Between the tides: Modelling the elevation of Australia's exposed intertidal zone at continental scale. Estuarine, Coastal and Shelf Science, 223, 115-128. Available: https://doi.org/10.1016/j.ecss.2019.03.006

Bishop-Taylor, R., Sagar, S., Lymburner, L., Alam, I., & Sixsmith, J. (2019b). Sub-pixel waterline extraction: Characterising accuracy and sensitivity to indices and spectra. Remote Sensing, 11(24), 2984. Available: https://doi.org/10.3390/rs11242984

Nanson, R., Bishop-Taylor, R., Sagar, S., Lymburner, L., (2022). Geomorphic insights into Australia's coastal change using a national dataset derived from the multi-decadal Landsat archive. Estuarine, Coastal and Shelf Science, 265, p.107712. Available: https://doi.org/10.1016/j.ecss.2021.107712

dea-coastlines's People

Contributors

pindge avatar robbibt avatar vnewey avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

dea-coastlines's Issues

Improvements to DEA CoastLines utilities

Via @GeoRach:

  • Export summary table for each polygon in the shp input (incl. mean, mode 1 (and 2, 3 if several?), SD, min, max)
  • Address this warning "More than 20 figures have been opened. Figures created through the pyplot interface (matplotlib.pyplot.figure) are retained until explicitly closed and may consume too much memory. "
  • Add second y-axis to indicate % of dataset as line plot of histogram (optional?)
  • For rate-of-change histogram plots also add n for each histogram bar (again, optional)
  • Have the "interactive profile selection" tool default zoom to the preceding "statistics points histogram analysis" extent, because that's a likely user workflow
  • To help novice users include both options in descriptive comments for "distance" vs "width", and None vs "*.shp" options
  • For interactive polygon and transect lines, export an .shp (line or poly) so the analysis location is recorded.
  • Support multiple polygons in a dataset by allowing the user to specify a field in the dataset, and loop through polygons and write to file
  • Rates of change are currently different in log vs linear histogram plots due to the non-log version dropping non-significant points rather than setting them to 0 m / year change. This should be updated so values are the same for both

image

Remove experimental breakpoints field

This was experimental, and didn't end up being an idea solution to the problem of identifying non-linear breakpoints in shoreline change time-series.

Add gapfill flag to coastline quality

This would allow users to determine whether a given coastline is based on exclusively data from that calendar year, or gapfilled with data from neighbouring years.

Add a maximum change envelope column to statistics file

The only change I'd suggest, to accomodate the impact-recovery type changes would be to have a value of the 'the coastline has moved by up to z metres at this location' where z is the range between maximum landward and maximum seaward. This should capture both monotonic and event-recovery type changes in a single metric and let users evaluate whether or not it's erosion, cyclical, step, gradual etc from the profile

Filter outliers before calculating change

Add Geoserver/WMS fields as part of generation process

Currently a few additional fields are required as part of the GeoServer WMS so that they can be rendered correctly via WMS and on Terria: conf_time (for statistics only) and abs_time (for statistics and summary layers).

In the future, it would be good to add these automatically during the statistics and summary layer generation step to save having to manually run them every time.

Missing integration test and codecov

setup docker-compose stack to perform integration test.
1. setup Dockerfile for dea-coastlines - to be done by @robbibt
2. setup docker-compose with dea-coastlines docker and postgres odc indexed database - to be done by @pindge
3. README update for test setup including how to run docker - to be done by @pindge
4. test output = expected - to be done by @pindge
5. setup codecov - to be done by @pindge

Questions need to be clarified

What odc products and minimum required datasets need to be indexed for dea-coastline test?

https://explorer.dev.dea.ga.gov.au/product/ga_ls8c_ard_3/regions/089083, https://explorer.dev.dea.ga.gov.au/product/ga_ls5t_ard_3/regions/089083,
https://explorer.dev.dea.ga.gov.au/product/ga_ls7e_ard_3/regions/089083

What kind of test output are we expecting? and how would we verify output is as expected

What command to run for docker

https://gist.github.com/robbibt/1939b55898d84dff0bd90266fbd222cb#file-coastlines-yaml-L186-L200
https://gist.github.com/robbibt/1939b55898d84dff0bd90266fbd222cb#file-coastlines-yaml-L264-L268

Move tidal modelling funcs to DEA Tools

This will make it easier to access these tools from one central location, rather than duplicating the code across both DEA Coastlines, DE Africa Coastlines and DEA Intertidal repos.

Needs FES2014 access in Sandbox first.

Address non-linear dynamics by filtering statistics points by angle variance

Currently, rates-of-change statistics are calculated by measuring the nearest distance to shorelines from each year in the time series. This works well for linear erosional/progradation dynamics, but less well in environments undergoing highly dynamic, non-linear change through time (e.g. sandbars, river mouths etc).

One way to address this would be to calculate the variance of angles to each nearest shoreline. In the case of linear erosion/progradation, all angles should be similar. However, more complex examples of change are likely to have a larger range of angles. This angle variance could then be used to filter or flag relevant statistics points as lower certainty.

Topological error in coastline generation

Coastline generation for certain Albers tiles fail with a shapely.errors.TopologicalErrordue to a potential invalid shapely.geometry.multilinestring.MultiLineString object.

This affects the following (incomplete) list of tiles:

  • 8003
Loading dea/20200713
  Loading requirement: dea-env/20200713
IllegalArgumentException: CGAlgorithmsDD::orientationIndex encountered NaN/Inf numbers
Traceback (most recent call last):
  File "/g/data/r78/DEACoastLines/deacoastlines_statistics.py", line 1522, in <module>
    main()
  File "/g/data/r78/DEACoastLines/deacoastlines_statistics.py", line 1507, in main
    output_path=f'output_data/{study_area}_{output_name}')
  File "/g/data/r78/DEACoastLines/deacoastlines_statistics.py", line 1229, in contour_certainty
    vector_class = gpd.clip(contours_gdf, vector_mask.loc[i].geometry)
  File "/g/data/v10/public/modules/dea-env/20200713/lib/python3.6/site-packages/geopandas/tools/clip.py", line 174, in clip
    line_gdf = _clip_line_poly(gdf[line_idx], poly)
  File "/g/data/v10/public/modules/dea-env/20200713/lib/python3.6/site-packages/geopandas/tools/clip.py", line 69, in _clip_line_poly
    clipped["geometry"] = gdf_sub.intersection(poly)
  File "/g/data/v10/public/modules/dea-env/20200713/lib/python3.6/site-packages/geopandas/base.py", line 562, in intersection
    return _binary_geo("intersection", self, other)
  File "/g/data/v10/public/modules/dea-env/20200713/lib/python3.6/site-packages/geopandas/base.py", line 60, in _binary_geo
    geoms, index = _delegate_binary_method(op, this, other)
  File "/g/data/v10/public/modules/dea-env/20200713/lib/python3.6/site-packages/geopandas/base.py", line 51, in _delegate_binary_method
    data = getattr(a_this, op)(other, *args, **kwargs)
  File "/g/data/v10/public/modules/dea-env/20200713/lib/python3.6/site-packages/geopandas/array.py", line 546, in intersection
    self._binary_method("intersection", self, other), crs=self.crs
  File "/g/data/v10/public/modules/dea-env/20200713/lib/python3.6/site-packages/geopandas/array.py", line 478, in _binary_method
    return getattr(vectorized, op)(left.data, right, **kwargs)
  File "/g/data/v10/public/modules/dea-env/20200713/lib/python3.6/site-packages/geopandas/_vectorized.py", line 723, in intersection
    return _binary_geo("intersection", data, other)
  File "/g/data/v10/public/modules/dea-env/20200713/lib/python3.6/site-packages/geopandas/_vectorized.py", line 285, in _binary_geo
    for s in left
  File "/g/data/v10/public/modules/dea-env/20200713/lib/python3.6/site-packages/geopandas/_vectorized.py", line 285, in <listcomp>
    for s in left
  File "/g/data/v10/public/modules/dea-env/20200713/lib/python3.6/site-packages/shapely/geometry/base.py", line 649, in intersection
    return geom_factory(self.impl['intersection'](self, other))
  File "/g/data/v10/public/modules/dea-env/20200713/lib/python3.6/site-packages/shapely/topology.py", line 70, in __call__
    self._check_topology(err, this, other)
  File "/g/data/v10/public/modules/dea-env/20200713/lib/python3.6/site-packages/shapely/topology.py", line 38, in _check_topology
    self.fn.__name__, repr(geom)))
shapely.errors.TopologicalError: The operation 'GEOSIntersection_r' could not be performed. Likely cause is invalidity of the geometry <shapely.geometry.multilinestring.MultiLineString object at 0x147950ffcdd8>

Add "eroding"/"prograding" description column to statistics

Regarding adding a simple plain-text description to charts of coastal change through time that will be displayed on Terria, e.g. pseudo-code "This location has {"eroded" if rate_time < 0 else "accreted"} by {abs(rate_time)} metres per year since 1988"? :

Ok so unfortunantly for our templating system we can only use a true/false flag, so if the data could have hasEroded property that would make our life easier

If a simple boolean column hasEroded could be added, this could be used to generate the chart:

{
    "catalog": [{
        "type": "wms",
        "name": "Coastline",
        "url": "https://nonprod.geoserver.dea.ga.gov.au/geoserver/wms",
        "layers": "dea:DigitalEarthAustraliaCoastlines",
        "featureInfoTemplate": "This location has {{#hasEroded}}eroded{{/hasEroded}}{{^hasEroded}}accreted{{/hasEroded}} by {{rate_time}} metres per year since 1988</br></br><chart title='{{name}}' y-column='Value' x-column='Year' column-units='{{terria.timeSeries.units}}' preview-x-label='YEAR'>Year,Value\n1988,{{1988}}\n1989,{{1989}}\n1990,{{1990}}\n1991,{{1991}}\n1992,{{1992}}\n1993,{{1993}}\n1994,{{1994}}\n</chart>" 
    }]    
}

Compared to a simpler implementation:

{
    "catalog": [{
        "type": "wms",
        "name": "Coastline",
        "url": "https://nonprod.geoserver.dea.ga.gov.au/geoserver/wms",
        "layers": "dea:DigitalEarthAustraliaCoastlines",
        "featureInfoTemplate": "This location has changed by {{rate_time}} metres per year since 1988</br></br><chart title='{{name}}' y-column='Value' x-column='Year' column-units='{{terria.timeSeries.units}}' preview-x-label='YEAR'>Year,Value\n1988,{{1988}}\n1989,{{1989}}\n1990,{{1990}}\n1991,{{1991}}\n1992,{{1992}}\n1993,{{1993}}\n1994,{{1994}}\n</chart>" 
    }]    
}

Implement temporal filter to remove non-coastline noise

Throughout the time series, some individual annual composites are affected by noise in ocean areas, possibly caused by unmasked clouds or sunglint. As these areas of noise appear semi-randomly with no relationship to the preceding or subsequent year, we should be able to apply a temporal filter to remove them from the dataset.

Proposed method:

  1. Take annual composites and threshold to identify land/water
  2. Dilate land area by X m to account for moving coastlines. Threshold TBA, but based on sensible max threshold of year-to-year change
  3. Compute a rolling sum of this dilated layer
  4. Create mask to remove any pixels that do not have a neighbor in either the preceding or subsequent annual composite (e.g. n > 1 neighbor).

image

Show non-significant (i.e. net-stable) points in WMS

This would allow users to click on a non-eroding or non-prograding beach and still get an informative graph back showing temporal change.

This would be particularly useful for wave-dominated beaches without persistent temporal trends, or where coastline movement is driven by fluctuating climatic drivers.

A mockup of how this would function (see also #39, #30, #29):

if wms_t_sig:

    if wms_t_grew:
        beach grew by wms_t_abs (+- wms_t_conf) 
    
    if wms_t_retr:
        beach eroded by wms_t_abs (+- wms_t_conf) 
    
    if wms_t_less:
        this rate was smaller than X warning        
    
if wms_t_nsig:

    beach was net stable since 1988 (e.g. no retreating or growing trend)

Add certainty flag for points with rates less than validation accuracy

Rates of change points with rates less than the validated accuracy of the data can still provide useful information for interpreting coastal change, particularly along transitions between erosional and progradational coastlines. However, we should add a certainty warning to these points that appears when they are clicked on in DEA Maps so that users are made aware that they are less reliable.

Investigate using Smartline as alternative to initial coastal mask

Pros:

  • Much simpler code
  • More flexibility to modify MNDWI threshold

Cons:

  • Would need to be modified in some areas of extreme change

E.g. update contour_preprocess with rasterisation of smartline buffer:

    buffered_geom = smartline_gdf.simplify(30).buffer(2000)
    coastal_buffer = rasterize(shapes=buffered_geom,
                               out_shape=yearly_ds.geobox.shape,
                               transform=yearly_ds.geobox.transform,
                               all_touched=True).astype(bool)
    buffer_ocean = ~coastal_buffer
    buffer_land = ~coastal_buffer

Implemented morphological filter to improve false positive cloud detection

Morphological operators can be used to remove false positive cloud detection along long, narrow bright sandy beaches. For example:

image

This recipe makes the largest difference to overall accuracy:

# Close mask to remove small holes in cloud, open mask to     
# remove narrow false positive cloud, then dilate    
mask = odc.algo.binary_closing(mask, 2)    
mask_cleaned = odc.algo.mask_cleanup(mask, r=(10, 10))

Use consistent prefix to label WMS-only attribute columns

Currently a number of columns in the rates of change statistics file are included only for the purpose of ordering and labelling the data in the WMS, and do not need to be included in the data package itself:

  • abs_time
  • retreat
  • growth
  • conf_time
  • conf_soi

To seperate these columns from others so it's clear they are used for web services only, these columns could be renamed consistently like

  • wms_abtime
  • wms_retr
  • wms_grow
  • wms_ctime
  • wms_csoi

List of future updates to DEA CoastLines web services

GeoServer

  • Add additional higher resolution summary layer using identical styling, to improve transition between high and low zoom levels
  • Update "annual rates of change"/"statistics" points labels to include standard error (e.g. "-2.4 m (ยฑ0.5)":
    rate_time + ' m (ยฑ' + 1.96 * se_time + ') ')
  • Show "annual rates of change"/"statistics" points labels only where abs_rate_time >= 0.2
  • Reduce "good" certainty line widths by ~20%
  • Reduce "uncertain" certainty line widths by ~35%
  • Increase transparency of "uncertain" shorelines from 40% to 30%
  • [ ] Fix This is a This is a, which generates map and See the [object Object] for more information in "About" section of WMS

Terria

  • Update text to be more contextual (e.g. "Coastline has retreated by X") using new growth/retreat boolean columns
  • [ ] Add sentence on SCE ("Coastline has changed over XX m total")
  • [ ] Use specific legends for both Supplementary_layers layers
  • Add message like "Zoom in and click on rates-of-change points to view a chart of coastal change over time" to summary layer statistics
  • Make statistics points layer the priority when both annual coastlines and statistics points are clicked on?

docker image fails to build

(base) ubuntu@:~/dea-coastlines$ docker-compose up
WARN[0000] The "DB_PORT" variable is not set. Defaulting to a blank string. 
[+] Building 11.5s (13/13) FINISHED                                                                                                                                                                              
 => [internal] load build definition from Dockerfile                                                                                                                                                        0.0s
 => => transferring dockerfile: 32B                                                                                                                                                                         0.0s
 => [internal] load .dockerignore                                                                                                                                                                           0.0s
 => => transferring context: 34B                                                                                                                                                                            0.0s
 => [internal] load metadata for docker.io/osgeo/gdal:ubuntu-small-3.4.1                                                                                                                                    0.0s
 => [1/9] FROM docker.io/osgeo/gdal:ubuntu-small-3.4.1                                                                                                                                                      0.0s
 => [internal] load build context                                                                                                                                                                           0.0s
 => => transferring context: 16.19kB                                                                                                                                                                        0.0s
 => CACHED [2/9] RUN apt-get update     && apt-get install -y     build-essential     git     python3-pip     libpq-dev python3-dev     ca-certificates     && apt-get autoclean &&     apt-get autoremove  0.0s
 => CACHED [3/9] COPY requirements.txt /tmp/                                                                                                                                                                0.0s
 => CACHED [4/9] RUN pip install --no-cache-dir --upgrade pip     && pip install --no-cache-dir -r /tmp/requirements.txt     --no-binary rasterio     --no-binary shapely     --no-binary fiona     && pip  0.0s
 => CACHED [5/9] RUN mkdir -p /code                                                                                                                                                                         0.0s
 => CACHED [6/9] WORKDIR /code                                                                                                                                                                              0.0s
 => [7/9] COPY . /code/                                                                                                                                                                                     0.3s
 => [8/9] RUN pip install /code                                                                                                                                                                             6.7s
 => ERROR [9/9] RUN  deacoastlines-raster --help   && deacoastlines-vector --help                                                                                                                           4.3s
------                                                                                                                                                                                                           
 > [9/9] RUN  deacoastlines-raster --help   && deacoastlines-vector --help:                                                                                                                                      
#0 3.944 /usr/local/lib/python3.8/dist-packages/geopandas/_compat.py:111: UserWarning: The Shapely GEOS version (3.8.0-CAPI-1.13.1 ) is incompatible with the GEOS version PyGEOS was compiled with (3.10.3-CAPI-1.16.1). Conversions between both will be slow.                                                                                                                                                                  
#0 3.944   warnings.warn(                                                                                                                                                                                        
#0 3.944 Traceback (most recent call last):                                                                                                                                                                      
#0 3.944   File "/usr/local/lib/python3.8/dist-packages/dea_tools/coastal.py", line 49, in <module>
#0 3.944     from otps import TimePoint
#0 3.944 ModuleNotFoundError: No module named 'otps'
#0 3.944 
#0 3.944 During handling of the above exception, another exception occurred:
#0 3.944 
#0 3.944 Traceback (most recent call last):
#0 3.944   File "/usr/local/bin/deacoastlines-raster", line 5, in <module>
#0 3.944     from coastlines.raster import generate_rasters_cli
#0 3.944   File "/usr/local/lib/python3.8/dist-packages/coastlines/raster.py", line 48, in <module>
#0 3.944     from dea_tools.coastal import model_tides, pixel_tides
#0 3.944   File "/usr/local/lib/python3.8/dist-packages/dea_tools/coastal.py", line 52, in <module>
#0 3.944     from dea_tools.pyfes_model import TimePoint, predict_tide
#0 3.944   File "/usr/local/lib/python3.8/dist-packages/dea_tools/pyfes_model.py", line 23, in <module>
#0 3.944     from pyfes import Handler
#0 3.944 ModuleNotFoundError: No module named 'pyfes'
------
failed to solve: executor failed running [/bin/sh -c deacoastlines-raster --help   && deacoastlines-vector --help]: exit code: 1

Allow minor versioning by allowing stats to be run without raster generation

Currently, running the DEACL statistics code requires the user to specify a single output_name string that was originally used to generate the DEACL rasters. This is used in the file names of the output vector files, which means that all runs of the stats code will overwrite each other until a new version of rasters (with a new output_name) is generated.

To avoid this and allow multiple versions of the vector files to be generated from a single raster generation, the DEACL stats code should support both an input input_name (e.g. v0.3.0), and an output output_name (e.g. v0.3.1).

Two tiles fail due to KeyError in `skimage.measure.find_contours`

Contour extraction on two Albers tiles are currently failing with a KeyError. This has been raised on the scikit-image Github here:
scikit-image/scikit-image#4505

These two tiles may not be able to be processed until this bug has been resolved and the new version of scikit-image made available on the NCI.

6922:
image

2524:
image

Error:

Traceback (most recent call last):
  File "/g/data/r78/DEACoastLines/deacoastlines_statistics.py", line 137, in contours_to_multiline
    for i in find_contours(da_i.data, z_value) 
  File "/g/data/v10/public/modules/dea-env/20200612/lib/python3.6/site-packages/skimage/measure/_find_contours.py", line 137, in find_contours
    contours = _assemble_contours(segments)
  File "/g/data/v10/public/modules/dea-env/20200612/lib/python3.6/site-packages/skimage/measure/_find_contours.py", line 174, in _assemble_contours
    ends.pop(tail[-1])
KeyError: (66.0, 381)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/g/data/r78/DEACoastLines/deacoastlines_statistics.py", line 1024, in <module>
    main()
  File "/g/data/r78/DEACoastLines/deacoastlines_statistics.py", line 923, in main
    dim='year').set_index('year')
  File "/g/data/r78/DEACoastLines/deacoastlines_statistics.py", line 196, in subpixel_contours
    for i, da_i in da.groupby(dim)}
  File "/g/data/r78/DEACoastLines/deacoastlines_statistics.py", line 196, in <dictcomp>
    for i, da_i in da.groupby(dim)}
  File "/g/data/r78/DEACoastLines/deacoastlines_statistics.py", line 143, in contours_to_multiline
    fully_connected='high') 
  File "/g/data/v10/public/modules/dea-env/20200612/lib/python3.6/site-packages/skimage/measure/_find_contours.py", line 137, in find_contours
    contours = _assemble_contours(segments)
  File "/g/data/v10/public/modules/dea-env/20200612/lib/python3.6/site-packages/skimage/measure/_find_contours.py", line 174, in _assemble_contours
    ends.pop(tail[-1])
KeyError: (1109.0, 15)

Document functions in DEACL code

Add docstrings to all DEACL functions to aid in reproducability.

To do:

  • deacoastlines_generation.py
  • deacoastlines_statistics.py
  • deacoastlines_summary.py

Implement DE Africa improvements in Australia

Working list:

  • Package install from requirements
  • Closing operation on all time
  • Mask out inland section of coastal study area when generating annual masks
  • Continental export to subfolder
  • Hotspots radius param
  • Clone instructions
  • Upload supplementary data to s3
  • Annual certainty masking

Coastal change directionality incorrect in some areas of bright sand

@LeoLym encountered an issue where annual coastlines data appears nicely ordered and consistently eroding:

image

However, the derived rates of change statistics alternative between eroding and prograding (despite this not matching the underlining coastlines):

image

This is possibly caused by bright oversaturated white sand producing a MNDWI value similar to water, which confuses the analysis step which determines directionality by comparing the "wetness" of the baseline vs. comparison year.

DEA Maps example: https://maps.dea.ga.gov.au/#share=s-xSwMvkcagnD8fsqXAzZCbOM8bN4

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.