nismod / infra-risk-vis Goto Github PK

General priorities

• Extracts from global datasets to download data at country-level
• nismod/irv-frontend#13 (enables linking to map from asset, sharing link)
• nismod/irv-frontend#6

Implement unit and integration tests across the backend API

Linked to #113

Plot sum EAD for an asset over all hazards, for a given RCP/epoch.

Limit to sufficiently common RCP/epoch combinations

Include fluvial/pluvial flooding under future scenarios once processed (WIP).

There's a variety of attributes on different assets in different sectors.

Can we handle things generically? Current map over keys gives just-about readable labels. Ordering is data-dependent, and there's no way to break attributes into logical groups, or skip or combine them in the FeatureSidebar > List.

• water - currently only include potable network and a sample of abstraction points.
• transport - ports and airports, road nodes? check rail nodes (distinguish junctions from stations?)
• power - substations, check generation and lines

Remove tech-debt with duplicate workflows and move ETL/pipelines to best pract Snakemake repository structure

Separate road classes into layers

Check vector density of tertiary/unclassified roads - especially roads around Kingston in zoomed-out view. Can tippecanoe show more data here?

or submit a PR for Terracotta upgrade to do this?

Along with #19, click a region to display a summary sidebar

• Region name
• charts showing direct/indirect risk over epochs, scenarios, hazards
• charts showing breakdown by infrastructure sector
  

Seem to come up as darker areas, perhaps around tile joins?

Possibly related - do we already have (or can we test) adding terracotta optimize-rasters to the raster process pipeline?

Update when Terracotta Master (which has Postgres ingegration for backend DB) is released.

This also means we could drop MySQL DB

Implement a mechanism for supporting spatial data with a very large number of attributes, without increasing the size of vector tiles unnecessarily.

Current options:

• serve only basic data through MVT, build an API for fetching additional properties for the visible features, merge the data on the frontend side
• given the small number of users, move towards generating MVT on the fly through PostGIS and a web tile server like https://github.com/urbica/martin

Noting an idea for hazard plotting: there may be a particular hazard intensity (depth/wind-speed) threshold which is particularly relevant, so the map could plot the least return period (or interpolated return period) above which that threshold would be exceeded. The intensity threshold could be controlled by a slider or numeric input.

• mask zeros (or values <0.01m) to avoid plotting large non-flooded areas
• fix coastal layer naming and display for baseline. Rename TIFFs and layer name expectations to set RCP to baseline (not 4x5) in 2010 timestep.

Split layers by "asset type" column in input data:

• well
• treatment plant
• pumping station

Notes on efficient style updates, seems to be a problem for some updates via react-map-gl

Following the how the code handles style updates:

• react-map-gl hands off to mapboxgl.Map: react-map-gl/src/mapbox/mapbox.js#L358-L360
• mapbox.Map.setStyle gets down to diff, which generates the console warning we sometimes see: mapbox-gl-js/src/ui/map.js#L1407-L1418
• the diffing work actually happens between mapbox-gl-js/src/style/style.js#L490 and mapbox-gl-js/src/style-spec/diff.js#L319
• setFilter is in the list of supported diff operations at mapbox-gl-js/src/style/style.js#L74

Could be worth injecting some debugging if the full map repaints when we think it shouldn't need to. What precisely are we doing that forces a full style rebuild?

• Add place names to the map
  Show admin boundaries (optionally - in the sidebar section? or change at zoom levels?) (moved to separate issue: #32 )

Triggers validation error, shows up as internal server error in adaptation options table.

...
  File "pydantic/main.py", line 511, in pydantic.main.BaseModel.parse_obj
  File "pydantic/main.py", line 331, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for AdaptationCostBenefitRatioParameters
eael_days
  ensure this value is greater than or equal to 1 (type=value_error.number.not_ge; limit_value=1)

Split out from #26
Add administrative boundaries layer to map.
Enable toggling boundaries on/off.

Plot aggregate risk at admin boundary level: national/parish/district

Choose RCP/epoch, single hazard/combined

When hiding a layer from view, if the selected feature comes from that layer, the selection should be cleared.
Currently, the highlighted selection is still displayed, even though all other features of that layer aren't visible.

Steps to reproduce:

• click on a feature to select
• hide layer to which the feature belongs
• the highlighted feature is still visible

Follow-up from #78 and #79

• nismod/jamaica-infrastructure#12
• run any updated results into the database
• remove remaining correction factors:

  infra-risk-vis/src/details/features/adaptation/AdaptationSection.tsx

  Line 43 in c5ee7f2

  // TODO: remove factor when data is updated

  and

  infra-risk-vis/backend/backend/app/schemas.py

  Line 129 in c5ee7f2

  TODO: fix the data so we don't need this function.

There have been a series of versions and releases, project-driven:

• Argentina (released and handed over) - tag v0.1-argentina
• SEAsia (released and handed over) - tag v0.2-seasia
• Jamaica (released, may have further development) - tag v0.3.0
• Caribbean (released, may have small changes) - branch feature/caribbean-prototype
• HVT East Africa (in development, to demo) - branch feature/hvt-vis
• GRI (next)

Requirements:

• main branch development should work towards configurability and general use
• project deployments need specific configuration
• there is now some overlap in projects, which may see maintenance or further development

Suggest:

• explicit release branches for tools as deployed
• main line work going forward will be mostly focussed on GRI
• leave open the question of design to handle multiple levels/collections of analysis, auth and access, currently keeping things separate and branching off the codebase.

Mapbox GL is no longer open source as of v2.

Consider pricing and tiers for low usage.

Likely open alternative is the fork of v1 at https://github.com/maplibre/maplibre-gl-js/

Tileserver-gl (which we use to serve vector tiles) is looking at similar migration needs: https://github.com/maptiler/tileserver-gl/ issue 542

• Map location
• Intro text
• Drop OIA logo
• Drop later sections

The goal is to display the mapped damage value of the hovered feature in the hover tooltip.

Currently, the direct damages values which are shown as colours on the infrastructure map, are not shown anywhere in text.
While the Expected Annual Damage values are shown in bulk in the selected feature sidebar, the data for the current parameters is not displayed in the hover tooltip, and the total damage figures are not displayed anywhere in the app.

After separating out UI into new repo

Plot line chart in asset sidebar, EAD against epoch, different colours for RCPs, separate plot per hazard.

Example vega-lite spec - to be constructed from asset attributes:

{
  "$schema": "https://vega.github.io/schema/vega-lite/v5.json",
  "width": 300,
  "height": 200,
  "data": {"values": [
    {"rcp":     "2.6", "epoch": "2050", "ead":  5210000},
    {"rcp":     "2.6", "epoch": "2100", "ead":  7520000},
    {"rcp":     "baseline", "epoch":    "2010", "ead":  4250000},
    {"rcp":     "4.5", "epoch": "2030", "ead":  4800000},
    {"rcp":     "4.5", "epoch": "2050", "ead":  5250000},
    {"rcp":     "4.5", "epoch": "2100", "ead":  8100000},
    {"rcp":     "8.5", "epoch": "2030", "ead":  4850000},
    {"rcp":     "8.5", "epoch": "2050", "ead":  5610000},
    {"rcp":     "8.5", "epoch": "2100", "ead":  10500000}
  ]},
  "mark": {
    "type": "line",
    "point": {
      "filled": true
    }
  },
  "encoding": {
    "x": {"field": "epoch", "timeUnit": "year", "title": "Year"},
    "y": {"field": "ead", "type": "quantitative", "title": "Expected Annual Damages"},
    "color": {"field": "rcp", "type": "nominal", "title": "RCP"}
  }
}

• Implement Alembic in backend DB Model

For a given location - which may be a current or future infrastructure asset location - I would like to know all the hazards which may affect it.

• Do any hazards have no effect on the location? In that case, report no exposure.
• What are the depths/wind speeds at all return periods? Line graph with 1/return period on x-axis, depth on y-axis.
• What are the likelihoods of extremes, from an event set? Histogram of depths.

Implementation notes - may be an API call rather than pushing all data client side:

GET /exposure/rp?lat=0.00&lng=0.00

[
  {
    "hazard": "coastal", 
    "exposure": [0,0,0,1,2], 
    "return_period": [5,10,50,100,250] 
  }
]

^ exposure and return period entries correspond

GET /exposure/events?lat=0.00&lng=0.00

[
  {
    "hazard": "coastal", 
    "events": [2.3,7,2.3]
  }
]

^ events is list of non-zero depths from event set

The current highlight mechanism seems to be affected by the splitting of individual features into potentially multiple vector tiles.

How to replicate:

• hover over a long line feature, e.g. a high voltage power line spanning a large part of Jamaica - the whole feature will be highlighted
• click on the feature, it will be selected and information will be displayed in the feature sidebar
• move mouse away from the feature. Presumably, parts of the line will be de-highlighted, and only the immediate fragment that was clicked will be still highlighted

This seems to be because the current selection highlight mechanism doesn't work well with tiled data. Needs investigation to identify fix.

Colour transport assets / buidlings by economics (total/sector):

• transport flow allocation
• building economic sector allocation

Vector Tileserver
Backend API

Organise layer selection sidebar in the following manner:

• collapsible sections, each representing a spatial data layer that can be shown on the map
• sections have a "style" subsection (whenever relevant) which allows the user to choose what data is mapped on top of the selected spatial data (e.g. regions -> choose from population, aggregate damages etc)
• both the spatial layers (sidebar sections) and data styles (style dropdown entries) can be categorised into primary (focus) and secondary (context). This depends on which tab the user is currently viewing.

For example: when viewing the Risk tab, assets, regions and hazards are primary layers, while elevation is a context layer. When it comes to the regions layer, aggregate damages is a primary style option, while population and GDP are context style options.

Not all raster cells have numerical value displayed in tooltip on hover

Compare two cells from cyclones layer:

The issue seems to be with the looking up of the colour value. Perhaps the colour comparison has inaccuracies that cause no lookup value to be returned from the legend.

Search box to filter on place names, select to zoom to location.

The aim here is to display raster data (e.g. flood return period maps showing depth of flooding) directly, coloured to show data values (depth of flooding), still visible when zoomed out, and clearly pixelated at the resolution of the data when zoomed in.

The key map interaction is to pick out the depth values and show on hover in a tooltip (Flooding: 3.2m).

Otherwise we want to show and hide layers - change climate scenario, model, return period; potentially show multiple hazards overlaid.

Notes from @mz8i :

• Terracotta serves Cloud-Optimised GeoTIFFs as raster tiles
• deck.gl allows picking pixel values from a BitmapLayer on the client

Notes on related approaches:

• mapbox-gl-js allows finding pixel coordinates from a click on the map map.project and accessing the canvas element map.getCanvas, which might allow picking pixel values - if the map is created with options.preserveDrawingBuffer: true using webglContext.readPixels. Seems a bit involved, not sure if there's a neat way.
• leaflet and WebGL together in leaflet.tilelayer.glcolorscale - see blog post.

Infra-Risk-Vis repository can be split into seperate repos for Backend API, UI and ETL. This could have the following benefits amoungst others:

• Easier to pickup for developers / PR's etc
• Easier containerised builds
• Potential for simpler deployment to AWS services directly from repos
• Easier to integrate into CI/CD pipelines.

This relates to work on AssessmentPage.tsx for the east-africa project.

• numeric inputs
• text notes on any changes from default
• toggle to show/hide zero values
• summary wording to describe final outcome "positive effect on environmental sustainability, slightly positive effect overall..."
• add intervention
• include generic option, default neutral for all
• c/r/u/d locally
• save/load to file
• Fix weights slider (clicking resets to zero)
• include worked example
• map indicators to SDGs as possible - see p20 (4.1) report
• draft guidance and guide

Consider dependencies and requirements for the various services:

• raster server: python + terracotta, access to TIFFs and build sqlite db
• vector server: node + tileserver-gl-lite, access to mbtiles
• frontend dev server: node + react etc, access to src, proxy to other endpoints
• backend API server: python, pipenv, connect to database
• database

Separately, snakemake workflow processes tiffs for terracotta and vector data into database then into vector tiles. Current way to run snakemake is to change to backend directory, run pipenv shell to pick up dependencies and database connection environment variables, then change over to etl and run snakemake commands.

Goal: improve ease and reproducibility of initial setup and everyday spinning up for development. See initial sketch with docker-compose for backend API and database. Current state is to open four terminals/tmux screens and run:

• npm start
• npm run raster
• npm run vector
• pipenv run uvicorn backend.app.main:app --host localhost --port 8888 (with database connection details in .env)

See

For terraform config that sets up RDS