nasa-impact / covid-dashboard Goto Github PK

Get population dataset into the browser.

https://sedac.ciesin.columbia.edu/mapping/popest/covid-19/

Manil can connect us with SEDAC PoC if necessary.

Populate comparison slider:

• currently selected month against same month 5 years ago
• limit timeslider to 2010 - latest

When you generate a time series in the UI, the units listed for NO2 concentration are mol/cm^3. This should be changed to molecules/cm^2 or molec/cm^2.

Note there are two changes:

1. cm^3 -> cm^2
2. mol -> either molec or molecules (a 'mol' is a different unit)

In addition to the Data API (#3), we will need to store the configuration and content of each dataset.

This will live in the repo, where each dataset has its own folder inside app/assets/datasets. The spec will be documented in the README.

Had a conversation with Laura Lorenzoni this morning, about the water quality product. They will produce a single indicator, only for the spotlight areas. Available from June 19 onward.

In the meantime, we will start talking to the science team to make sure they produce data we can pull in.

Sent this email with background to Laura.

Dashboard
A prototype for the agency dashboard is in development here: http://cv-eo.surge.sh/. This contains the general outline of the site. Expect this to change every couple of days as new features are released.

The dashboard will contain a couple of views:

• the global products. This page shows data products with global coverage, focusing on map layers and timeseries data.
• the spotlight areas provide more detailed
• dataset pages, where your team can provide more background information about the data you produce.
  Some products will have data for both global and AOI views (eg. No2), others only for the spotlight areas (eg. Nightlights and Water Quality).

API
All data is served through an API. This contains mostly:

• map layers. Map data is served in Cloud Optimized GeoTIFF (COG) format. This format allows us to build a more dynamic experience for the user.
• indicator data. This is mostly used at the AOI level to generate line-charts with timeseries data. A good format would be CSV, but we can support others.
  We can support your team to generate this data in the right format.

Spotlight areas
The API contains an overview of the spotlight areas for all three agencies: https://8ib71h0627.execute-api.us-east-1.amazonaws.com/v1/sites
You should feel free to use this endpoint and integrate in your data generation process.

Next step
I'd love to have a conversation with your team. In particular about:

• what you are planning to produce. Spatial and temporal resolution, output formats, frequency of updates, etc.
• what kind of insights you're expecting to see, and we can communicate these clearly to your users
  Based on this conversation, we can come up with a more detailed plan about how our teams collaborate over the next couple of weeks.

We want to backward process so time series analysis from before COVID period to present is possible.

Right now, I'm thinking we can use AWS batch to parallelize wget of all HDF files, run processing, store the resulting COG on S3.

Note this will not be the only dataset on the platform so to whatever extent we can generalize the pipeline to be configurable, we should do that. We can also consider Cumulus.

It is possible that we will have access to global nightlights data.

What basemap should we use for the project?

Depending on the desired behavior of the area of interest feature, its UX may need to be reviewed:

Current behavior

• User draws an AOI and the insights panel shows information.

Questions

• Is the AIO related to the enabled layers, or does it show all available info about that area?
• Is the answer to the above question clear for the user?

To show the value and power of COG, we want to allow users to get time-series data for an arbitrary area:

1. user selects / draws an area
2. the API returns time-series data for the area (mean might be the most useful)

This should include weighting the values, to account for partially covered pixels. The AQ data is very coarse, and pixels near the edges will only be partially covered.

Next step

• build a proof of concept endpoint

Capturing notes based on our update meeting with MSFC today. Please comment if I missed or misinterpreted something.

Goals

By Thursday COB we’ll deliver a dashboard to the broader group of partners that:

1. shows a clear product vision of how users will interact with the data. Users should be able to see:
   1. global level insights versus micro-level insights
   2. how the situation is changing over time. Particularly comparing the current situation against a baseline
   3. easily highlight
2. allows NASA to tell a good story about the value of Application Optimized Data (COGs). Both towards the ESA and JAXA, as the science users that will provide the data.

Included in delivery

Concretely, the delivery should include:

• global products
  • monthly No2:
    • allow people to go back and forth in time with timeslider
    • stretch goal - before and after
  • population layer
  • ability to click an area, and dynamically get timeseries from COG
• super-sites - communicate how the super-sites will be integrated in the site. It’s unlikely we’ll have real data by Friday, we should figure out what we can achieve by then

For a full list, see this milestone: https://github.com/NASA-IMPACT/covid-dashboard/projects/1

There is a need to label the map and make it more clear what the user is looking at.

This is critical for the comparison feature. We need to clearly label what the user is seeing on both sides.

This is - to a lesser extent - also true for regular map mode (non-comparison). The only indication is the on/off toggle for the layer, which is not super obvious. Particularly when the application is loaded with default layers, and the user didn't have to turn on the layer. This is feedback I received from a couple of people on the Vulnerability side.

The OMI team will produce indicator data from their side.

To get a sense of the current structure, here's what they currently produce: OMNO2_TImeseries-sample.zip

Next step:

• review the structure and provide feedback to OMI team. @drewbo do you want to have a look together and see how we start to shape the indicator API?

We need to work on guidelines for data providers. First notes are here: https://paper.dropbox.com/doc/Guidelines-for-data-providers--A0fUy2eMPzNl47xauoMAs_VZAg-j6Xiiho6VPWgfm1LLif2J

Next steps:

• flesh out the loose notes - @olafveerman
• think about open questions COG - @drewbo

Right now the HDF to COG pipeline is hardcoded only to accept a particular OMI NO2 dataset. We would like to parameterize this process and make some components more bombproof so that in the future new datasets could be added and converted to COGs more easily.

Create a navigation dropdown menu for the datasets

Initial design of how we want to structure the Data API. Ultimately, this will power:

1. 🗺️ map based visualization
2. 📉 charts that highlight evolution over time, potentially by some level of aggregation (city, state, etc)

The immediate need for the May 15th dashboard are the map based visualization.

Next step

• come up with a brief description of the architecture to inform a) the data processing, and b) the frontend implementation.

We want to update data regularly. We can use some AWS scheduling (cloudwatch event) to execute a step function every X days. We probably want a lambda function which can query for new files and run same process (wget file, process, write to S3) from backward processing.

Also look into https://github.com/mapbox/ecs-watchbot and S3 Batch operations https://github.com/developmentseed/how/issues/253

Replicate

1. enable NO2
2. zoom the map a couple of levels
3. enable the compare feature

What happens

The left map stays on zoomed location. The baseline map on the right starts at the global view

What I expected

The baseline loads at the same zoom and location as the left map.

Document how to add COG color maps

Scaffold the dashboard:

• fork Vulnerability dashboard
• two sections: explorer + datasets
• CI push to Surge on develop
• add a placeholder dataset

Ingest indicator data + TIF for car counts in Beijing Airport

Work so far

Current code which generates a COG (https://gist.github.com/abarciauskas-bgse/1144867ca34f61556d8eb05a507ca01a) is just a straw man. It generates a valid COG but has not been "scientifically validated"

Information from data set experts

From Zachary Fasnacht:

1. You actually want to use the field /Data_Fields/ColumnAmountNO2TropCloudScreened. This field is the tropospheric NO2 with cloud fraction of greater than 30% screened out (among a few other screens mentioned in the Readme Lok sent along). This is typically what we look at when we are interested in near surface NO2 trends from OMI.
2. The coordinate system for OMI geolocation is WGS84. As far as I am aware, we do not currently have code for converting the data to tif format.

From Lok Lamsal:

We have data in the raster file format at https://avdc.gsfc.nasa.gov/pub/data/satellite/Aura/OMI/V03/L3/OMNO2d_HR/OMNO2d_HRM/. Please take a look at ".dat" files. You might be able to use those data instead.

Open questions

Are COGs a reasonable representation of this data for what we need to do? Assuming what we need to do is:

1. Visualize the data on a map
2. Generate time series
3. Support ML training

Show how spotlight location indicator data can be queried from the API

We should set up a super-site endpoint with basic metadata like:

• name
• bounding box
• polygon

Primarily built to power our own dashboard, but potentially also useful to have a single source of truth towards:

1. technical contacts for the data products. Particularly if they're providing indicator data for the AOIs
2. ESA and JAXA to pull into their dashboard

Thoughts @danielfdsilva @drewbo ?

Current behavior

• Timeline is enabled for the NO2 layer which has monthly data.
• User can select any month (within date domain) using the graph, or navigate month by month using the arrows.
• The graph shows the date domain of the data (extent to which the data exists), having no relation with the actual values.

Questions

• Should the timeline support multiple timeseries layers at the same time? How are charts with different domains handled? (Must also consider how layers are presented on the map)
• In case we have multiple timeseries present should we (and how do we) support different interval navigation (days or years instead of months).
• Should we add support to an overview chart for layers that have the needed data? How does everything fit within the available space?

We should mimmick the color range that the current NO2 products follow. This repo contains info about the color bar: https://gitlab.com/zfasnacht/global_mapping/-/tree/master/Colorbars

Currently the timeseries displayed in the insights panel is hardcoded to NO2.
This needs to be made dynamic to handle any timeseries layer.

Given the tight turnaround, we will not be producing wireframes or mocks. Instead, a quick sketch with main interface layout / components will be enough to inform development.

From Manil:

We need an page [...] explaining indicators something like this - https://sealevel.nasa.gov/understanding-sea-level/key-indicators/global-mean-sea-level

@ricardoduplos do we need to design this? Can we use a pattern from a project like Phenom?

The banner needs to be styled using the Earthdata design guidelines:

https://cdn.earthdata.nasa.gov/eui/latest/docs/eui/index.html#logo

Coordinating with Kevin Lange, our technical point of contact that will configure DNS.

JAXA intends to publish COG and indicator data through our API.

Waiting for them to:

• provide a sample of COG and indicator data
• let us know if they can push COG to a S3 bucket going forward

This was raised in a conversation with Barry Lefer.

The issue

Many datasets will have large outliers that make it hard to see nuance in the lower range. In the case of NO2, the high emissions in China make the rest of the world look relatively the same.

Dynamically adjusting scale

One way to solve for this, is to allow users to adjust the color scale dynamically.

Conceptually this would be the same as setting the scale of a gradient in a design tool. Default:

The middle stop would be adjustable, allowing the user to slide to the left or right.

I can imagine this is feature for advanced users, but makes the tool a lot more useful for them. It also ties in well with our story around the importance of Application Optimized Data.

cc @ricardoduplos

Current behavior

• Comparison is controlled from the timeline.
• Comparison baseline is set to the selected month minus 5 years #33.
• Comparison is implemented only for raster data (through adding a date to the tile endpoint).

Improvements

• Comparison controller should be outside the timeline as it should be possible to have this feature for a non timeseries layer (layers without a timeline component).
• Should be cleared what the baseline for the comparison is, and warn the user when it changes.

Questions

• Should the user be able to select the comparison baseline?
• Should we support other types of data types (features or style layers instead of raster tiles)?

NO2 monthly is currently upside down.

We need a timeslider that allows users to change what timestep we're showing a map layer for.

Let's assume we don't have a trend line for all datasets. The timeslider should be intuitive to use even without.

Contribute to document for ESA and JAXA to outline:

• tech stack for the frontend
• API capabilities

It's likely that we'll have to rename supersites to something else. Supersites (or Superfund site) is a term used by EPA to indicate polluted places in the US that require long-term cleanup.

Next step: figure out what to call them

What branding should we include? Only NASA? How are ESA and JAXA reflected on this site?

All three agencies are producing two types of products:

1. global datasets (eg. NO2 and population)
2. economic indicators for select AOIs, or supersites

These types are an organizing principle for the interface. We have a good idea how we incorporate the global layers. Now we need to incorporate the super-sites

Super-sites

There will be around 5 super-sites (AOIs currently being defined) for which economic indicators are produced. It's likely that each of these super-sites will have different indicators and require custom interaction. I can imagine we have (at least) the following interface elements:

• a detailed map of the site. ability to switch map layers, some of which are time-series
• ability to click on areas in the AOI and get detailed insights. Different graphs and indicators for each super-site

On the COVID Vulnerability tool, we prioritized building an agnostic insights panel to ease the setup and maintenance of different instances. This is less of a concern here. We should err on the side of flexibility. It's more important that we can present thoughtful content and user interaction for each AOI, than that this scales with ease.

Example super-site - port

Shared by Anca from ESA:

1. single AOI for the port area
2. sub-AOIs within a port (eg. area dedicated to oil, chemicals, etc)

For each sub AOI, they will produce an indicator per time-step. Eg. number of containers per day.

Each indicator has specific rules to classify whether a particular trend is neutral (no significant change), positive or negative. Eg. if the rolling average increases by X for Y consecutive days, this should be highlighted.

Next steps

Goal - for the Friday delivery, the interface should communicate how we propose to incorporate the super-sites. It's unlikely we'll have real data.

• get an overview of the super-sites #17
• think how we incorporate this in the interface @ricardoduplos

Test if we can pull in data directly from GIBS into the frontend.

@drewbo Do you know how we can discover the tile endpoints for these different layers?

@olafveerman @abarciauskas-bgse do you know yet what the infrastructure is likely to look like for generating COGs (or some other Application Optimized data set) from the NASA air quality data? I understand that you still may be gathering data about the data.

Manil can you provide any additional context on what you are looking for?

We need to get some good scientific content developed for the front page of the NASA COVID-19 dashboard. I would expect this will include text describing the variables, why they were chosen and what they show/why they are important. Additionally it would be good to have some trend graphs like NO2 trends over spotlight cities/regions/global and similar graphs about night lights intensity over regions. Can you please work with Manil to develop a plan to do this. Perhaps the PIs will be the ones to put this information together? I think it is important that we get the content developed and approved prior to 5 June.

Work with OMI to pull in their NO2 product and process into something that we can use in the dashboard.

Get the spotlight areas from the API: https://8ib71h0627.execute-api.us-east-1.amazonaws.com/v1/sites

Manil needs more info about our stack:

... list of requirements in terms of services, software stack for the dashboard deployment in NASA cloud?

Services

Assuming this is hosted on AWS NASA Cloud:

• frontend
  • served from S3
  • CloudFront - optional
• COG
  • served from S3
• API
  • ECS + Lambda
• Data processing
  • EC2 and Lambda (compute) via Step Functions (for forward processing) and AWS Batch (for backward processing). Both Step Functions and Batch will be leveraging EC2 instances via ECS.

DNS

Earthdata provides the domain and redirect to where we host it. Manil put me in touch with Stephen Berrick who will help set up earthdata.nasa.gov/covid19.

@abarciauskas-bgse @drewbo What's missing? How do we host the API?