Challenge 15- The CDS-RTTOV-box

Stream 1 - Software development for weather, climate and atmosphere

Goal

Development of a CDS toolbox application to visualize model atmospheres as they would be seen by satellites from space.

Mentors and skills

• Mentors: @EddyCMWF
• Skills required:
  • Coding (Python)
  • Earth observation
  • Experience with geospatial datasets

Note: Challenge is funded by Copernicus. Only nationals from the European Union and ECMWF Member States are eligible to apply (see Terms and Conditions).

Challenge description

Satellite observations are critical in modern weather forecasting systems and there is an urgent/ongoing need to train more scientists in their exploitation. Highly sophisticated space-borne sensors make incredibly accurate radiation measurements of emissions from the atmosphere, but the relationship between these measurements and the atmospheric quantities we want to understand (e.g. temperature, humidity and chemical composition) is complex.

Radiative transfer models (computer codes, e.g. RTTOV) exist which make this link and are embedded at the centre of our operational data assimilation and modelling systems.

This project would build a prototype web interface that would allow students, trainers and researchers to run these codes and visualise the outputs via a web interface. For example, seeing how a change in carbon dioxide in the atmosphere would influence the satellite measurements gives an insight into how best to use these measurements. Pollution events can be similarly visualised from a "satellite view" perspective.

The GUI would be built with the CDS toolbox which already has access to the ERA5 model output as well as a number of satellite products and other model simulations. The resulting "application" could then be published on the CDS website and advertised to the Earth Observation community, potentially inspiring further developments and applications.

Challenge 33- WEkEO in a glance

Stream 3 - Visualize weather, climate and atmosphere

Goal

The project aims at developing a prototype of a visualization tool based on the WEkEO* Platform.

*WEkEO is a joint ECMWF, EUMETSAT, MiO initiative to grant access to Copernicus datasets from different sources and incentive Users developing applications in the provided cloud frame. Computing facility will be provided by WEkEO.

Mentors and skills

• Mentors: @stewartchrisecmwf @wekeo
• Skills required:
  • Data handling
  • Python programming
  • Visualization
  • API and database interrogation

Note: Challenge is funded by Copernicus. Only nationals from the European Union and ECMWF Member States are eligible to apply (see Terms and Conditions).

Challenge description

What data/system do you plan to use?
A set of Copernicus data in WEkEO including CAMS, ERA5
and selected satellite (Sentinel5, Sentinel3).

What is the current problem/limitation?
Users have often expressed the need to visualize all data available in a defined time and geographical window for different purposes as:

• Identify the useful datasets
• Gather insights on a specific event
• Have a tool for communication, outreach and training
  Up to now, WEkEO offers a visualization platform allowing the overlay only selected datasets.

What could be the solution?
Based on the expected user’s needs, a desirable feature would be to easily select and visualize all datasets of interest for a specific event (e.g. a wildfire, a specific weather pattern, an intense pollution) before downloading all or part of them. This may also address the need to identify in advance which datasets fit the needs and in which way the searched feature is visible.

Some ideas for the implementation
Desired features:

• Select a spatial window and a temporal frame
• Report the available datasets
• Provide the option to select one or more variable and one timestep
• Map the selected ones on the spatial window for the selected time
• Produce an API to access/download the selected data

The proposed solution may gather from existing solutions.

Challenge 21 - Machine Learning to improve the CAMS global air quality forecasts

Stream 2 - Machine Learning for weather, climate and atmosphere applications

Goal

Develop an ML algorithm to predict (and correct) the time-varying bias of the global CAMS forecast for surface PM2.5, O3 and NO2 at the locations of air quality monitoring stations.

Mentors and skills

• Mentors: @JohannesFlemming , @miha-at-ecmwf @jerome-barre-ecmwf
• Skills required:
  • Machine Learning
  • Time-series analysis
  • Handling geospatial data
  • Basic understanding of air quality observations and models
  • Python

Note: Challenge is funded by Copernicus. Only nationals from the European Union and ECMWF Member States are eligible to apply (see Terms and Conditions).

Challenge description

CAMS/ECMWF runs a computer model to predict global air pollution at a spatial resolution of about 40x40km (grid boxes size). While the CAMS model predicts the observed air quality mostly reasonably well errors in the prediction can occur because of the necessary simplification of the CAMS model and the uncertainties in the input data such as the emissions.
The main task is to develop an ML approach to predict the forecast errors at the location of air quality stations in order to correct them as a post-processing step. The observations to be used are hourly observations of surface ozone, NO2 and PM2.5 from about 2000 stations worldwide as provided in the openAQ data repository.

We suggest the following steps towards the solution:

1. Build ML model
   Train the ML model to predict the difference between forecast and station observations using data from a recent previous year (2019 or 2018).
   The input to the correction algorithm can be the CAMS model-forecast of the air quality value (O3, NO2, PM2.5) and forecast meteorological parameters (temperature, wind speed, etc. ).

2. Test performance of ML bias predication with independent data
   Test the performance of the ML model for recent forecasts of the 2020-2021 period. Use basic error statistics such as bias, RMSE and correlation to compare the forecast accuracy of the ML-corrected forecast against the uncorrected forecast.

3. Model error analysis (optional)
   Investigate the importance of the individual predictors and do a spatial analysis to identify patterns that could be used to better understand or improve the forecast model.

Hi,

I attach some data in GRIB and NetCDF
There are 5 parameters on 3 time steps.

2m temperature.
Total precipitation
low cloud cover
medium cloud cover
high cloud cover

Hope everything is Ok ..

Sylvie
data.zip

Challenge 32 - Building interactive weather visualisations

Stream 3 - Visualize weather, climate and atmosphere

Goal

Python package to build animated and exciting visualisations in KML (e.g. for Google Earth), which combines visualisations from other tools (Magics, Metview, ...)

Mentors and skills

• Mentors: @StephanSiemen, @iainrussell
• Skills required:
  • Python
  • KML

Challenge description

A lot of data sets at ECMWF, especially data from the Copernicus services, tell very striking stories about our climate, weather and pollution. While tools exist to plot maps and time series on their own, we would like to provide Python tools for users to build animations combining multiple plots into a scene through scripts. We would like to make use of the rich sets of features the KML format offers to combine visualisations and metadata. Interest candidates will have a chance to work with ECMWF staff to extend existing Python tools and learn about the visualisation of weather and climate data.

Challenge 11 - Jupyter widgets to help process and explore meteorological data

Stream 1 - Software development for weather, climate and atmosphere

Goal

To bring many of Metview's existing desktop-based interactive tools into Jupyter notebooks.

Mentors and skills

• Mentors: @iainrussell, @sandorkertesz
• Skills required:
  • Javascript
  • Jupyter notebooks
  • Jupyter widgets
  • JupyterLab
  • Python

Challenge description

Metview is ECMWF's software package for accessing, processing and visualising meteorological data. It has a rich set of interactive GUI tools in a Linux/macOS desktop environment, making it easy to configure things such as plotting styles, and to explore data. Inside a Jupyter notebook environment, users can use Metview's python API to script tasks, but it lacks these interactive tools. We would like to provide these tools inside a notebook environment. In particular, we suggest that Metview's icon editors and its data examiners be implemented as Jupyter widgets. Integration into Jupyterlab as extensions could be a further step to take.

Challenge 24- CliMetLab - Machine Learning on weather and climate data

Stream 2 - Machine Learning for weather, climate and atmosphere applications

Goal

Extend new Python ML package and help to mature package

Mentors and skills

• Mentors: @b8raoult @floriankrb
• Skills required:
  • Python
  • ML
  • Handling of weather data and formats

Challenge description

CliMetLab is a Python package aiming at simplifying access to climate and meteorological datasets, allowing users to focus on science instead of technical issues such as data access and data formats. It is mostly intended to be used in Jupyter notebooks, and be interoperable with all popular data analytic packages, such as NumPy, Pandas, Xarray, SciPy, Matplotlib, etc. and well as Machine Learning frameworks, such as TensorFlow, Keras or PyTorch. Several tasks are proposed:

• Task 1: extend CliMetLab with so that offers user with high-level Matplotlib-based plotting functions to produce graphs and plot which are relevant to weather and climate applications (e.g. plumes plots, ROC curves, …).

• Task 2: the Python package Intake is a lightweight set of tools for loading and sharing data in data science projects. Extend CliMetLab so that it seamlessly interfaces with Intake and allow users to access all intake-compatible datasets.

• Task 3: Xarray uses the data format Zarr to allow parallel read and parallel write. Convert large already available datasets to xarray-readable zarr format, define appropriate configuration (chunking/compression/other) according to domain use cases, develop tools to benchmark when used on a cloud-platform, compare to other formats (N5, GRIB, netCDF, geoTIFF, etc.).

Challenge 31- ECMWF forecast data in 3D with Blender

Stream 3 - Visualize weather, climate and atmosphere

Goal

Extend existing visualisation software to support ECMWF forecast data + examples

Mentors and skills

• Mentors: @StephanSiemen , @sylvielamythepaut
• Skills required:
  • Python
  • xarray
  • Blender
  • git + GitHub

Challenge description

For creating high quality visualisations of forecast data, Blender has proven a valuable tool. There has been various attempts to load GRIB data but they have not been sustainable and scalable. A new project started called BlenderNC to offer a Python add-on using xarray to load NetCDF data. This project should use the xarray extension cfgrib to extend BlenderNC to be able to read and visualise GRIB data. It is also expected that the documentation at https://blendernc.readthedocs.io will be updated with examples of using GRIB data. Interested candidates have a chance to work on an exciting Python project and learn how to handle weather model data and how to visualise it.

Challenge 15 - Development of satellite simulation web tool

Stream 1 - Software development for weather, climate and atmosphere

Goal

Provide a training and insight tool for the information provided by satellites

Mentors and skills

• Mentors: @EddyCMWF
• Skills required:
  • Web interface building
  • GUI experience
  • Basic script programming to link already existing software modules
  • Interest in satellites and how they are used for earth observation

Challenge description

Satellite observations are critical in modern weather forecasting systems and there is an urgent/ongoing need to train more scientists in their exploitation. Highly sophisticated space-borne sensors make incredibly accurate radiation measurements of emissions from the atmosphere, but the relationship between these measurements and the atmospheric quantities we want to understand (e.g. temperature, humidity and chemical composition) is complex.

Radiative transfer models (computer codes) exist which make this link and are embedded at the centre of our operational data assimilation and modelling systems.

This project would build a prototype web interface that would allow students, trainers and researchers to run these codes and visualise the outputs via a web interface. For example, seeing how a change in carbon dioxide in the atmosphere would influence the satellite measurements gives an insight into how best to use these measurements. Pollution events can be similarly visualised from a "satellite view" perspective.

Challenge 16- ECMWF User Profile Page

Stream 1 - Software development for weather, climate and atmosphere

Goal

Provide users with their own profile web page showing a summary of their relationship and records with ECMWF. Links from this web page will provide an improved user journey to the services and systems where those relationships and records are held and can be updated (if relevant, and only in some cases).

Mentors and skills

• Mentors: @sylvielamythepaut @kiden
• Skills required:
  • This project would suit a person who can analyse how to extract information from a broad range of technologies and platforms in order to present a 'mash-up' of user profile information from various platforms onto a single custom web page.
  • Competent in html, css, javascript (library?), apis, information security. Knowledge of ForgeRock may be useful.

Challenge description

Users currently have no overview of their profile with us; in an age of GDPR/PIIP this isn't very transparent of us. When we investigate issues we often need to look at a user's records in several systems to discover if the problem lies in an incorrect record of their relationship with us, e.g. wrong policy, wrong name, missing from a mailing list, lost data; this would provide an overview of a user in one place across all these systems.

The vision is a web page summary of the following user information:

• Registration information (name, email, pwd alert if near to needing changing, etc) - data in AD; link to forgerock
• Data from and links to profile pages on other platforms
• Favourite charts (link to chart dashboard or mars api requests - resurface what's shown in apps recent or recurring data requests
• Their current agreements and licences (from where?)
• Mailing lists (from csv files)
• Policies / roles (from AD)
• Jobs they've applied for (can we get this from JobTrain?)
• sbu units used (how can we get a summary?)
• Events they have attended (accessible in json over API)
• Learning modules they're doing or have completed (link to learning portal)
• Atlassian activity (likes, forum posts, page edits) - confluence API
• DHS usage (link to https://dlm.ecmwf.int/ )

Each of these summaries should then link to the system or platform where the information comes from and where more detail is held and can be updated (if/when these exist).

Out of curiosity, and I recognize I don't fully know what I'm talking about here, but don't any students out there find it a bit unfair that ECMWF wants to use open source software and student developers to aid the assessment of forecast data that they charge thousands of dollars for people to use unlike nearly every other meteo agency in the world with a global weather model? Are the students who participated in this getting paid? This seems like exploitation to me. Students who participated in this and believe in the power of open source: I beg you to pressure your overlords to walk the walk.

Challenge 13 - ORIGAMI (global river gauges mapping)

Stream 1 - Software development for weather, climate and atmosphere

Goal

Develop a software to facilitate the mapping of river gauges on the drainage network of a distributed hydrological model.

Mentors and skills

• Mentors: @ecCinziaMazzetti @zservin32 @colonesej
• Skills required:
  • Image analysis
  • Pattern recognition
  • Python

Challenge description

Data and software
We plan to use stations coordinates and drainage networks from CEMS (EFAS and GloFAS) and HTESSEL at different spatial resolutions. We plan to use OpenStreetMap to identify rivers and derive metadata.

What is the current problem?
River gauges are mapped on the hydrological/land surface model to the pixel that better represents the location of the station in the model’s drainage network. This procedure currently uses river gauges coordinates and drained area information from the station and the model. It tries to identify the model pixel with the closest drained area in the proximity of the station’s coordinates. This approach can lead to large mistakes because of uncertain/missing information on the station’s drained area or issues with the model drainage network, sometimes occurring hundreds of kilometres away from the location of the station. Difficulties in stations mapping increase at coarser model resolutions and with the complexity of the drainage network. An accurate mapping is often only possible through a manual procedure, but this can cause long delays in the adoption of a station for either model calibration or verification.

What could be the solution?
We are looking for a solution that won’t make use of the drained area information but will instead mimic the human mapping procedure. We would like to use image analysis and/or pattern recognition techniques to match the real river to the one in the model and then map the station on the correct model pixel, also exploiting additional metadata such as the station name or the river name. The mapping of each station should include a quality flag showing a confidence level in the mapping result.

Ideas for the implementation
We envisage that implementation will include the following steps:

• extract the river map for the area surrounding the station and the available metadata, such as rivers names, from OpenStreetMap or any other open dataset.
• match the river image to the image of the model drainage network
• map the station using coordinates and metadata (like the name of the river or the name of a nearby location).

Challenge 22- ML4Land

Stream 2 - Machine Learning for weather, climate and atmosphere applications

Goal

Improve understanding of land surface cover characteristics and how these map into reanalysis variables such as surface temperature, using climate reanalysis such as ERA5 and ad-hoc exploratory 1km simulations.

Mentors and skills

• Mentors: @dueben @gpbalsamo @joemcnorton
• Skills required:
  • Previous experience with high-resolution land surface image (Copernicus Sentinel-3 & similar Satellites platforms) related to physical properties of the land surface (Snow-cover, Vegetation-cover, Urban-cover) that can be spotted from satellite would be advantageous.
  • Knowledge of some Machine Learning software (PyTorch and similar) and Machine Learning tools would also be an advantage.

Challenge description

Improve understanding of land surface cover:

• How medium-resolution modelling products such as ERA5 (31km or 1/4 degree) compare with aggregate satellite images for snow, vegetation and urban cover?
• About snow cover at 1km: how models and EO satellite images data differ?
• About urban cover at 1km: how models and EO satellite images compare at 1km?

Challenge 14 - Atmospheric composition data comparison toolbox

Stream 1 - Software development for weather, climate and atmosphere

Goal

Develop a set of reusable tools (with or without an interactive user interface) which will allow simple ad-hoc comparison of different atmospheric composition datasets.
The tools should include:

• unit conversion
• side-by-side visual comparison
• regridding
• time and geographic data aggregation
• computing and visualising of georeferenced statistics

Mentors and skills

• Mentors: @miha-at-ecmwf @federicofierli1 @Antjeinness
• Skills required:
  • Good knowledge of Python and libraries to read, analyze and visualize atmospheric composition satellite and model data (numpy, eccodes, pandas, xarray, matplotlib)
  • Some knowledge of meteorological data formats (GRIB, NetCDF, HDF) and libraries to read and manipulate the data (ecCodes, netcdf, cdo, nco, ..)
  • Experience of either developing Jupyter notebooks or (even better) web-based dashboard applications

Note: Challenge is funded by Copernicus. Only nationals from the European Union and ECMWF Member States are eligible to apply (see Terms and Conditions).

Challenge description

What is the current problem/limitation?
Satellite and model atmospheric composition data are usually stored in different locations using different data formats with non-compatible metadata so it's not a trivial task to compare them.

What data/system do you plan to use?
We would like to start with a few observed and modelled total column products, for example comparing GOME-2 ozone total column product (https://acsaf.org/offline_access.php) with ozone total column analyses from CAMS reanalysis (https://ads.atmosphere.copernicus.eu/cdsapp#!/dataset/cams-global-reanalysis-eac4) and the ozone monthly gridded dataset (https://cds.climate.copernicus.eu/cdsapp#!/dataset/satellite-ozone-v1).

We would like to use the existing ECMWF, EUMETSAT and Copernicus infrastructure (Wekeo, Climate Data Store, Atmosphere Data Store, European Weather Cloud) as much as possible.

What could be the solution?
Many Python code examples already exists (1, 2) which demonstrate how to retrieve, read and understand each of the data products and convert them into a similar target format, like xarray Dataset object.

Some of the tasks that could be done under the project:

• making an inventory of the relevant sources of existing code
• review and consolidate the existing code and perform a gap analysis
• based on the findings develop the missing tools
• publish the tools as either well-documented collection of Jupyter notebooks or use them as a backend to a web application
• the tools and the application should not be limited to a specific platform

[1] https://gitlab.eumetsat.int/eumetlab/atmosphere/atmosphere
[2] https://github.com/CopernicusAtmosphere/jupyter-notebooks

This sketch is just one option. We welcome any other innovative proposals to address the stated challenge.

Challenge 12 - Size, precision, speed - pick two: implementation

Stream 1 - Software development for weather, climate and atmosphere

Goal

This project is a follow-up of the ESoWC 2020 data encoding optimisation challenge.
Based on the results and the findings of the completed project we will implement improved data packing configuration in our production streams. We would also like to analyze some new atmospheric composition and meteorological datasets.

Mentors and skills

• Mentors: @miha-at-ecmwf @juanjodd
• Skills required:
  • Some knowledge of meteorological data formats (GRIB, NetCDF) and libraries to decode and manipulate them (ecCodes, netcdf, cdo, nco, ..)
  • Some knowledge about data encoding (data packing, accuracy, compression methods)
  • Knowledge of a software library to compute and present the results
  • Some familiarity with Chemical Transport Modelling (CTM) or Numerical Weather Prediction (NWP) to be able to better appreciate this challenge would be beneficial

Note: Challenge is funded by Copernicus. Only nationals from the European Union and ECMWF Member States are eligible to apply (see Terms and Conditions).

Challenge description

Data and software
We plan to use the CAMS global real-time forecast dataset, ecCodes and NetCDF libraries to test different configurations and estimate data encoding errors and software library to compute and present results (Python, R or Julia).

What is the current problem?
Due to non-optimal data encoding configuration, there is a lot of artificial precision in our data. Datasets are expensive to archive and move and difficult to use.

What could be the solution?
We would like to remove artificial precision from the encoded fields without any loss of information. At the same time, we need to be conscious of operational constraints, so data encoding and decoding steps do not become prohibitively expensive. The desired solution would be a combination of data encoding settings and step to achieve this goal.

Ideas for the implementation
Things to address: more appropriate packing methods, encoding float arrays, explore usage of suitable data compression algorithms.

Challenge 23- Mapping Emissions of Air Pollutants

Stream 2 - Machine Learning for weather, climate and atmosphere applications

Goal

Derive suitable proxies for spatial and temporal mapping of emissions.

Mentors and skills

• Mentors: @annaagustipanareda @gpbalsamo @joemcnorton @mparrington
• Skills required:
  • Experience of use with big data
  • Experience of Machine Learning techniques

Note: Challenge is funded by Copernicus. Only nationals from the European Union and ECMWF Member States are eligible to apply (see Terms and Conditions).

Challenge description

What data/system do you plan to use?
We plan to use :

• Either estimates of NOx emissions (based on atmospheric inversions) or direct concentration observations from satellites.
• 3D atmospheric fields from the ECMWF model of variables relevant to transport and chemical loss (e.g. temperature and wind).
• Any proxy data that can be linked directly to emissions (e.g. nightlight data, traffic data, etc…).

What is the current problem/limitation?
To model and forecast emissions of chemical tracers in the atmosphere, a suitable estimate of emissions is required. Emission estimates from inventories are often either fixed in time or vary on a long-timescale (e.g. monthly/yearly). This fails to capture the true variability in emissions due to changes in activities (e.g. rush hour). Where proxy data are used, they are often either out of date or do not offer suitable variability.

What could be the solution?
The underlying processes of emissions are wide-ranging, including differing fuel types, activity types, or even social changes in human behaviour. Several proxy datasets can be used to improve estimates of emissions and also include variables which could be optimised within an emissions model. For example, having a map of population density is likely to correlate well with emission sources. An aim of this project would be to offer suitable options for which proxy data should be used for estimating emissions.

Ideas for the implementation
The system could follow the design ideas of existing fossil fuel data assimilation systems (FFDAS), but should explore novel avenues for estimating emissions by identifying datasets which correlate well with emissions. Observation data could include NO2 observations from Sentinel-5p or inversion estimates; alternatively existing inventories could be used with spatial and interannual variability. The input proxies are open to many possibilities but example start points might include population density maps, nightlight data and TomTom traffic data.