Management Project for coordinating the Climate Service co-creation process of the CLARTIY CSIS. High-level tasks (issues) are defined in this repository and then broken down into concrete implementable issues in different sub repositories.
Home Page: https://csis.myclimateservice.eu/
Hi! We've tried (@luis-meteogrid) creating a user account using this form:
https://csis.myclimateservice.eu/user/register
is this still in use? We received an email saying that it was under review.
Thanks!
On Climate twins adaptation
The initial climate twins paper as published in 2011 describes the climate twins application “being designed to communicate climate changes in an intuitive and understandable way by showing regions which have now similar climate conditions as a given Point of Interest (POI) in the future to learn from adaptation options”.
Making use of the climate twins concept for CLARITY focusing on climate services objectives dealing with climate risk (=extreme events) / vulnerability and adaptation requires to adapt the research approach not finding similarities in average climate signals with respect to daily temperature maximum distribution and daily precipitation volume patterns for selected decades towards extreme event occurrence. Thus the appropriate objective must be finding similarities in extreme events to learn from cities which have experienced similar events earlier and frequently to cope better with climate risk, to better adapt to climate change and to better mitigate vulnerability.
The CLARITY concept shall focus on extracting extreme event occurrence patters (n of heat waves of certain length, n of summer days or tropical nights during prior decades and years) and extracting the same indicators for extreme events screening future climate data at the European scale, from appropriate climate simulations (in our case from Euro-Cordex ensembles).
By identifying cities or regions with a certain number of prior extreme events of critical magnitude and extracting the expectations on extreme events for the own city allows to review the activities of those critically exposed cities which show a similar environmental and spatial framework (characteristics of terrain and land use) in earlier years or decades.
By reviewing the damage repair and adaptation activities in cities with similar earlier experience, each target city can learn from activities in those earlier exposed cities to get better prepared and to cope better with future climate hazards.
Besides generating maps of extreme event patterns for prior and future decades cities can mark their location and provide a link where their adaption measures have successfully responded to such extreme events. In this way those cities can actively help other cities, facing future climate hazards, in learning from their earlier experience.
Due to our taxonomy design (e.g. hazards and elements at risk), only a subset of the "relevant" taxonomy terms should be shown to the users when choosing e.g. hazards or elements at risk. This issue is illustrated on example of hazards taxonomy below:
Currently, the user can only choose "heat", not "temperature", "extreme-heat" or "hot-days-75p" for filling in certain fields. For example in "Adaptation Options Effects" bundle.
Proposed best practice for assuring that users can only use the chosen terms from this taxonomy when adding/editing entities AND also only choose these terms in view filters is described in following posts.
This task is responsible for defining the data structure of the Adaptation Options resource that will be included in CLARITY data package. This includes indicating:
how is the resource encoded (e.g., store the information in raster or vector format). Besides, in the case of using vector based data, a set of "standard" attributes must be defined for containing the information, so that all resources of this type have always the same set of attributes (and names) and CLARITY tools can process them in the same manner.
what formats are supported for this specific type of resource (e.g., for raster data: netcdf, geotiff, etc.; for vector data: geojson, shapefile, etc.)
what metadata attributes are used to describe this resource (refer to CLARITY General Data Package specification and https://frictionlessdata.io/specs/data-resource/).
The definition of the Adaptation Options resource will be based on EMIKAT
In this task the following partners MUST be involved: AIT
This task is responsible for defining the data structure of the Risk Map(s) that will be included in CLARITY data package. This includes indicating:
how is the resource encoded (e.g., store the information in raster or vector format). Besides, in the case of using vector based data, a set of "standard" attributes must be defined for containing the information, so that all resources of this type have always the same set of attributes (and names) and CLARITY tools can process them in the same manner.
what formats are supported for this specific type of resource (e.g., for raster data: netcdf, geotiff, etc.; for vector data: geojson, shapefile, etc.)
what metadata attributes are used to describe this resource (refer to CLARITY General Data Package specification and https://frictionlessdata.io/specs/data-resource/).
In this task the following partners MUST be involved: PLINIVS
This task must be done by PLINIVS
This task is responsible for defining the data structure of the Vulnerability Map(s) that will be included in CLARITY data package. This includes indicating:
how is the resource encoded (e.g., store the information in raster or vector format). Besides, in the case of using vector based data, a set of "standard" attributes must be defined for containing the information, so that all resources of this type have always the same set of attributes (and names) and CLARITY tools can process them in the same manner.
what formats are supported for this specific type of resource (e.g., for raster data: netcdf, geotiff, etc.; for vector data: geojson, shapefile, etc.)
what metadata attributes are used to describe this resource (refer to CLARITY General Data Package specification and https://frictionlessdata.io/specs/data-resource/).
In this task the following partners MUST be involved: METEOGRID/PLINIVS/ZAMG
GUI / Tagging functions:
• Mark areas /cities with an extreme event marker (heavy precipitation, extreme heat)
• provide a link where information on the local event is stored:
GUI / Viewing functions:
View areas with threshold x above a number of extreme event days:
• average annual number of summer days
• average annual number of tropical nights
• average annual number of heat episodes > 3 day, >5 days, > 10 days
• average annual number of frost days
• average annual number of frost episodes > 3 day, >5 days, > 10 days
• average annual number of storm days
GUI / Retrieval functions
(1) Pre-selection of events by event type above a threshold (frequency, magnitude
(2) Spatial selection of the pre-selected events by drawing a polygon or a rectangle
(3) Printing a map of the selection (pdf)
(4) Point on the event markers to search for links were hazards and adaptation measures are documented - or
(5) list the links provided in the selection for reading or printing (pdf)
@patrickkaleta , the following Drupal 8 modules need to be installed in order to use the ingest JSON data packages into our drupal data model:
Migrate (in core)
Migrate Plus 8.x-2.x
Migrate Tools 8.x-2.x
Migrate Source JSON 8.x-1.x (currently, requires this patch for Drupal 8.1.x+ compatibility)
(source: https://www.jeffgeerling.com/blog/2016/migrate-custom-json-feed-drupal-8-migrate-source-json)
Implement the different Table Components for HC, EE, VA, ... as React Web Applications:
The following 4-steps approach, which can also be applied to the development of the Map Component or any other external AJAX Application that is integrated into the main Drupal site, has to be followed:
In order to perform automated screening studies, the complete modelling workflow should IMHO be implemented as follows:
Ideally all i/o data we need for HC+LE, Impact & adaptation calculation is in one database (or a cluster). But ATM only for Impact Calculation a "database" (AIT EMIKAT) is used. So either implement the whole process in EMIKAT (which cannot be done by AIT personnel only), or perform / store HC+LE, Impact & adaptation calculation in a PostGIS database and let EMIKAT access these data via Postgres REST API or Geoserver/WCS.
pre-calculate (all?) Hazard Events (HeatWaveOccurence Matrixes, 12x12 km raster) for Europe and put them into a PostGIS database.
this process should be semi-automated (= a script). we have to re-calculate when bias corrected EU-CORDEX data is available is available, right? The (R) scripts should be made available on github, if possible (open science and such ...). I don't think that size matters here, Netdcf files are less than 3 MB. Apart form the re-calculation on bias corrected data, this process has to be performed only once, the result is stored permanently in the database and used for HC+LE downscaling.
Pre-calculate (important: as 500x500 (?) grids, not features) the Local Effects Input Layers (those derived by Mario from pan-European datasets like UrbanAtlas) for Europe / major European cities and put them into a PostGIS database.
the process is semi-automated. Feature extraction scripts are on github. So doing the calculation for whole Europe / major European cities shouldn't be a problem. Re-calcualtion is only needed when e.g. Urban Atlas, etc. data is updated. So the results are permanently stored in the database and used for HC+LE downscaling and adaptation.
However, size does matter, urban_atlas_shp.zip is > 40 GB. And processing power (CPU, RAM, ..) might be a limiting factor, too. But in theory, we just need to precaculate and store the analysis grid (500x500m) for while Europe / supported major cities or regions, right? We don't need single features (buildings, roads, ...) for HC+LE and adaptation calculation, right? Those feature layers are nice for to have for visualisation (but OSM should be good enough), but not needed for impact calculation.
Does precalculation make sense? Possibly for predefined regions. Currently the process is manual (arcGIS) but has to be automated. A script, or even better a stored procedure (e.g. using postgis GDAL functions) has to be developed. Moreover, the re-calculation must be supported on-the-fly in real-time on a limited number of grid cells taking as parameter the user-defined study area (or even better a predefined region -> then we can reuse all materialised views for the regions, of course without adaptation options applied) and possibly the hazard event ids (events selected by the user or preselected in the Data Package), the RCP and the time-period. Then publish materialized views as layers on geoserver, EMKIKAT can query them via WCS. This should be fast, especially, if stored database procedures are used to do the HC-LE calculation (materialised views are cached!). Then expose a REST (e.g. http://postgrest.org ) or GraphQL (e.g. https://github.com/graphile/postgraphile) API for e.g. the Table Component on top of those views, generic standard sql queries (aggregate functions, etc.) can be used to create generic SQL queries that feed the API.
Currently managed and calculated in Emikat. Does precalculation make sense? IMHO not. -> see adaptation options.
Adaptation Options modify properties of the Local Effects Input Layer and are applied on a single analysis grid cell, not a single feature (building, roads, ....), right? If cells are in a database, simply make a spatial select (user-defined study area polygon or better: predefined region) from Local Effects Input Layers and apply +/- to properties (e.g. albedo+=0.5) and store in a materialized view. If we decide to support only pre-defined regions (e.g. for major European cities like metropolitan region of Naples) instead of arbitrary user-defined study areas, we can even re-use the Local Effects Hazard materialised views for the respective region and create an 'adapted' materialised view on top of it. Storage might become a problem (if the system is used) But we can clean those temporary adapted views after a certain time period (e.g. 14 days), unless the user has bought a subscription (= if you don't pay for it, studies are deleted after 14 days).
Then re-calculate Local Effects Hazard Layers for the selected study area using a script or even better the stored database procedure that makes use of the 'adapted' (those with adaptation options applied!) LE Input materialized views . If the user makes a preselection for one RCP, one timeperiod or even a few events (e.g. just low and high) then there is even less to re-caculate. All of this should IMHO be possible in real-time if all i/o happens in the same database instance. Then publish HC+LE materialized views as layers on geoserver, EMKIKAT can query them via WCS.
We need some place where we can easily list and find all service endpoints instead of having to review tons of emails/issues for finding the service url we need everytime.
Indicators:
Extreme events by decade 2051-2060, 2071-2080, 2091-2100:
• decadal average of the annual number of summer days (>25°C minimum temperature)
• decadal average of the annual number of tropical nights (> 20°C minimum temperature)
• decadal average of the annual number of heat episodes > 3 day, >5 days, > 10 days with maxima >25°C)
• decadal average of the number of heat days (> 25°C)
• decadal average of the annual number of frost days (< 0°C maximum temperature)
• decadal average of the annual number of frost episodes > 3 day, >5 days, > 10 days with maxima <0°C)
• decadal average of the annual number of storm days (maximum windspeed > y m/sek.))
(indicators shall be confirmed with the climate experts.
Setup and configure Drupal Docker Container as basis for CSIS.
Related source code repository of Building Block "UI Integration Platform": docker-drupal
This task is responsible for defining the data structure of the Hazard Map(s) that will be included in CLARITY data package. This includes indicating:
In this task the following partners MUST be involved: METEOGRID/PLINIVS/ZAMG
Hi Pascal,
I think it was you who created Mario's credentials for AIT server. We would need also for @DanielRodera and @maesbri
This task is responsible for defining the data structure of the Impact Map(s) that will be included in CLARITY data package. This includes indicating:
how is the resource encoded (e.g., store the information in raster or vector format). Besides, in the case of using vector based data, a set of "standard" attributes must be defined for containing the information, so that all resources of this type have always the same set of attributes (and names) and CLARITY tools can process them in the same manner.
what formats are supported for this specific type of resource (e.g., for raster data: netcdf, geotiff, etc.; for vector data: geojson, shapefile, etc.)
what metadata attributes are used to describe this resource (refer to CLARITY General Data Package specification and https://frictionlessdata.io/specs/data-resource/).
In this task the following partners MUST be involved: PLINIVS
we need some sort of loook-up table for determining for each area in Europe the event-probability (maybe implemented as REST service).
-->action: ZAMG will take care of generating it --> For each grid point we have one frequent event, occasional and rare event (this is done for the baseline scenario + the 3 rcps scenarios (and for each rcp there are 3 future scenarios))
--> the idea is to keep it as possible for the screening phase. More complex solution for selecting/applying the hazard events to the study area could be part of the definition of a future business model (e.g., provide better methods of calculation in a pay mode)
Setup and configure CKAN Docker Container.
Related Project: ckan
Implementation of the Modelling Workflow for Hazard Local Effects with help of
The results are stored in the respective data- / meta-data repositories of the CSIS. See also Configure the CSIS with some datasets
The implementation has to be documented in this wiki in the CSIS Architecture.
Documentation of the Implementation of the Modelling Workflow in the Data Package Wiki the CSIS Architecture. See also clarity-h2020/csis-architecture#4
The documentation should give answer to the following questions
@negroscuro @ghilbrae @stefanon
The documentation will serve as main input for D4.3 "Technology Support Report" and will contribute to the documentation of the Emergent Architecture (see D4.2 CLARITY CSIS Architecture v1.0) of the CSIS. Thus, we should be able to create an architecture diagram that looks similar to this one (Example from the CRISMA Project):
We need a private git repository for storing and sharing among partners the drupal configuration files.
Implementation of the model defined in #21.
This task is linked to issue #9
Need a data package with
The meta-information has to be published as a data package.
The data has to be published on relevant services (grid maps for hazard & population)
This tasks relates to the creation of the "Catalogue of local data sources and sample datasets" (D2.2). It depends on the following tasks:
This task is connected to
We should integrate maps and data from EEA's Urban Adaptation Map Viewer into the CSIS.
We could show some of their Urban-Adaptation-datasets on our maps and embed the city profiles as external application via iFrame.
I don't expect that this will save us some work regarding local effects data processing and impact calculation, but it will enhance the CSIS with a lot of useful information, especially the city profiles:
WDYT?
But we need a formal / service level agreement to use their data and APIs (WMS, WFS, etc.). Denis?
Please check if we want to use Rasdaman (or Geoserver / GeoNode) for storing the results of the Emikat (e.g. raster hazard, exposure, impact) and update the documentation of the modelling workflow accordingly.
This tasks partially depends upon
On Climate twins adaptation
The initial climate twins paper as published in 2011 describes the climate twins application “being designed to communicate climate changes in an intuitive and understandable way by showing regions which have now similar climate conditions as a given Point of Interest (POI) in the future to learn from adaptation options”.
Making use of the climate twins concept for CLARITY focusing on climate services objectives dealing with climate risk (=extreme events) / vulnerability and adaptation requires to adapt the research approach not finding similarities in average climate signals with respect to daily temperature maximum distribution and daily precipitation volume patterns for selected decades towards extreme event occurrence. Thus the appropriate objective must be finding similarities in extreme events to learn from cities which have experienced similar events earlier and frequently to cope better with climate risk, to better adapt to climate change and to better mitigate vulnerability.
The CLARITY concept shall focus on extracting extreme event occurrence patters (n of heat waves of certain length, n of summer days or tropical nights during prior decades and years) and extracting the same indicators for extreme events screening future climate data at the European scale, from appropriate climate simulations (in our case from Euro-Cordex ensembles).
By identifying cities or regions with a certain number of prior extreme events of critical magnitude and extracting the expectations on extreme events for the own city allows to review the activities of those critically exposed cities which show a similar environmental and spatial framework (characteristics of terrain and land use) in earlier years or decades.
By reviewing the damage repair and adaptation activities in cities with similar earlier experience, each target city can learn from activities in those earlier exposed cities to get better prepared and to cope better with future climate hazards.
Besides generating maps of extreme event patterns for prior and future decades cities can mark their location and provide a link where their adaption measures have successfully responded to such extreme events. In this way those cities can actively help other cities, facing future climate hazards, in learning from their earlier experience.
When trying to upload a shapefile through the GeoNode interface , I get this error:
401,
HTTP Status 401 – Unauthorized
Type Status Report
Message No AuthenticationProvider found for org.springframework.security.authentication.UsernamePasswordAuthenticationToken
Description The request has not been applied because it lacks valid authentication credentials for the target resource.
Apache Tomcat/9.0.10
Might be solveable like this: GeoNode/geonode#3175
Could you have a look? Thanks!
Setup docker infrastructure and nginx reverse proxy with TLS on csis.clarity-h2020.eu.
Related project: docker-compose-letsencrypt-nginx-proxy-companion
In order to make the re-projection, we'd like to know more about the output GRID.
We already know that it must be EPSG 3035 but we also need the reference raster.
I guess we'd also need a reference raster for Europe, one for Naples, and, in the future, one for each DC and probably, one for each study area we'd like to cover. In this case, who will be defining this reference raster? (Is this specified in the Data Package? Is it created when the user selects an area? Any other option?)
Hi! We are stating with the DC4 Data Package but our(my) user does not have permissions to edito or create one. The user we are using at Meteogrid rn is the one I have (username: angela). Could someone give this account permissions to do that?
Thanks!!
Register (sub) domain names for myclimateservice.eu and point DNS entries to IP of csis.clarity-h2020.eu
Background data
• open street map content (source: OSM)
• satellite image (source: Google)
• digital elevation map / terrain shading (source: EEA)
• NUTS 1 and NUTS3 region borders (source: Eurogeographics)
• Population distribution map (source: EU pop grid, Eurostat)
• selected event indicator(s) as choropleth maps
(source: extracted from EUROCORDEX model runs.
This task will provide a comprehensive example of CLARITY Data Package with all possible resources it can contain.
In this task the following partners MUST be involved: METEOGRID/PLINIVS/ZAMG/ATOS/AIT
This task is broken down into the the following sub-tasks:
CSIS Developers need to create an account at https://csis.myclimateservice.eu/user/register
Depends on
This task involves identifying and collecting information about the different datasets necessary in the project and more particularly the DCs.
The task must also clarify which datasets are private/public and therefore can be made publicly available via CSIS platform for other users (or prevent their access)
This task is led by METEOGRID but REQUIRES contributions from PLINIVS, AIT, SMHI, ZAMG and AEMET
Setup docker services for OPeNDAP server Thredds & ERDDAP.
This task provides the general aspects and details of CLARITY data package specification (taking as basis frictionless data package specifications: https://frictionlessdata.io/specs/). Further details on the specific attributes for specialized resources contained in the data package are provided in #14, #15, #16, #17, #18, #19
@fgeyer16 , @patrickkaleta I need to create a "references" property (in the dp_resource object) which is an array of unlimited number of subarrays (which have a lenght = 3), any idea on how can I model this in Drupal?
For the moment it would be fine if we cannot limit the length of the sub-arrays. The content of the subarrays is just text.
See below an example with 2 sub-arrays:
"references": [
["@mapview:ogc:wms", "#/path", "https://clarity.meteogrid.com/geoserver/Local_Effects/wms?service=WMS&version=1.1.0&request=GetMap&layers=Local_Effects:LE_Tmrt_FrequentEvent_1971-2000_observations_Naples&bbox=4647500,1947000,4720500,2008000&width=725&height=768&srs=EPSG:3035&format=image/png"],
["@resource:hazard-event:heat-wave-duration:$temperature=28:$duration=6", "#/resources/[name='hazard_event-HW.6d_28C']"]
]
This task is responsible for defining the data structure of the Exposure Map(s) that will be included in CLARITY data package. This includes indicating:
how is the resource encoded (e.g., store the information in raster or vector format). Besides, in the case of using vector based data, a set of "standard" attributes must be defined for containing the information, so that all resources of this type have always the same set of attributes (and names) and CLARITY tools can process them in the same manner.
what formats are supported for this specific type of resource (e.g., for raster data: netcdf, geotiff, etc.; for vector data: geojson, shapefile, etc.)
what metadata attributes are used to describe this resource (refer to CLARITY General Data Package specification and https://frictionlessdata.io/specs/data-resource/).
In this task the following partners MUST be involved: METEOGRID/PLINIVS/ZAMG
Setup and configure GeoNode Docker Container.
Related project: docker-geonode
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.
