njcoast / cyberspatial Goto Github PK

https://mapproxy.org/

Disable Data and Maps dropdowns, with the requirement that when the user is logged in he/she sees both dropdowns, otherwise only About is visible (check bootstrap if it supports that).

Configuration of Geoserver endpoint to accept vector tiles. There is a vector tiles tutorial in the geoserver documentation.

We need to have an understanding how the GIS components scale with size.

Configure Kubernetes secrets to correctly deploy Django Keys and other sensitive data.

Scripts to automatically generate documentation from embedded code comments using Sphinx.

The tile workflow need to be extensible to generate 3D tiles using one of the emerging standards at the same time we generate the contours. This could potentially be used with 3D terrain mapping in future version of the model. One emerging specification is the Cesium Tile specification that has been submitted to a OGC working group.

Complete documentation with focus on system architecture and how scaling can be accomplished. Use diagram in the template.

https://docs.google.com/document/d/1KeVk4gw1cDi9TqTS5528Ty0bC0C2cekiQKByuRf-vc8/edit

Configure 3 Ubuntu VM's with one as public facing master and two slaves.
These will be used to have the website online and also as general development setup.

hostnames are:

• njcoast1.crc.nd.edu (public IP)
• njcoast2.crc.nd.edu (private IP)
• njcoast3.crc.nd.edu (private IP)

specs:
Ubuntu 16.04 LTS with 2 cores, 2 GB RAM and 25 GB disk

Only two rules have been added: one allowing SSH from anywhere (still restricted in /etc/hosts.allow though) and another allowing access from our backup server for the nightly TiBS tape backups.

Login with ND netID and passwords.
Chris and James have sudo.

Django models need to be added to GeoNode to support Role based permissions in data access. These role data models would also support search (discovery) of datasets based upon rules associated with each role that specifies data type preferences.

Replicating Maploom functionality to GeoNode, look at Leaflet Draw: https://leaflet.github.io/Leaflet.draw/docs/leaflet-draw-latest.html

Kristina provided the CSS for the user dashboard. CSS and custom graphic files attached.
Mockup:

https://app.zenhub.com/files/88267239/b00ca30b-43aa-43ad-828d-ffbe88bf22c5/download

Refactoring python model code to generate GeoJSON vector tiles from matplotlib contours. There is a utility code that can generate contours called geojsoncontour.

Relevant links are:
Change to generating vector tile sets:
Mapbox tiles

Generate GeoJSON contours from matplotlib,
Matplotlib Library

[Matplotlib basemap utility)[https://github.com/matplotlib/basemap]

Use Vector Tiles on GeoServer

Geoserver python configuration library.

Mapbox geojson vector tile server

Youtube talk that explains vector tile implementation in GeoServer.

Blog post on terrain models and vector tile layers

Stable cyberinfrastructure fully containerized and ready for population.

Architecture:

• Google cloud container engine (GKE) or AWS Kubernetes
• location: US central
• Nodes: 3 virtual machines which run the containers
• 5 containers: Django, Geoserver (Java-based), PostGIS server, Rabbitmq, ElasticSearch
• All the containers can see each other
• External accessibly are Django and Geoserver via Nginx proxy

Software stack:

• Django container runs a Gunicorn Python tool (web server)
• GeoServer is a Java web app and has Catalina (Tomcat - web server) in front of it
• PostgreSQL with PostGIS
• RabbitMQ for message passing
• ElasticSearch (unknown for what it's being used right now)
• Kubernetes to manage containers and scheduling
• Helm to manage the Kubernetes via templates

Database footprint:

• PostgreSQL container and can only be accessed from other containers within the kubernetes cluster. v9.6 incl. PostGIS additions
• Image: mdillon/postgis
• imageTag: 9.6-alpine
• there is dev and prod DB (separate from each other)

being used for API and user documentation.

https://www.gitbook.com/

Configuration of persistent disk storage snapshots for Google Cloud infrastructure. Insure that instances of GeoNode and Geoserver can be live migrated.

As a user
I want to use the map with various tools to manipulate, share and store map/layer data.

We are not concerned about roles for now, just note that there are 3 different roles for which we need to adopt a permission schema to control features. Right now, we just want to have a user being able to use the site.

Mockup: http://www.crc.nd.edu/~kfurse/nj-coast_front-mock/map-p.html

AC:

• Display of base map, centered at user’s specified municipality’s coordinates
• GIS layers available (for this role) listed adjacent to map
• Layers are grouped by category
• users can click the layers they wish to toggle on and off
• (List of available GIS layers should load dynamically based on the intersection of role and designated municipality (there are GIS layers that are available statewide and others that may be specific to a municipality)). will come later with role schema
• users should be able to share maps with other users
• users can annotate maps (James is exploring options - Leaflet), refer to him for more details)
• users can share map annotations with others
• Users should be able to mouse over hazard intensity visualization to see surge height or wind speed at a given location (needs to be specified in GeoJSON model output)
• User has access to Storm hazard prediction tool to run a custom scenario to be made available as a new GIS layer in their dashboard. the SHP tool will be the next piece of work

Configure Kubernetes to deploy the Docker-SNORT image configuration. This will also necessitate including the proper ssl certificates for njcoast.us.

Celery tasks to trigger ElasticSearch indexes of GeoNode metadata don't appear to be configured correctly. ElasticSearch container need to be added to Kubernetes deployment scripts.

As a PO
I want to have a realistic view of all tasks
so I can schedule additional resources, track the budget and over time task progression.

Tim and Swapna are conducting following scans:

11/21 nmap
11/28 ZAP (Admin account needed)
12/4 Code review
? Snort

Extend secrets persistent volume to include Django Environment variables and the secret deployment key.

Geoserver allows persistent base layers to be stored in the local_settings.py and automatically loaded into new instances. ArcGIS layers can also be added in this way. See the Geonode mailing list for a discussion. A sample base layer is:

Add ESR world ocean reference base layer to geoexplorer

source: new ol.source.XYZ({
url: 'http://server.arcgisonline.com/ArcGIS/rest/services/' +
'Ocean/World_Ocean_Reference/MapServer/tile/{z}/{y}/{x}'
})
What to change in production:
http://docs.geonode.org/en/dev/deploy/production.html
How to add XYZ:
GeoNode/geonode#1286
{
"source": {"ptype": "gxp_olsource"},
"type":"OpenLayers.Layer.XYZ",
"args":[
"hot", ["http://a.tile.openstreetmap.fr/hot/${z}/${x}/${y}.png", "http://b.tile.openstreetmap.fr/hot/${z}/${x}/${y}.png", "http://c.tile.openstreetmap.fr/hot/${z}/${x}/${y}.png"], {"transitionEffect": "resize","attribution": "osm_attribution"}],
"visibility": True,
"fixed": True,
"group":"background"
},

Kristina provided the CSS for the landing page. CSS and custom graphic files attached.
http://www.crc.nd.edu/~kfurse/nj-coast_front-mock/about.html
https://app.zenhub.com/files/88267239/d28fd98b-ac83-4a0f-a662-5b4acb455e64/downloadhttps://app.zenhub.com/files/88267239/5feb9a05-7215-4b40-ac03-1e5e470d87b8/download

Comment out People and Groups in the About dropdown and have the text as single page.

Notes from Kristina:

1. I added to the css, as well as modifying some of the classes I defined for the homepage. Had to make a couple of changes to the homepage html. I added a “watermark” class and assigned it to the NJcoast watermark on the homepage top gray box (). This stopped my css from repositioning any image that has .emphasis as a parent. I also linked Storm Prediction Tool near the bottom of the page to an anchor on the About page. I think that’s it.
2. Again, my html is broken (dropdowns don’t work, menu doesn’t collapse properly on small screen widths, etc.). Just a warning it can’t be pulled verbatim.

http://docs.geonode.org/en/master/tutorials/advanced/geonode_production/production.html

Minicube fails to mount the host directories specified in the Kubernetes configuration files for minikube. This prevents both postgresql and geoserver persistent data volumes from have the correct data structure when the pods start. It is unknown if this issue exists with the GKE persistent volumes.

It may be possible to use Kubernetes for primary job management of computational models. We don't want extra resources that are not needed, but to scale as necessary.

Migrate Docker Gateway image for nginx configuration in kubenetes deployment.

NGINX/Gunicorn/Django is not correctly routing the Geoserver endpoint to njcoast.us/geoserver and njcoast.us/geoserver/web.

Remove links for all 3 planer buttons on the frontpage on dev server.

1. Update production with the latest code from dev.
2. Commit code to Github

As a dev
I want to set up kubernetes in AWS
so that I can bring njcoast.us online.

Check out AWS with Kubernetes to replace GKE.
A decision needs to be made for whether we go with the ND account or create our own.

Setup environment for Django development. Please talk to James about the required stack and existing dependencies.
Also he needs to show where to commit your code (repo/branch).

As a user
I want to go through a tutorial/training
so that I get a better understanding of the tools and different features.

AC
1.How will the tutorial be triggered/ accessed from?
2.What is the scope of the tutorial?
2.1.Account creation? explaining the 3 different roles
2.2.Maps/Layers/Annotation tools/ Share/Create a Map,Layer/Download
2.3.Storm Hazard Projection Tool

Auto-deployment testing scripts for TravisCI or for Jenkins CI test servers.

Check why the map preview icon for the test map is not being displayed.
http://njcoast.us/maps/?limit=100&offset=0

Create a new project plan for the SLA in Ominplan with an updated timeline of all Milestones M4 to M8.

We need to setup a build server. Investigate Travis

https://blog.travis-ci.com/
http://thylong.com/ci/2016/deploying-from-travis-to-gce/

Some shape files are quite large. Receiving a "413 Request Entity too large" from NGINX when uploading a shape file via the layer upload system. This can be configured through the nginx.conf file used to build the container.

https://www.cyberciti.biz/faq/linux-unix-bsd-nginx-413-request-entity-too-large/

Kristina provided the CSS for the landing page. CSS and custom graphic files attached.
Mockup: http://www.crc.nd.edu/~kfurse/nj-coast_front-mock/

https://app.zenhub.com/files/88267239/aa79619a-aeb5-4164-9b99-67537d5872ad/download
https://app.zenhub.com/files/88267239/23132582-9fc2-4093-a301-31e472c9a88a/download

We want a dummy template in the repo but actual deployment information secure in the CRC redmine instance.

Explore roles EM, Planner and First responders what layers they can access.

https://docs.google.com/spreadsheets/d/1cSIux0LqNdfX63E37PuK44a3FjKaYc61WuKT1MbnbRo/edit#gid=1459124170

Duplicate: https://app.zenhub.com/workspace/o/njcoast/cyberspatial/issues/123

Setup endpoint for authentication for T&M Map layers.

Design of Restful components for bot the Job management and individual computational models. We need to consider the extra metadata necessary to describe the model result's response in known controlled vocabularies. The Earthcube xdomes.org is probably a good starting point for building out the controlled vocabularies.

Addition of map editing and annotation features to GeoNode instances. Basis for code could be Maploom or the Mapstory project that has a GitHub Organization.

We need to fill out the following doc:
https://drive.google.com/drive/u/0/folders/0B9fGRYX4RNaSTE1XREpSZU1QS2s?ths=true

https://docs.google.com/document/d/1wfZzasAtbup8SxpOEf0gi2TeOtAt1xGObQiN9UivwxI/edit