Giter Club home page Giter Club logo

odm2 / odm2datasharingportal Goto Github PK

View Code? Open in Web Editor NEW
29.0 20.0 7.0 21.24 MB

A Python-Django web application enabling users to upload, share, and display data from their environmental monitoring sites via the app's ODM2 database. Data can either be automatically streamed from Internet of Things (IoT) devices, manually uploaded via CSV files, or manually entered into forms.

License: BSD 3-Clause "New" or "Revised" License

Python 50.35% CSS 3.63% JavaScript 13.69% HTML 32.33%
odm2 envirodiy citizen-science

odm2datasharingportal's Introduction

ODM2

The next version of the Observations Data Model.

For more information about the ODM2 development project, visit the wiki.

Have a look at the ODM2 paper in Environmental Modelling & Software. It's open access!

Horsburgh, J. S., Aufdenkampe, A. K., Mayorga, E., Lehnert, K. A., Hsu, L., Song, L., Spackman Jones, A., Damiano, S. G., Tarboton, D. G., Valentine, D., Zaslavsky, I., Whitenack, T. (2016). Observations Data Model 2: A community information model for spatially discrete Earth observations, Environmental Modelling & Software, 79, 55-74, http://dx.doi.org/10.1016/j.envsoft.2016.01.010

If you are interested in learning more about how ODM2 supports different use cases, have a look at our recent paper in the Data Science Journal.

Hsu, L., Mayorga, E., Horsburgh, J. S., Carter, M. R., Lehnert, K. A., Brantley, S. L. (2017), Enhancing Interoperability and Capabilities of Earth Science Data using the Observations Data Model 2 (ODM2), Data Science Journal, 16(4), 1-16, http://dx.doi.org/10.5334/dsj-2017-004.

Getting Started with ODM2

SQL scripts for generating blank ODM2 databases can be found at the following locations:

View Documentation of ODM2 Concepts

For more information on ODM2 concepts, examples, best practices, the ODM2 software ecosystem, etc., visit the Documentation page on the wiki.

View Diagrams and Documentation of the ODM2 Schema

Schema diagrams for the current version of the ODM2 schema are at:

Data Use Cases

The following data use cases are available. We have focused on designing ODM2 to support these data use cases. Available code and documentation show how these data use cases were mapped to the ODM2.

  • Little Bear River - Hydrologic time series and water quality samples from an ODM 1.1.1 database. Implements an ODM2 database in Microsoft SQL Server.
  • PRISM-XAN - Water quality depth profiles and samples from Puget Sound. Implements an ODM2 database in PostgreSQL.

Our Goal with ODM2

We are working to develop a community information model to extend interoperability of spatially discrete, feature based earth observations derived from sensors and samples and improve the capture, sharing, and archival of these data. This information model, called ODM2, is being designed from a general perspective, with extensibility for achieving interoperability across multiple disciplines and systems that support publication of earth observations.

ODM2 Schematic

Credits

This work was supported by National Science Foundation Grant EAR-1224638. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

ODM2 draws heavily form our prior work with the CUAHSI Hydrologic information system and ODM 1.1.1 (Horsburgh et al., 2008; Horsburgh and Tarboton, 2008), our experiences workin on the Critical Zone Observatory Integrated Data Management System (CZOData), and our experiences with the EarthChem systems (e.g., Lehnert et al., 2007; Lehnert et al., 2009). It also extensively uses concepts from the Open Geospatial Consortium's Observations & Measurements standard (Cox, 2007a; Cox, 2007b; Cox, 2011a; Cox, 2011b; ISO, 2011).

References

See a full list of ODM2 related references

Cox, S.J.D. (2007a). Observations and Measurements - Part 1 - Observation schema, OGC Implementation Specification, OGC 07-022r1. 73 + xi. http://portal.opengeospatial.org/files/22466.

Cox, S.J.D. (2007b). Observations and Measurements โ€“ Part 2 - Sampling Features, OGC Implementation Specification, OGC 07-002r3. 36 + ix. http://portal.opengeospatial.org/files/22467.

Cox, S.J.D. (2011a). Geographic Information - Observations and Measurements, OGC Abstract Specification Topic 20 (same as ISO 19156:2011), OGC 10-004r3. 54. http://dx.doi.org/10.13140/2.1.1142.3042.

Cox, S.J.D. (2011b). Observations and Measurements - XML Implementation, OGC Implementation Standard, OGC 10-025r1. 66 + x. http://portal.opengeospatial.org/files/41510 (accessed September 16, 2014).

Horsburgh, J.S., D.G. Tarboton, D.R. Maidment, and I. Zaslavsky (2008). A relational model for environmental and water resources data, Water Resources Research, 44, W05406, http://dx.doi.org/10.1029/2007WR006392.

Horsburgh, J.S., D.G. Tarboton (2008). CUAHSI Community Observations Data Model (ODM) Version 1.1.1 Design Specifications, CUAHSI Open Source Software Tools, http://www.codeplex.com/Download?ProjectName=HydroServer&DownloadId=349176.

ISO 19156:2011 - Geographic information -- Observations and Measurements, International Standard (2011), International Organization for Standardization, Geneva. http://dx.doi.org/10.13140/2.1.1142.3042.

Lehnert, K.A., Walker, D., Vinay, S., Djapic, B., Ash, J., Falk, B. (2007). Community-Based Development of Standards for Geochemical and Geochronological Data, Eos Trans. AGU, 88(52), Fall Meet. Suppl., Abstract IN52A-09.

Lehnert, K.A., Walker, D., Block, K.A., Ash, J.M., Chan, C. (2009). EarthChem: Next developments to meet new demands, American Geophysical Union, Fall Meeting 2009, Abstract #V12C-01.

odm2datasharingportal's People

Contributors

aufdenkampe avatar darksinge avatar dependabot[bot] avatar elijahwalkerwest avatar fryarludwig avatar horsburgh avatar htaolimno avatar jcaraballo17 avatar maurier avatar ptomasula avatar shannarucker avatar srgdamia1 avatar tslawecki avatar xaviernogueira avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

odm2datasharingportal's Issues

Software Requirements Specifications

Create SRS document with all the knowledge we have. This will help us to know what we need to discuss with SDRC. Use this form for discussion if needed.

Add ability to upload a data file

Users who do not have an active Internet connection for their logger may want to post data files from the SD card on their logger to the website. Eventually need some capability to upload a file and parse the data into the database. Hopefully get to this if we can.

Create a very-low-bandwidth-use status page

Create a text-only, auto-refreshing status page for viewing most current readings from My Sites. This would make it much easier to check the current status of a station while out in the field and having poor data connection on a cell phone.

Create Sandbox Server

Need to set up a sandbox server so we can stage deployments. This should be in Linux so we can test our new Linux-based deployment strategy.

Lighter Weight Data Post Request

The current POST to send data to the website request is heavy because it contains a lot of information - e.g., tokens, tags, etc. Need to consider a lighter weight post request (or potentially even a get request) that simply has a list of ordered numbers, one for each measured variable.

Issues:

  1. How to make sure the values are always in the right order in the request?
  2. How to match the values to the right result on the back end
  3. How to get some security so we don't get spammed (check to make sure requests are authentic).

Create examples of web service requests

Need to post examples of web service POST requests for sending data to the data.envirodiy.org website with different encodings in the documentation for the website:

URL encoding
JSON encoding
...

Add ability to download data file as CSV

Need to add a button to each time series on the site details page that enabled download of the time series as a CSV file. The file should be simple - use the same file template used by ODMTools Python for time series export. @sreeder can get you the format.

Make it faster to include all possible variables from a sensor

For sensors commonly recording multiple variables, make it faster for the user to register that sensor in its most common configuration. So, for a Decagon CTD, have a shorter way to simultaneously register the same sensor with all three variables it can record.

Add data sharing agreement to the sign up page

Need to add a check box at the bottom of the sign up page with the following text:

  • I agree that all of the data I post to this site will be made public according to the Creative Commons Attribution Share Alike License (CC-By-SA).

You should also add the appropriate creative commons license badge/icon below the text:

https://licensebuttons.net/l/by-sa/4.0/88x31.png

Details of the license at: https://creativecommons.org/licenses/by-sa/4.0/

This URL should be linked in some way to the text and/or icon.

If a user does not check this box, do not create an account for them, pop up a message that says:

"To post data to this site, you must agree to the open data license."

Have pages auto-refresh as new data is added

As new live data streams in, it isn't visible until the page is manually refreshed. If possible, it would be great to have all pages with information about real-time data automatically refresh as they receive more values.

Multiple database types

Just wondering if this is going to be designed to connect to different ODM2 database instances dynamically (ones that users already have.) If so, what's the plan to make that work? That'd be interesting.

Allow users to include sensor serial numbers

Was there a reason this was not included? @aufdenkampe ? I don't remember if we deemed it excessive for citizen scientists. I think it would actually be more educational for them to realize that things like the differences between unique sensors can make a difference in the data quality. Of course, it could also lead to problems if people decide to change out their sensors without reprogramming their board with the new UUID's.

Add a page to display current time for clock sync

When hooked up to the WiFiBee, the Mayfly can only connect to a single IP address. It would be great if somewhere on your IP you could set up a very short page only displaying the current US NTP unix timestamp to help sync the clocks regularly. Something analogous to http://time.sodaq.net/.

The RTC in the Mayfly is supposed to be very good, but it never hurts.

Make tabular modal window scrollable

Right now the tabular display of data pops up in a modal window that is not scrollable. This is a problem for high frequency data that make the table disappear off the top of the page. We need to constrain the size of the modal window that pops up and make the tabular display of the data scrollable if it doesn't fit. Example below:

image

Add Help Text to Forms

Users need to know what to enter into each of the fields on the input forms and so some hover text would be helpful so users know what to enter into each one. @horsburgh will write text and @jcaraballo17 will implement.

Timestamp when viewing tabular data is incorrect

All the data from within an hour is displayed as 5-minutes after the hour, and after the wrong hour at that. Hopefully this is only a display issue. The most recent time in the screen capture should be 11:50 EST.

capture

Add ability to export Arduino code for a registered site

Once a user has registered a site and added sensors to it, add a button to the site details/registration page that exports a code block with the following:

  • SamplingFeatureUUID
  • Time zone/UTCOffset Information
  • SecurityToken
  • ResultUUIDs for any attached sensors

This would enable us to use standardized code for the rest of the Arduino code, but the users can paste this code into the standard code. Still need to define exact content and format of the code that is exported.

Make the time the last data point was recieved prominanty visible.

For easily checking the status of each station, we need to know quickly the last time a data point was received. Currently the only way to find the time is to go to each site, then click on the tabular data view, then scroll to the bottom. I would like to see the time of the most recent point above the plots for each station. I would also prefer to see the time of the most recent point at each station on the "My Sites" pane in place of the "Registration Date".

Add ability to delete a registered site

Need the capability to delete a site once it has been registered. This would have the action of deleting all traces of this site and any associated data from the underlying database. There should be a warning message the pops up telling the user that deleting a site is permanent and that all associated data will also be deleted. They must confirm to delete the site.

Automatically merge turbidity data from two ranges

The Campbell OBS3's output turbidity in two ranges, which seems to be quite confusing to users. When plotting long-term turbidity, it would be helpful to users to create a "merged" plot of the turbidity. (That is, the low range value when the sensor is inside the limits of the low range, and the high range otherwise.)

Modifications to Page Navigation

We want to encourage users of the site to sign up, but we do not want to discourage potential users from browsing existing data. So, make the following changes to navigation behavoir of the site:

  • Remove the "My Sites" link for users that are not signed in to the site. It should appear when they have signed in.
  • Change the "Browse Sites" link and page so that it is public and available to anyone regardless of whether they are signed in or not. Viewing individual site pages by clicking on links on the "Browse Sites" page should also be possible for unauthenticated users.

@jcaraballo17 - feel free to reassign to @mauriel133 if this is better addressed by him, but hopefully you can get to this before next week.

Time Zone Support

Need to make sure datetimes sent to the web service have time zone or UTCOffset information and are handled correctly in the database.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.