pecanproject / betydb-yaba Goto Github PK

Right now a generic message is returned when a CSV file has too few or too many of the required columns. Returning a message that lists the column names expected, and indicates that the CSV files must contain only those columns, and all of them, would be more helpful

The array needs to be initialized inside the loop at the following location:

Right now, it accumulates every point from each geometry object and doesn't start fresh for each object.

This issue aims to generate a coverage report for python tests in Py_test action. Right now, it produces a warning and exits

For now, you can see the run for actions here: BETYdb-YABA/actions --> YABA app and client tests

@gsrohde brought up the following points in this comment: PecanProject/betydbdoc-dataentry#4 (comment)

These questions should be answered in the README and / or the betydbdoc-dataentry documentation:

• Does each user who wants to interact with a database have to deploy their own copy of the YABA app?
• Can it be used to connect to any BETYdb database, or does it have to be run on the same machine where the database lives?
• What controls the permissions on accessing and altering the database?

Need to define api endpoints that will be used in the metadata upload workflow https://osf.io/v7f9t/wiki/Metadata%20Entry%20Workflow/

The idea is that you will have an endpoint for each of the add_* functions in https://github.com/az-digitalag/betydbtools/pull/2/files

But probably easiest to follow the sql here https://osf.io/v7f9t/wiki/Metadata%20Entry%20Workflow/

Completion criteria

• published on swaggerhub
• add a *.yaml file in openapi 3.0 schema to the repository

@dlebauer @kimberlyh66 @gsrohde I have added the schema of CSV in Swagger under examples .Now i implemented for experiments endpoint only.
https://app.swaggerhub.com/apis/y51/bet-ydb_yaba/1.0.5#/experiments/Meta.insert_experiments

Here only i added the columns and one row as example.

Please review it.

The following line should be removed since data variable already has the notes:

Causes a crash due to data_g probably not having notes field.

When loading plots through the YABA interface plot names are not correctly matched up with the shapes in the shapefile leading to incorrect plot boundaries.

The goal of this issue is to implement data validation for shapefiles, spreadsheets before API data ingestion by modifying existing API routes functionality.

Like, the code in this function responds to /yaba/v1/experiments (for inserting experiments in the BETY database), we can modify this, by accepting one more parameter as status(which can be true or false), if false then return a valid response after validating data otherwise continue with data insertion. In this example, we can check the status after line 64, if true we can return a response from there only and stop further execution otherwise continue with data insertion.

if(all(x in accepted_columns for x in columns)):
>> if(status == false)
>>>> msg = {'Message' : true }
>>>> return make_response(jsonify(msg), 200)

So, what do you say?
@dlebauer @KristinaRiemer @saurabh1969 @Chris-Schnaufer

This is not needed b/c

1. people won't be using this on mobile.
2. navbar not needed (this is a form / workflow UI not a navagable website)

Why isn't this display style part of the "Link" element CSS? The bar may change its elements spacing when the user clicks it & it may be better to start out with correct spacing.

Originally posted by @Chris-Schnaufer in #50 (comment)

Need to draft what the UI for an end user who wants to upload metadata to BETYdb will look like.

This will be needed for Phase II of GSOC but it will help to have this sooner. See also https://osf.io/v7f9t/wiki/GSOC%202019%20Workplan/

This may require a few iterations.

Completion criteria - a document with images of each web page in the metadata upload workflow to be implemented.

There should be a page in that I can point data providers to that describes how to prepare and upload data. This would loosely follow the steps outlined in the workflow that uses SQL https://osf.io/v7f9t/wiki/Metadata%20Entry%20Workflow/

This can be drafted in markdown and later moved to OSF

Find a way to extract RGB channels from digital images in both R and python.

For python, using imageio with this code example. For R, looking into raster package and this code example.

Currently the design column is required in the CSV file. This requirement should be removed from the code

• super linter
• tests for python and js
• code coverage (eg codecov or sonarcube)
  • PR requires code coverage to stay the same or increase

Write stylesheets for application endpoints to upload metadata

Put together list of deliverables and specific tasks in a new OSF page for the first deadline

@kimberlyh66 please

• review the BETYdb-YABA API
  • Specification on Swaggerhub https://app.swaggerhub.com/apis-docs/y51/bet-ydb_yaba/0.0.1
  • openapi specification https://github.com/PecanProject/BETYdb-YABA/blob/master/yaba.yaml
  • Compare endpoints to the documentation in OSF / functions you have created.
• Fork the version @saurabh1969 put on swaggerhub https://app.swaggerhub.com/apis/y51/bet-ydb_yaba/0.0.1
  • Improve / expand documentation where possible
  • suggest any other changes

Created a shape-visual.js and gmap.js component that uses react-leaflet to visualize the GeoJSON data.

• interface/src/components/shape-visual.js
• interface/src/components/gmap.js
• interface/src/components/stylesheets/shape-visual.css

We are going to merge #5. Then we can refactor to combine the contents of input_files and yaba_test/input_files, which are duplicates

Actually, when I was inserting sites table using the client route,

curl -F "fileName=@input_files/sites.csv" \
-F "shp_file=@input_files/S8_two_row_polys.shp" \
-F "dbf_file=@input_files/S8_two_row_polys.dbf" \
-F "prj_file=@input_files/S8_two_row_polys.prj" \
-F "shx_file=@input_files/S8_two_row_polys.shx" \
http://localhost:6001/sites

the app returned the response 410 and logged this error,

ERROR:root:Traceback (most recent call last):
yaba_api_1 | File "fiona/_shim.pyx", line 74, in fiona._shim.gdal_open_vector
yaba_api_1 | File "fiona/_err.pyx", line 270, in fiona._err.exc_wrap_pointer
yaba_api_1 | fiona._err.CPLE_OpenFailedError: '/code/temp/shp_file' not recognized as a supported file format.
yaba_api_1 |
yaba_api_1 | During handling of the above exception, another exception occurred:
yaba_api_1 |
yaba_api_1 | Traceback (most recent call last):
yaba_api_1 | File "/code/Meta.py", line 132, in insert_sites
yaba_api_1 | data_g1=gpd.read_file(shp_file_target)
yaba_api_1 | File "/usr/local/lib/python3.7/site-packages/geopandas/io/file.py", line 77, in read_file
yaba_api_1 | with reader(path_or_bytes, **kwargs) as features:
yaba_api_1 | File "/usr/local/lib/python3.7/site-packages/fiona/env.py", line 398, in wrapper
yaba_api_1 | return f(*args, **kwargs)
yaba_api_1 | File "/usr/local/lib/python3.7/site-packages/fiona/__init__.py", line 254, in open
yaba_api_1 | layer=layer, enabled_drivers=enabled_drivers, **kwargs)
yaba_api_1 | File "/usr/local/lib/python3.7/site-packages/fiona/collection.py", line 154, in __init__
yaba_api_1 | self.session.start(self, **kwargs)
yaba_api_1 | File "fiona/ogrext.pyx", line 484, in fiona.ogrext.Session.start
yaba_api_1 | File "fiona/_shim.pyx", line 81, in fiona._shim.gdal_open_vector
yaba_api_1 | fiona.errors.DriverError: '/code/temp/shp_file' not recognized as a supported file format.

But when I use the yaba_api route for inserting sites table,

curl -F "fileName=@input_files/sites.csv" \
-F "shp_file=@input_files/S8_two_row_polys.shp" \
-F "dbf_file=@input_files/S8_two_row_polys.dbf" \
-F "prj_file=@input_files/S8_two_row_polys.prj" \
-F "shx_file=@input_files/S8_two_row_polys.shx" \
http://localhost:5001/yaba/v1/sites

it succeeded in doing so. Also, when I am making a request using the browser, both routes log the same error. Don't know what is causing this.

@Chris-Schnaufer @saurabh1969 @dlebauer @KristinaRiemer

• Set up TravisCI
• Writing unit tests for YABA Flask api.
  • for sample data at each endpoint

Completion criteria:

• Commit with .travis.yml
• Sample data uploaded for testing
• also merge sample data and test data #11

The following line needs to check for a file name before trying to remove the file - there's a chance it hasn't been set:

This issue aims at writing tests for the interface part of the YABA app, using React Testing Library and Jest.

Is BETYdb YABA only going to be used to load data from the Phoenix area? If not, a means of allowing the user to specify the GMT offset should be added

Originally posted by @Chris-Schnaufer in #50 (comment)

The code at

This needs to be changed to first check for a Z value, and if it's missing adding on a 0 (zero) Z value

Created a requests.js file to hit client routes for data upload.

• interface/src/components/requests.js

For GitHub actions, can you use a matrix for testing the app, interface, etc. folders? It's easier to maintain a single YAML file than multiples of the same with minor variations.

https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idstrategymatrix

The idea was to see if matrix could be used to combine the two Python tests into one file. I feel that putting Python and Node tests into the same file defeats simplification. If putting the Python app-test and client-tests into the same file using matrix makes things less simple, then that shouldn't be done.

Originally posted by @Chris-Schnaufer in #63 (comment)

Draft design documents from: #1

Note that these are a very crude approximation of the core functionality. We can discuss design, layout, content in the review.

Welcome

Upload

Upload Error Page

Validation / Review Page

Upload Successful

@saurabh1969 please describe and work done to date and put in a feature branch

A proposed validation / review page. The goal of this is to make sure that the person uploading the metadata has a chance to confirm that the metadata they are uploading is correct / consistent with how they think it should be It can look like this:

Overview -

• Use psql transactions to validate data with the backend (start session w/ begin and end with commit or rollback)
• the position of each rectangle is defined by the geometry field in the shapefile - these will go into the sites.geometry field
  • for the first plot you can get the cultivars for each plot through a many-to-many join with sites thus: select name as cultivar, sitename as plot from cultivars join sites_cultivars on cultivars.id = sites_cultivars.cultivar_id join sites on sites_cultivars.site_id = sites.id
  • for the next two plots you can find experiments for each site through the experiments_sites table, and treatments through the experiments_treatments table
• Use Leaflet to visualize
  • you can use shapefile extension
  • later it might be better to convert shapefile to geojson that leaflet handles natively.

Bonus: (can create a new issue for this): a page overlaying plots on a satellite or drone imagery to confirm the correct positions in space.

• Implementation of metadata upload api with respect to: https://app.swaggerhub.com/apis-docs/y51/bet-ydb_yaba/1.0.1

• SQL Query as per : https://osf.io/v7f9t/wiki/Metadata%20Entry%20Workflow/

• Upload will be done to the following tables:
  1.Experiment
  2.Sites
  3.Treatments
  4.cultivars
  5.citations
  6.experiments_sites
  7.experiments_treatments
  8.sites_cultivars
  9.citations_sites

Completion criteria:
☐ Setup Flask
☐ Creating endpoints for each table
☐ Pull Request

This issue aims at

• Setup Docker containers for Interface and Visualization part of the YABA app
• Modify docker-compose.yml for the YABA app

Also, address CORS security issue in #50 (comment)

We need to define the user story, how an extractor developer would define and put this information in. Mostly a placeholder for now until we have a use case to determine the need and mechanism, e.g. from @cmbrown89 and/or @Chris-Schnaufer

This should provide a GUI front end to the python client. https://github.com/PecanProject/BETYdb-YABA/blob/master/client/client.py#L27

It should accept the fields:

• Experiment Name
• Start Date
• End Date
• Description
• Design

And should be inserted into the experiments table in BETYdb using the API client

Currently it appears elevation is set to 115 by default

• Short term fix - set to 0 by default
• Potential fix:
  • Better fix: ask user
  • Best: use lat/lon to query elevation from public API

The aim of this issue is to enable CORS (Cross-Origin Resource Sharing) for the application to allow resource sharing from other origins. Actually, the frontend application(currently under-development) needs to post data to the YABA app but it isn't allowed as of cross-origin policy, for which we need to enable CORS in the YABA app.

It can be done using Flask-CORS extension
https://flask-cors.readthedocs.io/en/latest/

Go through process (perhaps in next season) and take notes on what needs to be improved to streamline the process.

Can include improved template and also validation, e.g. at least send error messages back in response

Set up the environment for development using node js, create-react-app module.

This should be documented in README so that someone else can quickly get the application up and running

• Link to this template (if not already) https://docs.google.com/spreadsheets/d/1Fr_8xYOucyCQ9WH5_1bfPTMpm3IhKX5oqW4UOMvPoSY/edit#gid=0
• Also add tables to readme w/ information from README spreadsheet (you can use markdown tables generator)
• Need to add shapefile specs

The aim of this issue is to make the python codebase follow all standard rules by a linter. Right now, some warnings are ignored in the pylintrc file, make the codebase follow all these standard rules. This will make the YABA app better.

Actually, there were a lot of warnings for inconsistent-return-statements, missing-module-docstring and somehow Pylint action treats them as an error, not warning and so it fails the test for linting.

Originally posted by @im-prakher in #63 (comment)

Created an error page if an error occurs while uploading tables to the YABA app.

• interface/src/components/error.js
• interface/src/components/stylesheets/error.css

After the API key is entered in the Welcome page, it should be validated (#18)

• check that API key is valid
  • select count(*) from users where apikey = $user_provided_apikey should return 1
• confirm that user name. Display the name returned by the following query:
  • `select name from users where apikey = $user_provided_apikey

Implement functions in Python (like R package) that enable use of the YABA API (unlike the existing R package, this will use the API rather than ODBC database connection).This will be used for both scripted workflows and as a backend to the UI.

Completion criteria:

• Commit with client.py
• Integration with UI.

This should go in

• the YABA repo's README
• the BETYdb docs.
• These docs should be referenced in the PEcAn API documentation

User content:

• Link to interface
• Description/screenshots of data tables/shapefiles (see #54), point out example data

Developer content:

• Endpoints
• How to deploy interface

Created a success page after successful upload.

• interface/src/components/success.js
• interface/src/components/stylesheets/success.css

remove leading and trailing whitespace,

eg. this: ABC DEF ,Lactuca sativa,, should become ABC DEF,Lactuca sativa,,

for all fields

Would it make sense to set environment variables here that reference the other YABA server addresses that can be accessed through ReactJS's process.env? This way only one place needs to change if the server information changes

Originally posted by @Chris-Schnaufer in #69 (comment)

The aim of this issue is to set port and host environment variables for the interface and visualization part in the docker-compose file itself so that can be used in these parts through ReactJS's and Node.js process.env

For reference- Reading an environment variable in react which was set by docker

There are currently two issues:

1. In credentials.yaml, the PGHOST setting varies depending on the user's IP and breaks (at least on Macs) when set as localhost
2. The sites table endpoint is not working

@Chris-Schnaufer might be able to help @saurabh1969 troubleshoot #1?