rubyforgood / abalone Goto Github PK
View Code? Open in Web Editor NEWA data tracking and analytics app for abalone conservation efforts.
License: MIT License
A data tracking and analytics app for abalone conservation efforts.
License: MIT License
The histogram is at /reports
under the Growth tab.
We need to adjust the calculation we are using for the histogram on the backend. There are some calculations in app/lib/aggregates.rb
, but we need to make sure they match the below requirements:
For histograms, bins to be by 1 cm increments, so 0-0.99 cm, 1-1.99 cm, etc. Our largest animal is just over 18 cm and the max listed size for a white abalone is 10 inches (25.4 cm), so it's probably safe to say we wouldn't ever have one over 30 cm. For a specific bar in the histogram, here is a scenario: Say I want to know X, the number of animals in the 4-4.99 cm bin, for the SF15-77 cohort animals:
N = the number of animals that were 4-4.99 cm in length in the most recent uploaded spreadsheet (Untagged AND Tagged animals) with length data for that cohort
S = the total number of animals measured in the same spreadsheet, i.e., the most recent uploaded spreadsheet with length data for that cohort (Untagged AND Tagged animals)
T = the total number of estimated animals in the cohort = the total number counted on the most recent count closest to the length measurement date minus the mortalities since then
X = (N/S)*T ... or the proportion of animals that were measured that were in that size bin, scaled up to the total predicted number of animals in that cohort.
The histogram calculations should have the following parameters (you should be able to pass in one or both):
Use the module ImportJob
for most of the logic.
Acceptance Criteria:
As a user, when I upload a CSV and it fails to process, I go the the page /file_uploads
and:
BLOCKED by Issue #58 : We need the correct data on the backend first.
We need to make sure the Length Histogram is rendering correctly at /reports
under the Growth tab. The current report has incorrect data AND incorrect formatting.
Acceptance Criteria:
We use Highcharts, a JavaScript library for rendering all graphs and charts.
This issue builds on the devise work done in #76
We need a way for new users to be created through the app. This should be as lightweight and minimal as possible. There is no guarantee we can send email reliably, otherwise devise-invitable
would be a good choice.
administrate
to Gemfile, bundle
rails generate administrate:dashboard
user
modelusers
table was already created in PR 76user
model, it should include name
and email
. Devise will probably want password
and maybe some other stuff.Confirm that you can do the following:
As an authenticated user:
Don't worry too much about roles - do whatever is simplest and least code. If that means everyone is an admin, start there.
Please pop into the #abalone channel in rubyforgood.slack.com if you have any questions or want to grab this issue.
Create a job to import the Spawning Success data.
Can we render the /file_uploads/#id
page nicer? Currently headers and columns are cut off.
Tests need to be written for ImportJob
, which is a module used by all the CSV import jobs.
Unit Tests
The following methods should be unit tested:
Acceptance Criteria:
sample_date_files/wild_collection
on the page /file_uploads/new
.spec/jobs/tagged_animal_assessment_job_spec.rb
Notes:
Much of the logic for the file uploading/parsing is already written in the module concern ImportJob
.
The heaviest lifting will be adding appropriate validations to the WildCollection
model. Please refer to the data dictionary and the notes below for coding these.
**For columns that have the Facility
, let's make these foreign keys to the Facility.rb
model we already have. There should only be certain facilities (see the seeds.rb
file and users have the ability to add new ones.)
WildCollection.rb
Acceptance Criteria:
sample_date_files/tagged_animal_assessment
on the page /file_uploads/new
with correct data formats. I should see errors if there are incorrect data formats.Please refer to the data dictionary and the notes below for coding the model validations. For an example, please look at: Example of WildCollection model validations and spec
TaggedAnimalAssessment.rb
**If there is a column for Facility, let's make these foreign keys to the Facility.rb
model we already have. There should only be certain facilities (see the seeds.rb
file and users have the ability to add new ones.)
This resulted from a conversation in this PR: #88
Currently, when a CSV is processed with some invalid rows, we reject the invalid rows and save the valid rows. However, we should not save any rows to the db because this risks the user of duplicating data. Let's reject the entire CSV and tell the user to fix the invalid rows and re-upload the file.
Analytics:
We will need to work with Kristin's team to ensure that upload errors are not happening too frequently.
Render data
Related to counts, one graph I’d love to be able to produce easily is one that shows the total number of animals that make it to ~1 year old each year. We have a “first count” when the animals are between 10-12 months old.
BLOCKED: UntaggedAnimalAssessmentJob
must be created first.
Tests need to be written for UntaggedAnimalAssessmentJob
.
Please use a fixture file for testing in this directory: db/sample_data_files/untagged_animal_assessment/
Feature/Integration Tests
The following contexts and expected outcomes should be tested:
ProcessedFile
record should be createdProcessedFile
record should be createdProcessedFile
record should be createdUntaggedAnimalAssessment
records should be createdProcessedFile
record should be createdUntaggedAnimalAssessment
records should be createdAcceptance Criteria:
sample_date_files/mortality_tracking
on the page /file_uploads/new
.spec/jobs/tagged_animal_assessment_job_spec.rb
Notes:
Much of the logic for the file uploading/parsing is already written in the module concern ImportJob
.
The heaviest lifting will be adding appropriate validations for the MortalityTracking
model. Please refer to the data dictionary and the notes below for coding these. For an example, please look at: Example of WildCollection model validations and spec
**If there is a column for Facility, let's make these foreign keys to the Facility.rb
model we already have. There should only be certain facilities (see the seeds.rb
file and users have the ability to add new ones.)
MortalityTracking.rb
#_of_morts
> 1), can we create multiple entries in the db for each individual animal, and can we extract each animal's tag from the Note
column??Users should be able to select multiple CSV files of the same type to be uploaded/parsed on this page: http://abalone.blrice.net/file_uploads/new.
Acceptance Criteria:
As a user, I can upload a maximum of 10 CSV files in bulk of the same type (e.g. Untagged Animal Assessment) from my computer.
Our deployed site is running on: http://abalone.blrice.net/
We need to get DelayedJob working in production. Maybe check out https://github.com/collectiveidea/delayed_job/wiki/Delayed-Job-tasks-for-Capistrano-3
Most of the code for this CSV import is already written in ImportJob
. Import this module into an UntaggedAnimalAssessmentJob
and write any custom methods; it should look similar to the TaggedAnimalAssessmentJob
.
How many animals do I have from each spawning date right now?
Acceptance Criteria:
sample_date_files/tagged_animal_assessment
on the page /file_uploads/new
.spec/jobs/tagged_animal_assessment_job_spec.rb
Notes:
Much of the logic for the file uploading/parsing is already written in the module concern ImportJob
.
The heaviest lifting will be adding appropriate validations for the UntaggedAnimalAssessment
model. Please refer to the data dictionary and the notes below for coding these. For an example, please look at: Example of WildCollection model validations and spec
**If there is a column for Facility, let's make these foreign keys to the Facility.rb
model we already have. There should only be certain facilities (see the seeds.rb
file and users have the ability to add new ones.)
UntaggedAnimalAssessment.rb
Acceptance Criteria:
sample_date_files/population_estimate
on the page /file_uploads/new
.spec/jobs/tagged_animal_assessment_job_spec.rb
Notes:
Much of the logic for the file uploading/parsing is already written in the module concern ImportJob
.
The heaviest lifting will be adding appropriate validations for the PopulationEstimate
model. Please refer to the data dictionary and the notes below for coding these. For an example, please look at: Example of WildCollection model validations and spec
**If there is a column for Facility, let's make these foreign keys to the Facility.rb
model we already have. There should only be certain facilities (see the seeds.rb
file and users have the ability to add new ones.)
PopulationEstimate.rb
We need basic authentication for at least one user. Login page.
ActiveStorage will not work on Heroku for temp file storage for reasons listed here:
https://devcenter.heroku.com/articles/active-storage-on-heroku. Currently file uploading breaks on production for this reason :(
Let's just use a good ole' postgresql table instead called TemporaryFile
to temporarily store the CSV. Workflow should be:
Add Tagged Animal Assessment file upload category.
Add data import job for Tagged Animal Assessment spreadsheets.
What sizes are animals from each spawning date? I'd like to be able to select a measurement event (a certain population on a certain date) or a group of measuring events (different populations near the same dates or same population over time) to generate a histogram of lengths ideally binned in 1-cm increments.
There should be an area where the user can select parameters in the upper right hand side of the histogram chart at /reports
under the Growth tab. The user should have the ability to select a measuring event (i.e. a certain population on a certain date) or a group of measuring events (different populations near the same date).
Acceptance Criteria:
We use Highcharts, a JavaScript library for rendering all graphs and charts.
We want to have a basic linter set up and enforced for new PRs.
Acceptance Criteria:
sample_date_files/spawning_success
on the page /file_uploads/new
.spec/jobs/tagged_animal_assessment_job_spec.rb
Notes:
Much of the logic for the file uploading/parsing is already written in the module concern ImportJob
.
The heaviest lifting will be adding appropriate validations for the SpawningSuccess
model. Please refer to the data dictionary and the notes below for coding these. For an example, please look at: Example of WildCollection model validations and spec
**If there is a column for Facility, let's make these foreign keys to the Facility.rb
model we already have. There should only be certain facilities (see the seeds.rb
file and users have the ability to add new ones.)
SpawningSuccess.rb
spawning_date
for this CSV...Tests need to be written for TaggedAnimalAssessmentJob
.
Please use this fixture file for testing: db/sample_data_files/tagged_animal_assessment/Tagged_assessment_12172018 (original).csv
Feature/Integration Tests
The following contexts and expected outcomes should be tested:
ProcessedFile
record should be createdProcessedFile
record should be createdProcessedFile
record should be createdTaggedAnimalAssessment
records should be createdProcessedFile
record should be createdTaggedAnimalAssessment
records should be createdUse Capybara or MiniTest? See https://chriskottom.com/blog/2015/11/testing-rails-background-workers/
Set up a deploy pipeline via DigitalOcean.
Acceptance Criteria:
sample_date_files/pedigree/
on the page /file_uploads/new
.spec/jobs/tagged_animal_assessment_job_spec.rb
Notes:
Much of the logic for the file uploading/parsing is already written in the module concern ImportJob
.
The heaviest lifting will be adding appropriate validations to both models. Please refer to the data dictionary and the notes below for coding these.
**If there are columns that have Facility
, let's make these foreign keys to the Facility.rb
model we already have. There should only be certain facilities (see the seeds.rb
file and users have the ability to add new ones.) For an example, please look at: Example of WildCollection model validations and spec
Pedigree.rb
PedigreeParents.rb
**Need model and migration for this
We need to make sure the Length Histogram is rendering correctly: http://abalone.blrice.net/reports.
This should create bins of 1 cm increments, so 0-0.99 cm, 1-1.99 cm, etc up to 30cm.
We also need to be able to input parameters of 1) Cohort or multiple cohorts and 2) Date range (optional - default is the most recent).
--> Will need to model a select dropdown of cohorts
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.