nasa-petal / petal Goto Github PK

@bruffridge

this is a test

Containerize PeTaL into a single startup script, i.e. using docker-compose to spin up the neo4j server and potentially also to pre-fill the database before starting the Django site.

We need more data in Petal

Summary of Issue

Right now all the cards are on the left side so on the right side there's more space. Centering all the elements will look more good. And pagination on the bottom is not working properly

Issue Type

Enhancement

Description

I will center the cards and the search bar. Font size are small and all seems very close, so I will increase padding too.
Pagination on the bottom is not working properly, so I will fix to just show if there's another page else they will be disabled

Example

I will use Material-UI as this project is already using it and will focus on making it more user-friendly and aesthetic

Right now it's look like this...

Summary of Issue

CSV write Bug in OptimizedCatalog.py

Issue Type

• Bug
• Enhancement

Description

Current
"Craseonycteris thonglongyai (Hill,1974)" -> Craseonycteris thonglongyai (Hill | 1974)

Should be
"Craseonycteris thonglongyai (Hill,1974)" -> Craseonycteris thonglongyai (Hill,1974)

When OptimizedCatalog is run, it creates csv files from json objects to load into neo4j. These csv files were inaccurate due to commas within the JSON values. The new code should work with commas.

Example

Environment

All

Operating System: Windows 10

Python environment: Anaconda Python 3.7.1

Packages:

arrow==0.15.7
asgiref==3.2.10
beautifulsoup4==4.9.1
bibtexparser==1.2.0
bitflow==0.6.0
blis==0.4.1
boto==2.49.0
boto3==1.14.10
botocore==1.17.10
catalogue==1.0.0
certifi==2020.6.20
chardet==3.0.4
click==7.1.2
cycler==0.10.0
cymem==2.0.3
Cython==0.29.14
Django==3.0.7
docutils==0.15.2
efficientnet-pytorch==0.6.3
en-core-web-sm==2.3.1
fake-useragent==0.1.11
future==0.18.2
gensim==3.8.3
idna==2.9
importlib-metadata==1.7.0
jmespath==0.10.0
joblib==0.15.1
kiwisolver==1.2.0
lxml==4.5.1
matplotlib==3.2.2
murmurhash==1.0.2
neo4j==4.0.0
neobolt==1.7.17
nltk==3.5
numpy==1.19.0
pandas==1.0.5
Pillow==7.1.2
plac==1.1.3
plotly==4.8.1
preshed==3.0.2
pyparsing==2.4.7
PySocks==1.7.1
python-dateutil==2.8.1
pytz==2020.1
regex==2020.6.8
requests==2.24.0
retrying==1.3.3
s3transfer==0.3.3
scholarly==0.3.3
scipy==1.4.0
seaborn==0.10.1
six==1.15.0
smart-open==2.0.0
soupsieve==1.9.6
spacy==2.3.2
sqlparse==0.3.1
srsly==1.0.2
stem==1.8.0
thinc==7.4.1
torch==1.5.1+cpu
torchvision==0.6.1+cpu
tqdm==4.46.1
urllib3==1.25.9
wasabi==0.7.1
wikipedia==1.4.0
zipp==3.1.0

Find papers with data we want to add to PeTal
Extract results from said papers
Categorize subject of study into engineering terms (seal whiskers might be under fluid dynamics, capillaries might be under heat transfer)
Have certain 'stats' recorded based on engineering term of interest (for biomimetic studies on capillaries or other fluid flow structures, stats might include Reynolds number, Nusselt, or Grashof)

Summary of Issue

Remove duplicate/unneeded static files and organize them into one location

Issue Type

• Maintenance

Description

Duplicate/unneeded .jpg, .png, .js, .json files.
Need to organize into one location, or have a set template for them.

Vik and PAht has the files

Hey Lucas,

Hate to ask you another question, but am I possibly missing some code or data here? I ran

./run config/mock_species_articles.json

and then after checking that the database actually had data in it. I ran

./run config/search.json

to create the index file. I got this error message

AttributeError: 'Result' object has no attribute 'records'

Documentation about the error message is attached! I really appreciate your time if you could help me with this!
AttributeError 'Result'object has no attribute 'records'.pdf

Based on the fields that are to be populated, a content extractor (perhaps from AllenNLP) would be used to generate a score of whether the inputted answer is correct with regards to the text and the previous answers that are in the system.

As an Engineer, I want to use a guided search (dropdowns) of functions to query for articles, species, etc.

Basically transfer legacy version bird_e2b.html from Flask to Django. Connect to neo4j database.

Based on the fields that are to be populated, a content extractor (perhaps from AllenNLP) would be used to generate a score of whether the inputted answer is correct with regards to the text and the previous answers that are in the system.

Store raw results from neo4j server in central accessible location.
Create optimal UI for combining the UX of all three visualization tools with a single set of data.

Edit the bitflow code to recursively recognize PeTaL modules, take multiple directories as input, and give more helpful error messages when modules aren't found or have errors.

Develop code to mine the following and store into. Store the data into NEO4J

• NTRS Database https://ntrs.nasa.gov/search.jsp?N=0

• Assign Elliot

Connect PeTaL UI inputs with NLP and Vision.

1. Create Flow-Chart/Diagram

2. Create Wire-frame

3. Create Mockup

PeTal UI should...

Take user inputs

• * Free-hand search
• * Guided search
• * Image search
• * Browser search/selection
• * User-defined taxonomy input

Allow user to store data gathered from PeTaL tools

Allow user to selectively visualize stored data in PeTaL tools

Wire-frame - https://projects.invisionapp.com/freehand/document/p5uNMEgWJ
Flow Diagram - [PeTaL Flow-Diagram v3.0].pptx](https://github.com/nasa/PeTaL/files/5108889/PeTaL.Flow-Diagram.v3.0.Autosaved.pptx)
Mockup - PeTaL Mockup v1.2.pptx
PeTaL Mockup v3.0 [Autosaved].pptx

Different files inside module/* have manually (in the python code) entered connection strings. We need to change this so that it uses the same configuration file (json)

Example files:
site/search/bird.py
modules/libraries/download_wikipedia/wiki_to_neo.py

There may be other files that need updating.

Summary of Issue

Updates sass files and recompile styles.css

Issue Type

• Enhancement

Description

Remove unneeded and unused .scss files from static/sass folder. Update paths in the .scss files. Recompile styles.css based on the edited .scss files.

Environment

N/A

Create UI to enable user inputs...

• through BIRD dropdowns
• autocomplete BIRD search with neo4j data
• connect BIRD dropdowns to neo4j data
• from central/top located search bar
• through an Ontology Viewer
• through BioMole
• of an user-defined taxonomy

Connect to neo4j database

Summary of Issue

<A brief summary of the issue (used for release notes).>

Issue Type

• Bug
• Enhancement

Description

For bugs, explain what is happening and how it differs from the expected behavior.

For enhancements, include a description of the proposed enhancement.

Example

For bugs, if at all possible, provide a short snippet which reproduces the issue. Or if it is a user interface issue, include the steps needed to cause the bug and include
screenshots as needed.

For enhancements, if possible, give some suggestions for fixes. This could be in the form of code or UI mockups.

Environment

<For bugs, list the environment where the bug was noticed. Can say "All" if the bug is not environment specific. For enhancements, use N/A>

Operating System: <e.g. OS X 10.14.6, Windows 10, Ubuntu 16.04>

Python environment: <e.g. Anaconda Python 3.7.1>

Packages: <versions of PeTaL, and other pertinent Python packages. You can just paste the output of pip freeze or conda list here.>

Get an error when running this:
./run config/mock_species_articles.json

Anyone else getting this error when starting up the Django web server? I don't see a data/index file in the repo anywhere, am I supposed to generate this file somehow?

https://github.com/nasa/PeTaL/blob/13d280b79343d1858cddef13391c74e88a13025b/site/bird/search.py#L25

Poetry offers modern python package management and wraps both venv and pip. It also makes distribution as a library much simpler.

Guide to writing user personas

User Personas will be documented here:
https://docs.google.com/document/d/1In5t369vErfVZKHZMbjynCx090P0PYUmrEsZtN3tKDM/edit

Summary of Issue

Add autocomplete capability to the BIRD search tool

Issue Type

• Enhancement

Description

The main goal is to create UI that will display an autocomplete list based on a string/list of selections given. Creating the processing of the input (NLP) to decide what terms or set of words to list as selections is not the goal of this enhancement.

Example

Possibly similar to Google's UI

Environment

N/A

UI: search form, results output.
Backend: get list of results to return.

A new branch is needed so external contributors may submit pull requests. This is so repository maintainers have a single branch to use for reviewing all external contributions.

Add Maju's segmentatioin work. Vik or Paht can send you the files

Summary of Issue

BIRD dropdown menu lists will be populated with relevant Neo4j data.

Issue Type

Enhancement

Description

Three dropdown menu's should be created. Functions, Species, Environment.

Example

Here is a link as an example end product for the species dropdowns.

How to create an initial domain model

• Assign Elliot

Summary of Issue

Bugs when parsing and loading COL (Catalogue of Life) data into neo4j

Issue Type

• Bug
• Enhancement

Description

Problems in OptimizedCatalog.py

• When adding edges to existing nodes, the code would not create/add anything.
• When data does not give the entire path of the taxonomic hierarchy for each species or other taxonomic categories, the code does not return the correct results.
• Node property name: includes author and date in addition to the scientific name.

Example

Most species and taxon nodes are not connected

Environment

All

Operating System: Windows 10

Python environment: Anaconda Python 3.8.3

Packages:

arrow==0.15.7
asgiref==3.2.10
beautifulsoup4==4.9.1
bibtexparser==1.2.0
bitflow==0.6.0
boto==2.49.0
boto3==1.14.10
botocore==1.17.10
certifi==2020.6.20
chardet==3.0.4
click==7.1.2
cycler==0.10.0
Cython==0.29.14
Django==3.0.7
docutils==0.15.2
efficientnet-pytorch==0.6.3
fake-useragent==0.1.11
future==0.18.2
gensim==3.8.3
idna==2.9
jmespath==0.10.0
joblib==0.15.1
kiwisolver==1.2.0
lxml==4.5.1
matplotlib==3.2.2
neo4j==4.0.0
neobolt==1.7.17
nltk==3.5
numpy==1.19.0
pandas==1.0.5
Pillow==7.1.2
plotly==4.8.1
pyparsing==2.4.7
PySocks==1.7.1
python-dateutil==2.8.1
pytz==2020.1
regex==2020.6.8
requests==2.24.0
retrying==1.3.3
s3transfer==0.3.3
scholarly==0.3.3
scipy==1.4.0
seaborn==0.10.1
six==1.15.0
smart-open==2.0.0
soupsieve==1.9.6
sqlparse==0.3.1
stem==1.8.0
torch==1.5.1+cpu
torchvision==0.6.1+cpu
tqdm==4.46.1
urllib3==1.25.9
wikipedia==1.4.0

Implement image segmentation using EfficientNet
Implement object detection using EfficientNet
Send results (taxonomy, patterns, objects) to UI
Store results in neo4j

Find the data necessary to cluster and extract data type, shape and size in order to accurately create topic modeler

• Assign Elliot

Pull Kunle's branch of petal https://github.com/kakinpe1u/PeTaL/tree/feature_elasticsearch
and https://github.com/kakinpe1u/PeTaL/tree/feature_indexer

Try to see how we can integrate Kunle's elastic search with Petal.
If we can't integrate then we need to know.
This needs to be done ASAP.

we'll move current master branch to a new branch, and use master branch to check in working, tested code.

Based on the fields that are to be populated, a content extractor (perhaps from AllenNLP) would be used to generate a score of whether the inputted answer is correct with regards to the text and the previous answers that are in the system.

• Assign Elliot

Summary of Issue

Error when scraping for JEB articles and storing in neo4j

Issue Type

• Bug

Description

Problems with JEBModule.py.

Only scrapes first article result page. Not the successive pages.

When adding edges to existing nodes, code would not create/add anything into neo4j database.

UUID assignment problem when storing data to neo4j database.

Example

Run a config .json file from main.py with NormalCatalog and JEBModule in the bitflow:whitelist.

Environment

All

Operating System: Windows 10

Python environment: Anaconda Python 3.8.3

Packages: all packages in PeTaL poetry package manager

react with material-ui perhaps? if a full fledged app UI framework like material-ui is too heavy for our MVP (and probably is) I'm ok with just using SASS -> CSS.