servicenet's Introduction
servicenet's People
servicenet's Issues
Map different data files to a common structure
As known, data files from different providers have a different structure. We should map those structures to our entities and persist them in the database.
Develop logic for sorting importance of updates
As we discussed on our call, the home page should display at most 3 lines of updates per record cluster.
For records that have greater than three updates, we should choose the three 'highest priority' updates to display in full and then display text indicating there are more, e.g. "There are 5 additional updates for this record"
The ranking for update type:
'Recommendations' are always higher priority than 'Updates'
'Updates' are always higher priority than 'Conflicts'
The ranking for fields:
- Address
- Phone
- Name
- Service Type
- All other fields
Recommendations with a higher number of partners should outrank those with only a single matching partner. Same for Updates and Conflicts.
Create basic user login system
The mapped data from the providers should not be duplicated
For each provider, all entities should be persisted only if they're unique.
Acceptance Criteria:
- Same data, persisted twice results in no new entities
- There are no unwanted duplicated entities within one data upload
Research matching algorithm approaches
Using the representative data from the data repo, draft an architectural suggestion for approaches around how we can match records across various partner datasets with different structures/formats/fieldnames etc
Add a custom scheduler to the project and a page for its administration
Some of the partners will provide their data as an API that can be queried. It'll have to be downloaded automatically after a specific period of the time, but there has to be an option to manually trigger it as well. Probably the best way to achieve it is to have a simple page for scheduler administration. There should be listed all schedulers with a button to automatically trigger them. If it can be easily done, there could also be information about previous and next trigger dates.
Add activity log to single record view
Add map view of multi record
Add side-by-side compare to single record view
In side panel add an overview of matches in partner systems:
Clicking, always shows logged in user's record on left - matching record (complete original record, with normalized fields where possible) on right. When there are multiple matches, the user should be able to flip between records on the right side. We should have a feedback mechanism for each match allowing a user to manually remove a match when they think it's wrong - this should remove the match relationship in our DB (and maybe negate weights if we ever get to ML)
Create single record view
Placeholder parent task for the single-record view a logged in user gets when clicking on a single row in the list view.
More detailed tasks will be added for conflict resolution UI, side panel activity view, side by side record view, etc
Add searching activities to the main page
Add a list of resources with related conflicts to the main page
The parent ticket can be found here: #11
According to the mock https://user-images.githubusercontent.com/12927553/48443971-55f41d00-e758-11e8-96ad-6596c7a6092c.png add searching activities to the main page.
Add custom bean validator before saving mapped data
As the amount of data from partners is going to be really huge, it is required to validate mapped entities before saving them. Otherwise, none of the data will be persisted, because there was a DB exception with a single entity. Doing it manually with each entity is not the best solution, so we should use something like Spring Bean Validation to handle this. It would be nice to have details about beans that were validated with errors stored in the database, so it could be verified and fixed then, but fixing those errors should be a separate feature.
This should be done after #176
Map data from LAAC to a common structure
An adapter for data from partner LAAC should be implemented, with a wiki page, describing how it's mapped.
Introduce a Bootstrap theme
Select an existing, free Bootstrap theme and introduce it to the site in order to change the default bland JHipster look.
Remove AnonymousDataAdapter
When there will be a data adapter for another provider (which uses a file based data) available, the AnonymousDataAdapter should be deleted - no resources should be spent for updating it, as it might not be used anymore, and keeping not up-to-date classes might lead to confusions. AnonymousDataMapper and its data model should be deleted as well. The proper insert in the database as well. It might be restored in the future, so it should be done within one commit, and it's the main reason for a separate issue for such a small change.
Add a list of activities to the main page
The parent ticket can be found here: #11
According to the mock https://user-images.githubusercontent.com/12927553/48443971-55f41d00-e758-11e8-96ad-6596c7a6092c.png add a list of all resources that have been uploaded/read from the logged-in Database Manager’s system.
Acceptance criteria:
- the list should show the same level of information as on the mock,
- the list does not have to look exactly the same as the mock,
Add backend for uploading files
All uploaded files should be sent to the server and be persisted in the non-relational database. Use the provided upload page for this. Only authorized users should be able to upload a file. If the file is in a different format than JSON, it should be converted before saving.
Generate Service.Net starter with JHipster
Use JHipster to generate:
- react/redux + typescript
- Java Spring Boot Backend
- PSQL as the database
- Docker container support
Prepare matching algorithm architecture
The purpose of this issue is to prepare architecture for the matching algorithm. The usage of comparing methods should be used in tests for now. It will be plugged to the app later. See https://github.com/benetech/ServiceNet/wiki/Matching-algorithm-approaches for more details about the algorithm.
Add static analysis tools
Add static analysis tools to the repository:
- PMD
- SpotBugs (if possible for Java 11)
- Checkstyle
- ESLint (adjust if added by JHipster)
use rules from: https://github.com/SolDevelo/static-code-analysis-rules
Create data file upload screen
There should be available data file upload page. For authorized users, link to this page should be visible in a navbar. The user should be allowed to select a file using file dialog or drag it into the page. After the file is loaded, there should be a button to confirm its upload available, but no action has to be attached to the button at the moment.
Update the application to Java 11
JHipster doesn't support creating an app with Java 11, so it has to be converted manually. This link should be helpful: https://gist.github.com/mraible/716f4d2ab205c0e747d51119fc5d9cd8#gistcomment-2598658
Implement names matching algorithm
See https://github.com/benetech/ServiceNet/wiki/Matching-algorithm-approaches for details, and implement name and alternate name matching algorithms.
Define database schema
Generate schema from JDL definition
Generate schema code from the schema definition. Take this steps:
- Before generating defined dtos/services for the schema. We can have he admin UI for each of the entity types defined. Use mapstruct for dtos.
- Manually add relations to jhi_user mentioned as comments in the schema
- Replace integer primary keys with UUID primary keys
Add reports for data updates
Uploading a data should produce a report, so it can be later reviewed. It can be displayed in a simple table, for example showing how much entities were updated, when the upload was done. Maybe info about how long it takes to persist the data or to compare it with other ones (looking for conflicts). What is most important for it to be easily extendible, as in the future more data might be reported. The report should be generated both in the scheduler while collecting the data with the API and when uploading data files manually.
Create primary multiple record list view
this is the parent task for creation of the main home page which will show a list of the currently logged in user's records (with matches when applicable).
More specific tasks will be added to allow for sorting, searching, filtering, map view, etc. beyond just the basic list view.
Add user profile to set default views
Load partner names dynamically on file upload dropdown
The list of partners to select should be loaded dynamically from the backend (all system accounts). As a part of this ticket, all partners from stage 1 of the project should be loaded to the database automatically when the app starts.
Ability to filter multi record view
Sub of #11, feel free to break into sub issues if you'd like
Filter by time frame, should probably be radio buttons
Once we figure out how to normalize this, ability to filter service type:
Ability to filter by which partner DB the match was found in:
Add sorting activities to the main page
The parent ticket can be found here: #11
According to the mock https://user-images.githubusercontent.com/12927553/48443971-55f41d00-e758-11e8-96ad-6596c7a6092c.png add sorting functionality.
Default sort should be based on the number and age of updates of matching records in partners’ databases.
Sorting options:
- the number of updates,
- age of updates.
Add sorting to multi record view
sub of #11
Should add the ability to sort by:
- most activity (a record with many matches and/or many updates)
- most recent updates
- Maybe at a later date, ability to sort by distance. I think once this option is selected, an additional input box would need to be displayed to enter a zip code or address
Support upload data in parts
Some data will be uploaded in few dependent to each other files. We should handle those uploads, to make the data consistent.
Map data from ShelterTech to a common structure
An adapter for data from partner ShelterTech should be implemented, with a wiki page, describing how it's mapped.
Add MongoDb to the project
- Add MongoDB to docker configuration (compose)
- Add MongoDB connection configuration to config files
- Include MongoDB on the health checks page
- Introduce Spring Data code that can be later on used to consume Mongo: https://spring.io/projects/spring-data-mongodb
Implement description matching algorithm
See https://github.com/benetech/ServiceNet/wiki/Matching-algorithm-approaches for details, and implement a description matching algorithm.
Create wiki page describing First Provider Data Mapping
To ensure mapping is done properly, all mapped fields should be described in the wiki page. It can be easily verified then, to detect potential problems.
Adjust the database schema to include multiple locations for organizations
Some organizations might have multiple locations - it should be included in the database schema. When it'll be available, some data adapters should be adjusted too - e. g. data from iCarol contains multiple locations for an organization, so all of them should be persisted.
Connect with Eden API and map their data to the common structure
After #65 will be finished, there has to be created a special scheduler to get data from the Eden API. This data has to be mapped to the common structure and persisted after the scheduler collects the data.
Add indicators for conflicting matches/updates from partners in multiview
Add a map view to the main page
The parent ticket can be found here: #11
According to the mock https://user-images.githubusercontent.com/12927553/48443971-55f41d00-e758-11e8-96ad-6596c7a6092c.png add a map view to the main page.
Add conflict view on single record screen
Add filtering activities to the main page
Add a list of resources with related conflicts to the main page
The parent ticket can be found here: #11
According to the mock https://user-images.githubusercontent.com/12927553/48443971-55f41d00-e758-11e8-96ad-6596c7a6092c.png add filtering activities to the main page.
Add field exclusions per partner settings
Some partners have expressed the desire to prevent certain fields from being shared/displayed to partners (contact related fields most commonly).
As a result, we've said that any partner who doesn't share a specific field won't be able to see that field from other partners either.
We will need to create a DB table to store partner+field exclusions. Those fields will then be removed from any view of that partner's records (both when viewing their own record, and when a partner views their records, and won't appear in suggested conflicts, etc).
These fields can however be used by the system for matching purposes (weighted into the probability but never displayed).
Create an admin usertype that can upload files as a different user/org
Connect Users with SystemAccounts
While creating a new User, it should be possible to select a SystemAccount for him. It should be also visible on the user management page.
Map data from SMCConnect to a common structure
An adapter for data from partner SMCConnect should be implemented, with a wiki page, describing how it's mapped.
Implement localization matching algorithm
See https://github.com/benetech/ServiceNet/wiki/Matching-algorithm-approaches for details, and implement localization matching algorithm.
Capture updates to entities as Metadata
HSDS uses the metadata entity to capture changes to entities. Add a hibernate based component that processes db transactions and is able to register these metadata changes to any resource (entity) changing.
- Define Hibernate based interceptor/event handler/other to create Metadata entries for any change happening to data
- Resource id should have the UUID of the resource referenced
Determine update activities architecture and update the wiki
Determine update activities backend architecture and add/update the wiki pages.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.