openreferral / servicenet Goto Github PK

As known, data files from different providers have a different structure. We should map those structures to our entities and persist them in the database.

As we discussed on our call, the home page should display at most 3 lines of updates per record cluster.

For records that have greater than three updates, we should choose the three 'highest priority' updates to display in full and then display text indicating there are more, e.g. "There are 5 additional updates for this record"

The ranking for update type:

'Recommendations' are always higher priority than 'Updates'

'Updates' are always higher priority than 'Conflicts'

The ranking for fields:

1. Address
2. Phone
3. Name
4. Service Type
5. All other fields

Recommendations with a higher number of partners should outrank those with only a single matching partner. Same for Updates and Conflicts.

For each provider, all entities should be persisted only if they're unique.

Acceptance Criteria:

1. Same data, persisted twice results in no new entities
2. There are no unwanted duplicated entities within one data upload

Using the representative data from the data repo, draft an architectural suggestion for approaches around how we can match records across various partner datasets with different structures/formats/fieldnames etc

Some of the partners will provide their data as an API that can be queried. It'll have to be downloaded automatically after a specific period of the time, but there has to be an option to manually trigger it as well. Probably the best way to achieve it is to have a simple page for scheduler administration. There should be listed all schedulers with a button to automatically trigger them. If it can be easily done, there could also be information about previous and next trigger dates.

This should include a list of historical updates by partner systems

Clicking the 'See Full' or Activity tab at top:

Plat locations on map and enable clicking a pin to display single record summary (which can be clicked to open single record view). Also has search bar to bring you to a specific pin

In side panel add an overview of matches in partner systems:

Clicking, always shows logged in user's record on left - matching record (complete original record, with normalized fields where possible) on right. When there are multiple matches, the user should be able to flip between records on the right side. We should have a feedback mechanism for each match allowing a user to manually remove a match when they think it's wrong - this should remove the match relationship in our DB (and maybe negate weights if we ever get to ML)

Placeholder parent task for the single-record view a logged in user gets when clicking on a single row in the list view.

More detailed tasks will be added for conflict resolution UI, side panel activity view, side by side record view, etc

Add a list of resources with related conflicts to the main page
The parent ticket can be found here: #11
According to the mock https://user-images.githubusercontent.com/12927553/48443971-55f41d00-e758-11e8-96ad-6596c7a6092c.png add searching activities to the main page.

As the amount of data from partners is going to be really huge, it is required to validate mapped entities before saving them. Otherwise, none of the data will be persisted, because there was a DB exception with a single entity. Doing it manually with each entity is not the best solution, so we should use something like Spring Bean Validation to handle this. It would be nice to have details about beans that were validated with errors stored in the database, so it could be verified and fixed then, but fixing those errors should be a separate feature.

This should be done after #176

An adapter for data from partner LAAC should be implemented, with a wiki page, describing how it's mapped.

Select an existing, free Bootstrap theme and introduce it to the site in order to change the default bland JHipster look.

When there will be a data adapter for another provider (which uses a file based data) available, the AnonymousDataAdapter should be deleted - no resources should be spent for updating it, as it might not be used anymore, and keeping not up-to-date classes might lead to confusions. AnonymousDataMapper and its data model should be deleted as well. The proper insert in the database as well. It might be restored in the future, so it should be done within one commit, and it's the main reason for a separate issue for such a small change.

The parent ticket can be found here: #11
According to the mock https://user-images.githubusercontent.com/12927553/48443971-55f41d00-e758-11e8-96ad-6596c7a6092c.png add a list of all resources that have been uploaded/read from the logged-in Database Manager’s system.

Acceptance criteria:

• the list should show the same level of information as on the mock,
• the list does not have to look exactly the same as the mock,

All uploaded files should be sent to the server and be persisted in the non-relational database. Use the provided upload page for this. Only authorized users should be able to upload a file. If the file is in a different format than JSON, it should be converted before saving.

Use JHipster to generate:

• react/redux + typescript
• Java Spring Boot Backend
• PSQL as the database
• Docker container support

The purpose of this issue is to prepare architecture for the matching algorithm. The usage of comparing methods should be used in tests for now. It will be plugged to the app later. See https://github.com/benetech/ServiceNet/wiki/Matching-algorithm-approaches for more details about the algorithm.

Add static analysis tools to the repository:

• PMD
• SpotBugs (if possible for Java 11)
• Checkstyle
• ESLint (adjust if added by JHipster)

use rules from: https://github.com/SolDevelo/static-code-analysis-rules

There should be available data file upload page. For authorized users, link to this page should be visible in a navbar. The user should be allowed to select a file using file dialog or drag it into the page. After the file is loaded, there should be a button to confirm its upload available, but no action has to be attached to the button at the moment.

JHipster doesn't support creating an app with Java 11, so it has to be converted manually. This link should be helpful: https://gist.github.com/mraible/716f4d2ab205c0e747d51119fc5d9cd8#gistcomment-2598658

See https://github.com/benetech/ServiceNet/wiki/Matching-algorithm-approaches for details, and implement name and alternate name matching algorithms.

Generate schema code from the schema definition. Take this steps:

• Before generating defined dtos/services for the schema. We can have he admin UI for each of the entity types defined. Use mapstruct for dtos.
• Manually add relations to jhi_user mentioned as comments in the schema
• Replace integer primary keys with UUID primary keys

Uploading a data should produce a report, so it can be later reviewed. It can be displayed in a simple table, for example showing how much entities were updated, when the upload was done. Maybe info about how long it takes to persist the data or to compare it with other ones (looking for conflicts). What is most important for it to be easily extendible, as in the future more data might be reported. The report should be generated both in the scheduler while collecting the data with the API and when uploading data files manually.

this is the parent task for creation of the main home page which will show a list of the currently logged in user's records (with matches when applicable).

More specific tasks will be added to allow for sorting, searching, filtering, map view, etc. beyond just the basic list view.

If a user wishes to set a default view for their multi record view (instead of having to set filters every time) we should have a tab to allow for that. This should populate the dropdowns in the multiview so they can do one-off exceptions here and there.

The list of partners to select should be loaded dynamically from the backend (all system accounts). As a part of this ticket, all partners from stage 1 of the project should be loaded to the database automatically when the app starts.

Sub of #11, feel free to break into sub issues if you'd like

Filter by time frame, should probably be radio buttons

Filter by activity type

Filter by location:

Once we figure out how to normalize this, ability to filter service type:

Ability to filter by which partner DB the match was found in:

The parent ticket can be found here: #11
According to the mock https://user-images.githubusercontent.com/12927553/48443971-55f41d00-e758-11e8-96ad-6596c7a6092c.png add sorting functionality.

Default sort should be based on the number and age of updates of matching records in partners’ databases.

Sorting options:

• the number of updates,
• age of updates.

sub of #11

Should add the ability to sort by:

• most activity (a record with many matches and/or many updates)
• most recent updates
• Maybe at a later date, ability to sort by distance. I think once this option is selected, an additional input box would need to be displayed to enter a zip code or address

Some data will be uploaded in few dependent to each other files. We should handle those uploads, to make the data consistent.

An adapter for data from partner ShelterTech should be implemented, with a wiki page, describing how it's mapped.

• Add MongoDB to docker configuration (compose)
• Add MongoDB connection configuration to config files
• Include MongoDB on the health checks page
• Introduce Spring Data code that can be later on used to consume Mongo: https://spring.io/projects/spring-data-mongodb

See https://github.com/benetech/ServiceNet/wiki/Matching-algorithm-approaches for details, and implement a description matching algorithm.

To ensure mapping is done properly, all mapped fields should be described in the wiki page. It can be easily verified then, to detect potential problems.

Some organizations might have multiple locations - it should be included in the database schema. When it'll be available, some data adapters should be adjusted too - e. g. data from iCarol contains multiple locations for an organization, so all of them should be persisted.

After #65 will be finished, there has to be created a special scheduler to get data from the Eden API. This data has to be mapped to the common structure and persisted after the scheduler collects the data.

Every record with matches should indicate the number of partner system having matches as well as which fields have conflicts/updates. When more than one partner has the same data, the partner names should be listed in a hover modal

The parent ticket can be found here: #11
According to the mock https://user-images.githubusercontent.com/12927553/48443971-55f41d00-e758-11e8-96ad-6596c7a6092c.png add a map view to the main page.

At first this will be read only, so we probably wont need the check and x at first

Add a list of resources with related conflicts to the main page
The parent ticket can be found here: #11
According to the mock https://user-images.githubusercontent.com/12927553/48443971-55f41d00-e758-11e8-96ad-6596c7a6092c.png add filtering activities to the main page.

Some partners have expressed the desire to prevent certain fields from being shared/displayed to partners (contact related fields most commonly).

As a result, we've said that any partner who doesn't share a specific field won't be able to see that field from other partners either.

We will need to create a DB table to store partner+field exclusions. Those fields will then be removed from any view of that partner's records (both when viewing their own record, and when a partner views their records, and won't appear in suggested conflicts, etc).

These fields can however be used by the system for matching purposes (weighted into the probability but never displayed).

While creating a new User, it should be possible to select a SystemAccount for him. It should be also visible on the user management page.

An adapter for data from partner SMCConnect should be implemented, with a wiki page, describing how it's mapped.

See https://github.com/benetech/ServiceNet/wiki/Matching-algorithm-approaches for details, and implement localization matching algorithm.

HSDS uses the metadata entity to capture changes to entities. Add a hibernate based component that processes db transactions and is able to register these metadata changes to any resource (entity) changing.

• Define Hibernate based interceptor/event handler/other to create Metadata entries for any change happening to data
• Resource id should have the UUID of the resource referenced

Determine update activities backend architecture and add/update the wiki pages.