keepid / keepid_server Goto Github PK

Hi backend,

I updated the .env file for encryption, and I also uploaded the google-credential file (google-credentials.json) in the google drive (security->Google KMS) that you need to have on your local machine (And make sure you don't add it to Git!!).

thanks :)

PDFControllerUnitTests works in terms of the code working (at this time). However,with the new changes to the service model, the methods are no longer static and need to be called using an instance of the service. It is probably best to make these into integration tests to follow the pattern of PDFControllerIntegrationTests for the other services.

https://www.figma.com/file/Ed03XkWo2YzJPr1hVADreW/ID-Categorization?node-id=216%3A133

While you are doing this, do you think you could start refactoring some of the current unit tests, especially around login. Use the new EntityFactory.

Make a report template for all of our support report types

I would start with testing if the href targets actually exist on the emails on all the functions in Emailutils. In addition, think about what kinds of email-related error codes could occur (including what happens if the file is missing? rn we are just returning null, but that is not very descriptive). I would take a look at User.UserMessage enums to try to get some ideas, and to copy the toJSON() method.
I would make a new folder in the tests folder and call it Security and make a new file called EmailUtilsUnitTests. You can look at ValidationUtilsUnitTests.java for reference.

This should take probably one or two days I think, and in the meantime I will look for other pieces of the code you can work on.
One huge thing is thinking about how to build a dashboard, and how we can query data on the backend. You should try to skim this documentation here. When you are reading, you should select the Java (sync) driver tab at the top of the sub articles.
https://docs.mongodb.com/manual/crud/

Basically, the dashboard will probably be querying our mongo database, so knowledge of queries will be useful.

Do research on the documents that nonprofits need to generate. See a couple links:

https://files.hudexchange.info/resources/documents/2020-HIC-and-PIT-Count-Data-Submission-Guidance.pdf
https://files.hudexchange.info/resources/documents/CoC-Analysis-Tool-Overview.pdf
https://files.hudexchange.info/resources/documents/Record-Report-RRH-Bed-Inventories.pdf

Also reports:
https://www.hudexchange.info/programs/coc/coc-giw-reports/
https://www.hudexchange.info/programs/coc/coc-performance-profile-reports/
https://www.hudexchange.info/programs/coc/coc-homeless-populations-and-subpopulations-reports/
https://www.hudexchange.info/programs/coc/coc-housing-inventory-count-reports/
https://www.hudexchange.info/programs/coc/coc-dashboard-reports/

@avwu99 @cccyyyr for now if you really want to look into these, you can, but its not a requirement
@JaxFoltz and @ivorine85 might be good for you guys to do a deeper dive on reports and things. It would be ideal if you could get back to us by 8/14

When you create an account for a new organization and try to press submit as the last button, it does not work

Given the file type generator, implement the full route, where the post request would come in and the following would occur:

Route to the correct file generator
Store a copy of the report in mongodb
send a blob of the file to the front end, so the front end downloads the report

Add tests

See: keepid/keepid_client#29
Basically, we receive an input string from the front end, and then check the mongodb database if that username already exists in the DB.

The code for checking the existing username is similar to this:
MongoCollection userCollection = db.getCollection("user", User.class);
User existingUser = userCollection.find(eq("username", username)).first();

    if (existingUser != null) {
      logger.error("Username already exists");
      ctx.json(UserMessage.USERNAME_ALREADY_EXISTS.toJSON().toString());
      return;
    }

You will need to create a function in UserController which takes in a request, get the username from the body, validate it against the db, and return a response code (you could send a boolean back or a UserMessage, whichever you think is better). Then you need to add that route to AppConfig

This class is the backbone of all encrypting. Refer to the SecurityUtils section of the security design document (listed below). For now, we don't have to worry about generating credentials. Instead, just have the credential file saved locally and use that for the time being. You can find the keepid-google-kms.json credential file in the google drive: Keep.id -> technical (Eng and Design) -> Engineering -> Security -> Google KMS

If anyone decides to start on this, just write a quick comment on this thread, so we don't have multiple versions of the class being implemented.
Check Comments before starting on this issue. If someone started on it, communicate with them and coordinate before opening another branch

Reference:
Security Design Document
Old encryptionUtils

Will be making a security branch. If you don't see it feel free to create it and base it off of master :)

Make a backend function that displays a list of all unannotated pdf’s
Accepts a new parameter which is whether to get unannotated pdf’s or not

Right now, we have a front end flow that will indicate what type of file is being uploaded. See figma: https://www.figma.com/file/Ed03XkWo2YzJPr1hVADreW/ID-Categorization?node-id=216%3A133
See this for where the documentType is passed in: https://github.com/keepid/keepid_client/blob/master/src/components/Documents/UploadDocs.tsx#L94

handle the extra parameter and save it as an optional field in the pdffile class as well as the form class.

Basically, query the database for all the organizations that we have:
What we need:

Org Name
Org Website
Org Address, city, state, zip code
Org Email
Org phonenumber

see: https://www.figma.com/file/7GXv80XxYvbHeHvMNz3HPB/Find-Organizations?node-id=5%3A636

Pagination will be added later

After this, write a test that can check for all the orgs in testUtils

Currently, the getMembers Handler in the UserController is responsible for 3 things:

1. Retrieving the members of an organization
2. Searching for members of an organization with first name and last name matching
3. Paginating the results

Server-side pagination has limitations. It requires querying the database ever time a page changes. Returning an array with all of the users to the frontend is not too expensive, and the frontend can utilize browser caching to speed up same-page visits. We should eliminate server-side caching altogether in this Handler.

Furthermore, the search functionality should be separated from the retrieval functionality. Searching for members of an organization should be implemented in a separate Handler that does only this. Also, consider changing the current Mongo query that does searching.

Note that there are two parts of searching: typing in the search query and hitting submit. While a user is typing, the frontend will send requests to the backend to receive recommendations. The backend should just send back the first 5 (or so) reasonable names.

On submit, all user info that matches the search query should be sent back, and the frontend will perform pagination.

In short, the three new Handlers should:

1. Retrieve all the members of an organization
2. Retrieve the top 5 (or so) names in an organization that match the search query
3. Send back all user info that matches the search query in an organization

The load-pfp seems to not always be returning the most recent pfp but instead going through an array (or something like that) and sometimes it will loose it's place and then send back previously uploaded photos (still maintaining the chronological order just not giving back the most recent photo)

I attached a zip file of a video showing the issue i'm talking about
returning_wrong_image.mov.zip

Also, the get activities seems to take a while in returning a value even if there aren't any activities. I haven't looking into this so much, it can be possible it's something on my end (frontend). But if you think there's any way to make this process faster then that would be great!

Lets start planning what that looks like here. It might also be good for @kofmangregory to write up a TDD for this. One approach is in memory DB, another approach is making a systematic way of mocking the DB using mockito.

We need an abstraction that can query activities from the db. This one might be more difficult, as we have to think about what type of activity we need. We can either lump all the activities together into one ActivityDaoImpl/ActivityDaoTestImpl and query it, or we could have a Dao representation for each Activity type that implements an ActivityDao interface.

Create a developer role

Use Google Keys

Look into file type generation:
Probable file types: .pdf, .csv, .xls, .docx?

I would look into generators for each of these file types, and start building out a rendering engine that would take in some request for generation through the api, such as
POST /generate-report
params:
-orgName or orgId or some identifier
-type of report to generate, distinguished by file type

Copy from the UserDao in the training server, and write a simple Dao object layer for the organization and getting organizations in the database.

Will post more details

End to end integration tests to test entire fileDao flow

Convert from json to gson library

Right now, PDFType takes a couple different forms:

• APPLICATION("application")
• IDENTIFICATION("identification")
• FORM("form")

However, this is confusing. What is the difference between application and form? What about an un-annotated form and an annotated form that is ready to use?

I think we need to meet with business team to specify all the specifics.

In addition, we will need to create a PdfDao. However, I think because Pdf functionality is pretty important to our application, it might make sense to create a PdfDocument interface and then have ApplicationPdf, IdentificationPdf, and FormPdf as three classes that implement the PdfDocument interface. Then, we can handle each case of PdfType without making it super confusing, but also having a standardized interface (for example when we need to convert Pdfs into inputstreams or when we need to run them through the PdfBox engine).

In applications, receive the signature image and put it into the PDF

@cccyyyr you can put your google document here. I have also tagged @steffen12 and @JaxFoltz for this

Person can upload an annotated pdf and it will replace the old pdf form
Mark in the database now that the pdf is annotated

Understand the PDFController code, especially the code that is related to pdf annotation
Understand the ApplicationForm.tsx react code, which uses the pdf annotation code on the server

While migrating our backend server to Daos, we need to update our old implementation of search. I will push a branch with some tentative changes.

It would be also good to implement fuzzy search here? That could be an interesting problem. Maybe you could implement this: https://en.wikipedia.org/wiki/Levenshtein_distance

To get word similarity? There are other alternate ways including Cosine similarity and Jaccard similarity that you can look into.

For UserController, OrgController, etc. a lot of fields come in from the request and from a session, and we directly do actions depending on this information. However, we should separate these two layers. One example of this is login, where we authenticate based off a username and password. The handler gets the information from the request, and then calls the argon2 password code right in the handler. However, we should separate this into two layers: the controller and the UserLoginService class.

UserController.loginUser()

UserLoginService(username, password)

The UserController will just obtain the variables from the request and perform validation on the controller level
The UserLoginService will do all the business logic.

Then, instead of making tests that have to do things like Unirest POST the server, we can just test the UserLoginService() directly and pass in our own fields

This should definitely make testing like 1000x easier also

When a new PDF form is uploaded, flag it in the database as unannotated

We need to add support for 2nd parties being able to delete and download files for clients. I've added support for get-documents and upload already in pdf controller, so it should be pretty similar to those two. Those updates have been merged into main, so you can create a new branch off of main and add to the pdf controller java file. If you have any questions, reach out to me for technical help, and to @loafyyy for design oriented questions :)

Right now, we need to make everything communicate using JSON - it is actually pretty messed up right now.

On the server, we use a mix of ctx.json and ctx.result
And even within ctx.json, sometimes we are sending a string, and sometimes JSON object

On the client, we use JSON.parse() and result => result.json(), while we should not use JSON.parse at all

We need to standardize this

Basically, I want to create a heirarchy of Activities where the ultimate parent is the Activity class
Under that, there are the CreateOrgActivity, CreateUserActivity, and UserActivity

Under CreateUserActivity, there will be a CreateClientActivity, a CreateWorkerActivity, a CreateAdminActivity, and a CreateDirectorActivity

Under UserActivity, there will be AuthenticateActivity, DocumentActivity, ChangeUserAttributesActivity,

Under DocumentActivity, there will be UploadActivity, DownloadActivity, DeleteActivity, and ViewActivity

public abstract class Actvity
Time occurredAt;

public class CreateOrgActivity extends Activity
Organization org

public class CreateUserActivity extends Activity
public class CreateDirectorActivity extends CreateUserActivity
public class CreateAdminActivity extends CreateUserActivity
User creator
public class CreateWorkerActivity extends CreateUserActivity
User creator
public class CreateClientActivity extends CreateUserActivity
User creator

public class UserActivity extends Activity
User user

public class AuthenticateActivity extends UserActivity
public class LoginActivity extends AuthenticateActivity
boolean isTwoFactor
public class PasswordRecoveryActivity extends AuthenticateActivity
String oldPasswordHash
String newPasswordHash
String recoveryEmail

public class DocumentActivity extends UserActivity

public class ChangeUserAttributesActivity extends UserActivity
String attributeName
String oldAttributeValue
String newAttributeValue

See https://www.baeldung.com/java-8-date-time-intro for info on Time - we want to move away from java Date in general in favor of the new Java Time objects

We are embarking on a journey to eliminate our unit test debt. The start of this journey is figuring out how much unit test debt we have.

For each publicly exposed method M in our controllers and utility files, it would be great to assess the following:

1. Is each unit of work done by M tested?
2. Does each test case that tests M test a single unit of work?

If the answer to these questions is "No" for a substantial number of methods, then we should answer the following:

3. Does our testing structure make it difficult to write complete tests, or have we just been lazy with the completeness of our tests?
4. Can we design a system for writing tests that makes it easier to test each unit of work done by each method? What are the current limitations to this?

For some reason, half the time this test passes and the other half the time it fails. In addition, it really likes to fail on travis and work locally. We should make this test more consistent.

We need better tests around Activities in general.

see https://www.figma.com/file/Ed03XkWo2YzJPr1hVADreW/ID-Categorization?node-id=216%3A133

See:

Research
#17

Deciding File Formats
#18

Generate Templates
#20

Full implementation
#21

Still need to do int tests for Application related routes

• Finish unit tests for services, will need to mock Stripe Object responses
• Make front-end UI changes according to Jonathan's Figma here
• Add alerts to display errors to the user instead of console.log statements in the console
• Move priceIds at PricingPage (front-end) to .env file so we don't hardcode the IDs into the code
• Change all responses for fetch statements to be based off response string from back-end
• Clean up back-end code and ensure apiKey is being used properly

We want to indicate the login history of all users that can be seen in their access history under account settings. I think what would be the most helpful would be to indicate:

• Date
• IP Address (and we can pull the physical address from there, see below)
• maybe device (if that isn't too difficult, read here: https://stackoverflow.com/questions/8515161/detecting-device-type-in-a-web-application and here https://developer.mozilla.org/en-US/docs/Web/HTTP/Browser_detection_using_the_user_agent)

I would sign up for https://ipinfo.io/
Here is the github https://github.com/ipinfo/java

it seems they have a free tier of up to 100k uses/month, which should suit us fine

Basically, when a login gets triggered, then store all this information in the db, maybe have an array in the user object that gets appended, with maybe a max length of like 1000 (to store the last 1000 logins).

Then, add a POST route (maybe call it /get-login-history or something) which will then query the db for the user object and obtain the array. Once this is done, we can work on the front end.

If you login as a client with username: Wormtongue and password: tongues0fFire, go to applications and then go to the first pdf file listed, it throws a SERVER_ERROR response. For some reason,

is throwing an IOException on that specific PDF file, which is then returning lots of errors to the front end. I believe this is a backend problem, and would like someone to diagnose.