rtang03 / fabric-es Goto Github PK

The scope of this task includes:

• 2 VMs in AWS. One represents ETC, another one represents PBOC

• Each VM consists of:

  • a relay, which forwards requests to the endpoint
  • a mock server, which pretends the etc or pboc endpoint and listen to the requests

• The relay includes following functions:

  • forward requests to the endpoint
  • capture the requests and response information
  • publish information to redis

This task needs 1 week.

@pangduckwai reports a bug in here.

},
Loan: {
documents: catchErrors(
async ({ loanId }, _, { dataSources: { document } }: Context) =>
document.repo.find({ byId: loanId }).then(({ data }) => data),
{ fcnName: 'Loan/docuemnts', logger, useAuth: false }
),
},

Resolution:
Need to fix it similar getProjection (i.e. field search function).

Currently, no authentication is required for querying data.

Need to do it, similar to command-side.

paginated api, e.g. getByEntityName resolvers

paginated info retrieval from Redis

Consider that Redisearch's command FT.SEARCH provide "LIMIT" and "SORTBY" flag. This is natively numbered-paged pagination. We shall adopt numbered-paged pagination, instead of cursor-based pagination, for the sake of simple simple implementation.

However, will concurrently explore the need and way to go for cursor-based pagination, or infinite scrolling.

currently, the unit test of auth package requires to running compose.1org.px-db-red.yaml.

This is 1org compose.

The curent "Create_Release" workflow is 2org.

In order to add the unit test of "auth" package, it needs some refactoring.

Paul discover the bug:

Paul Lai 9:52 AM
and please note that before registerAndEnrollUser was working only because of the bug (the missing return)

Ross` Tang 9:54 AM

registerAndEnrollUser requires admin credential, to operate. gw-node use Org Admin ID, to get authentenicted from auth-server. After successful authentication, it get ‘is_admin’ true, and passing on to apollo context.

And, the password is not checked locally, and it is sent to /oauth/authenicate api

Paul Lai 9:56 AM

no it is not checked at all, the password is wrong...

9:57

actually 'requires admin credential' is not true also. because now my user's is_admin is false. if using your original code (check is_admin, create the ForbiddenError object, and let it garbage collect immediately), it works

It needs a new page, for end user to show/list his wallet entry, if existed.

Or, create / remove / enroll digit wallet.

getProjection was removed. Need to re-implement it with Redis.

• upgrade to Fabric v2.1
• upgrade sdk version
• explore major dependency, like Apollo server

This may take 3 days.

This task should start as pre-requisite of v0.6.2.

The previous create release workflow is based Fabric V1.4.x. While now running Fabric V2.0.1, it shall upgrade create release workflow accordingly.

The current implementation does not have UI for end-user to create digital wallet. It requires to using Apollo playground, to send mutation request "RegisterAndEnrol" towards "admin" micro-service, via federated gateway.

This task shall

• based on "ui-account" implementation, to extend ui for Digital wallet implementation
• integrate with gw-org1, to send mutation request "RegisterAndEnroll".

Be noted that, they are referring to the SAME procedure.

• in business sense, this is to Create Digital Wallet
• in technical sense, this is a 2 steps procedure: (1) Register Fabric CA Identity in CA server; and (2) Enroll the above identity onto the server-side wallet, of gw-org1 VM.

The technical implementation will be made with simple REST call.

After query-handler is done, it needs a simple dashboard application to visualise the off-chain public data from Redis.

Currently, there are multiple deployments, for different needs:

• lib-dev-net -> for library develop
• gw-dev-net -> for image building
• gw-test-net -> for k8s deployment

They are based on slightly different configtx.yaml and docker-compose.

And also, the "~/deployments/boilerplate" has a few issues

• it makes use of both local MacOS command, and insider container command. It leads to require sudo to execute commands.
• it places the generated genesis.block and config.tx in the middle non-generated fixture code. It is hard to clean-up, and does gitignore
• the service name is using dot separator; that make k8s incompatible.
• the scripts are nested hierarchy. It makes k8s definition too complex. If needs to reuse configuration for both docker-compose and k8s, need to flatter the hierarchy.
• the script is not catered for running unit & integration in CI. Need to re-organize them

The Goal to make a single consolidate dev-net for multiple purposes.

need to add security layer, pm2 to proxy server

Currently, all fabric write operation is not having invocation control.

For better reactivity, will add Rxjs timeInterval operator, to avoid burst of invocation.

Tentative, add 100ms interval.

Be noted that some test may fail.

Currently, the reconcile commit and computed entity in Redis is insufficient to provide data for use by dashboard.

Here needs information two categories of meta data.
(a) meta data added, BEFORE commit is written into Fabric.

• _creator: enrollmentId
• _created: created at timestamp in second
• _ts: last modified timestamp in second

(b) meta data added, AFTER commit is reconcile into Redis

• __event: event sequence, to compute the currentState
• __commit: commit sequence, to compute the currentState

These additions of meta-data will facilitate the addition of new api for use by Dashboard and Reporting.

#54 is done, next/final refactoring is required for gw-org* and model*. After which, we can go for merge to master.

Currently, CI does not validate dev-net is healthy; and assumes dev-net is working fine.

Will take this approach.

https://hasura.io/blog/best-practices-of-using-jwt-with-graphql/

There are many full text search attributes, which are not implemented.

This is time-consuming, or unnecessary way of search.

When later implementing the UI/dashboard function, #52 , it should further evaluate which additional feature of full text search are required.

Need to improve

• the index schema. e.g. what to include in the tag; and what to include in the eIdx and cIdx

As a reminder, by https://oss.redislabs.com/redisearch/Tags/

"...searching for a tag without any modifiers will not retrieve documents containing it... "

For fabric 2.1:

• 5 VM will be created for 2 organizations

For gw-org1 and gw-org2:

• 2 VM will be created for 2 organizations

This task needs 2 - 3 weeks

Current PR is tentatively accepted. There encounters below problem with grpc + ubuntu

https://stackoverflow.com/questions/49758008/nodejs-error-failed-to-load-grpc-binary-module-because-it-was-not-installed-fo

grpc/grpc#15431

As a workaround, now bypasses the unit-test of fabric-cqrs; those tests are runnable and passed in local MacOS.

Let check later will face with similar problem with "Create Image" workflow.

This issue may disappear if later upgrade to Fabric v2.0

Originally posted by @rtang03 in #11 (comment)

Today, the auth server cannot start, and report below error.

TypeError: Cannot set property EntityManager of #<Object> which has only a getter at Object.<anonymous> (/home/app/src/index.ts:141:9) at Module._compile (internal/modules/cjs/loader.js:778:30) at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10) at Module.load (internal/modules/cjs/loader.js:653:32) at tryModuleLoad (internal/modules/cjs/loader.js:593:12) at Function.Module._load (internal/modules/cjs/loader.js:585:3) at Module.require (internal/modules/cjs/loader.js:692:17) at Module.Hook._require.Module.require (/usr/local/share/.config/yarn/global/node_modules/require-in-the-middle/index.js:80:39) at require (internal/modules/cjs/helpers.js:25:18) at Object.<anonymous> (/home/app/src/entity/ApiKey.ts:1:1)

This is related to below break of TypeOrm
typeorm/typeorm#6054

Need to add below to the package.json of "auth" package.

"resolutions": { "tslib": "1.11.2" }

This is a step, after #20 is complete, and before k8s deployment #15

the "upload" micro-service shall

• accept file incoming streaming to mimic machine-to-machine upload. Should be something like S3 api.
• accept multi-form upload for ui form upload
• the micro-service should allow flexible configuration to using different storage operation; both OBS, and EBS.

Currently, both root level Readme, and those of library code are out-of-dated. Need to fix it so that the published library gives correct instructions.

To be elaborated

Currently, the create image workflow is lacking of fully covered unit and integration test. After finishing previous task about the "consolidated dev-net" (PR #5 ), will improve the "create-image" workflow, to make use of it; so that every release increment will have unit- and integration gates, before an official images are created.

This task will start after the publish registry #6 is done. And, be completed before end of sprint 2.

The README.md and OPERATIONS.md are required to update

After #20, or #54 is done, will attempt to RediSearch to to power the ProjectionDatabase, with full text search.

Even without RediSearch, the #20 will attempt to implementation some search capability based on pure Redis.

Currently, the CI with integration test is running, with hello-world tests. The workflow is there; without real tests. This task will migrate the existing integration tests for library code, into the "packages/tester", so that they are running as well.

Also, the pre-existing integration tests are broken, because we refactor to "consolidated" dev-net. Both docker compose service name, and url addressing in connection profile are in-correct. This task will fix them altogether.

Finally, integration tests can run locally, and during CI

In order to facilitate extended team to using our library code, and create more boilerplates projects. We need to publish library code into package registry.

A repo rename and namespace rename are required, before continuing. Need to wait other team member to check in all codes, to avoid unexpected trouble, when git origin is changed.

Planned this weekend; take 1 -2 days

Before further proceed to #19 , I find that this is good to introduce reverse proxy, for each org, to improve the intercommunication, and add ssl layer more easily.

Currently, will attempt to use 'http-proxy-middleware' npm library; instead of ngnix.

I don't have solid comparison between using ngnix, and http-proxy-middleware.

@pangduckwai @hohowin @github-hkicl-cof , if you have comparison, let me know.

For the sake of ease development, follow below step.

1. complete issue-20, which includes the all implementation of Redis-based querydb, and projection db, and their event-notification. And, Redisearch as well.
2. after (1) is done, the "query-handler" will be able to reconcile, subscribeHub, and notify events. This step 2, will further refactor, the "query-handler", to relocate duplicated implementation, BACK to "fabric-cqrs" and "gateway-lib".
3. the query-handler will be trimmed down, and made to "query-handler" docker image.

This is partly historical coding practice, what Fabric's returning result are in form of Record<string, Commit>.

When migrating from in-query database to Redis. In order to minimise the change impact, some returning result from Redis are in TEntity[] and others in Record<string, TEntity>.

This discrepancy make ugly and unnecessary conversion.

Consider that Redis is natively using only array, and also Apollo resolver is friendly with array.

This task will revisit and attempt to change the Query-side operation to using array, for sake of cleaner code. The command side will remain unchanged.

A special attention is required to maintain the immutability requirement for Redux.

Given #6 is complete, we shall proceed to create the templated repository, for external parties to proceed development.

Currently, the unfinished templated repo is at https://github.com/fabric-es/fabric-es

It will fill in deployment network, and gw-org1/org2 sample code.

Currently, we are using in memory queryDB and projectDB. It makes the docker container runtime bigger and bigger, and being stateful container.

This task shall detach in memory queryDB, into Redis, so that the gw-org will be stateless again. This task is medium complexity, will take 2- 4 weeks to complete.

@pangduckwai created a dev-net2 (3org), and new naming convention. Need to revisit it, and refactor the library package, so that the underlying tests can run on this 3org network.

Also, it need to refactor CI and create_image workflow to change from 2org to 3org.

After enhancement, the starting network will be 3org; and is also the base topology for reference implementation.

It is critical for the postgres DB named "auth_db" to be created prior to start the auth service. Currently, the auth docker image does not have any checking of that.

It is suggested to add checking in auth docker image by meanings of an entrypoint.sh script.

Based on new technical design, this auth package will revamp.

In current release, the callback URL website is incomplete.

And, note that previous unit test was done successfully, with oauth calls. But it is well integrated, nor tested against the latest gw-org* implementation.

Lastly, in current unit-test in CI workflow, the unit-test is temporarily removed. It needs to add it back.

This task is considered done, after successful release is made, and gw-org* can make use of it.

After #54 is done, it will enable authentication, by integration with auth-server.

(1) ui-account does not handle properly, when access_token in auth-server expires. And, the ui-account will get 401 status code.

(2) after logon, and then click /home, the /home does not know this is authenticated user. This is because withAuthSync HOC has bug, and is removed.

Current deployment is made upon plain VM. A preliminary assessment shows a more robust production grade deployment should be based k8s. Previously experiments was successfully made to using local machine single node k8s cluster.

This task shall extend "dev-net" to be deployable to both docker-compose, and k8s.

This task is considered successful, when reference implementation is made on k8s on GCP. This is mid-term goal; medium priority.

Including:

• https
• WAF

This task needs 2 weeks.

ui-control is the ui for query-handler. It is not part of dev-net deployment.

The http-middleware-proxy is weak in security. In order to better fulfill the production-grade security requiremen; will switch to Ngnix.

Currently, the once token is obtained and set in cookie, it is removed unless explicit logout.

We need, for every page reload/refresh, it should validate the token, to ensure its validity. And, if invalidated, need to remove this token, even not clicking logout.

And, evaluate if we should use refresh token.

It will take multiple steps fabric v2.

There are a number of new features, deserved for this upgrade.

• enhanced private data. Currently, I don't exactly understand the new feature, especially how it impacts us. We need to figure out.
• external chaincode launcher. This may save us from cc deployment workflow, when the consortium grows.
• update nodejs dependency. Hopefully, it will also fix grpc issue with ubuntu.

Additional considerations:

• the new chaincode lifecycle is not useful, is likely to stick to the old.
• the fabric-sample is missing the network operator example.
• This is questionable, if the node-sdk is good quality. Previous attempt on V2 beta indicates the sdk is having lots of issues.

We need to determine if all packages upgrade should fall into sprint 3. Timing-wise, this is good timing. Priority-wise, not a must-do priority.

Therefore, we may attempt to split the upgrade into (a) deployment (current task), and (b) packages. If (a) is smooth, then go to (b).

after the authentication to done on ui-control, for each query request. #81 #77 #85

We then implement authenticated subscription upon Event Arrival.

In case of unexpected situations, such as redis down, or endpoint down, how to handle.

This task takes 1 week.