The *.orgbook.gov.bc.ca DNS records have been created. Update the domain names in the configurations.

The OrgBook:

• dev.orgbook.gov.bc.ca

• dev-api.orgbook.gov.bc.ca

• test.orgbook.gov.bc.ca

• test-api.orgbook.gov.bc.ca

• orgbook.gov.bc.ca

• api.orgbook.gov.bc.ca

• www.orgbook.gov.bc.ca

Environments to update:

• devex-von-bc-registries-agent-dev
• devex-von-bc-registries-agent-test
• devex-von-bc-registries-agent-prod
• devex-bcgov-dap-dev

The following error occurs during the initialization_and_load_tasks / bc_reg_corp_loader / load_and_process_bc_reg_corps / load_bc_reg_data_a process. The affected record is not marked with process_success = 'N' so the same record is picked up during subsequent processing cycles, effectively blocking all processing.

/data-pipeline/.venv/bin/python3 -u "bcreg/process-corps-generate-creds.py"
Caching data for parties and events ...
Caching data for corporations ...
timestamp out of range: "5483-12-03 00:00:00 BC"
CONTEXT:  converting column "firm_lp_xp_termination_date" for foreign table scan of "corporation", row 997
None
timestamp out of range: "5483-12-03 00:00:00 BC"
CONTEXT:  converting column "firm_lp_xp_termination_date" for foreign table scan of "corporation", row 997
None
Error during caching operation, switching to smaller cache size
Processing: 15.952490074036177
Traceback (most recent call last):
  File "/data-pipeline/bcreg/bcregistries.py", line 624, in get_bcreg_sql
    cursor.execute(sql)
psycopg2.DataError: timestamp out of range: "5483-12-03 00:00:00 BC"
CONTEXT:  converting column "firm_lp_xp_termination_date" for foreign table scan of "corporation", row 997
Traceback (most recent call last):
  File "/data-pipeline/bcreg/eventprocessor.py", line 938, in process_corp_event_queue_internal
    bc_registries.cache_bcreg_corps(specific_corps)
  File "/data-pipeline/bcreg/bcregistries.py", line 524, in cache_bcreg_corps
    self.cache_bcreg_corp_tables(specific_corps, generate_individual_sql)
  File "/data-pipeline/bcreg/bcregistries.py", line 575, in cache_bcreg_corp_tables
    _rows = self.get_bcreg_table(corp_table, corp_num_where, '', True, generate_individual_sql)
  File "/data-pipeline/bcreg/bcregistries.py", line 652, in get_bcreg_table
    return self.get_bcreg_sql(table, sql, cache, generate_individual_sql)
  File "/data-pipeline/bcreg/bcregistries.py", line 624, in get_bcreg_sql
    cursor.execute(sql)
psycopg2.DataError: timestamp out of range: "5483-12-03 00:00:00 BC"
CONTEXT:  converting column "firm_lp_xp_termination_date" for foreign table scan of "corporation", row 997
succeeded,  16 seconds

Remove the route to the Mara console for now, until we get some authentication in front of it. Add a script that can be used with authentication to get the latest loading stats - e.g. a script that use the OpenShift tunnel interface.

Integrate ZAP Scanning, starting with baseline scans.

Add rocket chat web hooks:

• 3 levels - error, warning, info

Migrate Jenkins pipeline scripts over to using the OpenShift Client Jenkins Plugin

• https://github.com/openshift/jenkins-client-plugin

Much, if not all, of the OpenShift Pipeline Plugin will not be supported against OpenShift API Servers later than v3.11.

Reference:

• Pipeline scripts in TheOrgBook

Refer to bcgov/TheOrgBook#390 for the numbers

Refer the Issue #42 for additional details.

The query inside same_as_existing_cred has been identified as a bottleneck. Explore other options and come to a decision as to how the logic should be refactored to remove the bottleneck.

On the "Ting" side (amalgamaTing):

• Corp will be in status HAM or HAO
• Relationship to "Ted" (amalgamaTed) company will be in corp_involved table (or search_ting_vw view)

On the "Ted" side (amalgamaTed):

• activation event will be one of the Amalgamation filings (there are many)
• relationship to "Ting's" (amalgamaTing's) will be in corp_involved (or search_ting_vw) by event_id of the amalgamation filing

• display the name in the grey box when displaying the credential
• display registration date as Not available when it is not available and other dates
• look into codes displaying (not descriptions)
• new filing type class for co-ops (added recently)

While running tests on the new WebHook notifications a few processing errors were logged to the database, but the WebHook notifications did not provide any indication of the issues.

Processing (event_by_corp_filing) error example;

• current transaction is aborted, commands ignored until end of transaction block

Posting (credential_log) error example;

• Could not locate credential type: registration.registries.ca/1.0.40 None

Received WebHook Notification:

• Ran bc_reg_event_processor

Add web hooks to mara framework to support "native" integration with rocket chat alerts

Add "admin" tasks in Mara for:

• list errors for credentials posted to TOB
• reset "failed" credentials (in order to re-post to TOB)
• clear status on all credentials (to re-post all)
• clear Event Processor database (to re-run all data)
• post a single credential to TOB
• run the pipeline in "single threaded" mode (post 1 credential at a time to TOB)

See Issue #42 for details.

Ensure we have the correct set of (active) credentials posted to OrgBook

Work with BC Registry to come up with an issuance reason code per event type that can be included in the Registration Credential. From TheOrgBook perspective, we'd like to present a timeline of the Credential and the Organization, and on that timeline, show this field (converted into English/French as appropriate) for why a Credential was issued (e.g. New Registration, Annual Renewal, Dissolved, etc.).

This is seen as a show stopper for launch of the post-SLN, so please prioritize this accordingly.

Presumably that means we need to know:

• The list of event types
• A name for each event type
• A way to detect the type of each event
• Translations for each event type

Having said that - if differences between event types are too subtle for the public, I would say it's perfectly reasonable to have one Credential Issuance Reason Code for multiple event types. Use good judgement.

The bcreg-x-agent containers are coming online and receiving requests before they are ready. Before they have completed their Indy Sync. When this occurs all agent traffic is rejected, causing errors to be reported and logged on the agent side.

This issue occurs when OpenShift is auto-scaling the pods under load. It is more easily repeatable and visible with the batch processing.

OpenShift uses health checks to determine when the container is ready to receive traffic. Currently we're using /heath endpoint which is not checking to ensure the container is actually ready to process requests.

Either update the \health endpoint or create/identify a new one that returns an Ok (200) response once the container is ready to receive traffic.

In OpenShift we have the option of using 2 different endpoints, one for liveliness and one for readiness. The liveliness probe is used to determine whether the container processes have hung, in which case the container will be killed. The readiness probe is used to determine when the container is ready to receive traffic.

tob-api containers are suffering from the same issue, so it may be best to fix at the von-x level.

Examples:
GENESIS_URL=https://raw.githubusercontent.com/sovrin-foundation/sovrin/1.1.50-master/sovrin/pool_transactions_sandbox_genesis
LEDGER_PROTOCOL_VERSION=1.3
AUTO_REGISTER_DID=0

Implementation details in PR #45

Update BC Reg X jobs to EITHER:

• support running multiple concurrent instances of the jobs

OR:

• detect if running concurrent and bail

Related to new corp types

Update to use API security

Also catch up with other new Aries agent features

Steps to reproduce:

• Starting with a running/working BC Registries Agent application, ensure it is posting credentials without error by posting some credentials.
• Restart the wallet-db container.
• Once the container has restarted completely, post some more credentials.

Observed behavior:

• The error, listed below, occurs until the bcreg-x-agent container is restarted and establishes a new connection with the wallet database.

Expected behavior:

• The error not occur.
• The application not rely on a permanent, persistent connection to the wallet database.

2018-10-10 13:48:43,407 INFO [vonx.indy.service]: Creating Indy credential offer for issuer bcreg, schema relationship.bc_registries
2018-10-10 13:48:43,408 INFO [aiohttp.access]: 172.51.124.1 [10/Oct/2018:13:48:43 +0000] "POST /bcreg/issue-credential HTTP/1.1" 200 25460 "-" "Python/3.6 aiohttp/3.1.2"
2018-10-10 13:48:43,408 ERROR [vonx.common.service]: Exception while handling request:
Traceback (most recent call last):
  File "/home/indy/.pyenv/versions/3.6.3/lib/python3.6/site-packages/vonx/common/service.py", line 255, in _handle_message
    reply = await self._service_request(request)
  File "/home/indy/.pyenv/versions/3.6.3/lib/python3.6/site-packages/vonx/indy/service.py", line 1174, in _service_request
    True)
  File "/home/indy/.pyenv/versions/3.6.3/lib/python3.6/site-packages/vonx/indy/service.py", line 688, in _issue_credential
    cred_offer = await self._create_cred_offer(issuer, cred_type)
  File "/home/indy/.pyenv/versions/3.6.3/lib/python3.6/site-packages/vonx/indy/service.py", line 747, in _create_cred_offer
    cred_type["ledger_schema"]["seqNo"]
  File "/home/indy/.pyenv/versions/3.6.3/lib/python3.6/site-packages/von_anchor/anchor/issuer.py", line 332, in create_cred_offer
    rv = await anoncreds.issuer_create_credential_offer(self.wallet.handle, cd_id)
  File "/home/indy/.pyenv/versions/3.6.3/lib/python3.6/site-packages/indy/anoncreds.py", line 245, in issuer_create_credential_offer
    issuer_create_credential_offer.cb)
indy.error.IndyError: ErrorCode.CommonInvalidState

For our production systems we want to receive notification when there are issues posting credentials.

We need to be able to enable/disable this feature easily since credential post can fail during bulk data loads causing a unnecessary flood of notifications.

Related to:
bcgov/TheOrgBook#812

The following error has caused the credential staging process in prod-tmp to stop at the affected record.

register_un_processed_corps: /data-pipeline/.venv/bin/python3 -u "bcreg/find-unprocessed-corps_actve.py"
register_un_processed_corps: Get last processed event
register_un_processed_corps: Prev event = 101931302, skipping initial corps_data_load ...
register_un_processed_corps: succeeded,  0 seconds
load_bc_reg_data_a: ★ 1:11m
load_bc_reg_data_a: /data-pipeline/.venv/bin/python3 -u "bcreg/process-corps-generate-creds.py"
load_bc_reg_data_a: Caching data for parties and events ...
load_bc_reg_data_a: Caching data for corporations ...
load_bc_reg_data_a: year 0 is out of range
load_bc_reg_data_a: year 0 is out of range
load_bc_reg_data_a: Traceback (most recent call last):
load_bc_reg_data_a:   File "bcreg/process-corps-generate-creds.py", line 11, in <module>
load_bc_reg_data_a:     event_processor.process_corp_event_queue_and_generate_creds(True)
load_bc_reg_data_a:   File "/data-pipeline/bcreg/eventprocessor.py", line 818, in process_corp_event_queue_and_generate_creds
load_bc_reg_data_a:     self.process_corp_event_queue_internal(True, True, use_cache)
load_bc_reg_data_a:   File "/data-pipeline/bcreg/eventprocessor.py", line 703, in process_corp_event_queue_internal
load_bc_reg_data_a:     bc_registries.cache_bcreg_corps(specific_corps)
load_bc_reg_data_a:   File "/data-pipeline/bcreg/bcregistries.py", line 510, in cache_bcreg_corps
load_bc_reg_data_a:     self.cache_bcreg_corp_tables(specific_corps, generate_individual_sql)
load_bc_reg_data_a:   File "/data-pipeline/bcreg/bcregistries.py", line 577, in cache_bcreg_corp_tables
load_bc_reg_data_a:     rows = self.get_bcreg_table(corp_table, corp_num_where, '', True, generate_individual_sql)
load_bc_reg_data_a:   File "/data-pipeline/bcreg/bcregistries.py", line 662, in get_bcreg_table
load_bc_reg_data_a:     return self.get_bcreg_sql(table, sql, cache, generate_individual_sql)
load_bc_reg_data_a:   File "/data-pipeline/bcreg/bcregistries.py", line 639, in get_bcreg_sql
load_bc_reg_data_a:     for row in cursor]
load_bc_reg_data_a:   File "/data-pipeline/bcreg/bcregistries.py", line 638, in <listcomp>
load_bc_reg_data_a:     rows = [dict(zip(column_names, row))
load_bc_reg_data_a: ValueError: year 0 is out of range
load_bc_reg_data_a: exit code 1
load_bc_reg_data_a: failed,  8 seconds

Update to bcgovimages/von-image:1.6-ew-11* images to pick up von_anchor v1.6.37.

A necessary 256-bit compliant encoder has been added to von_anchor, affecting credentials. Unfortunately it is not backwardly compatible with older credentials so they will need to be reloaded.

Environments to update:

• devex-von-bc-registries-agent-dev
• devex-von-bc-registries-agent-test
• devex-von-bc-registries-agent-prod
• devex-bcgov-dap-dev (prod-tmp)

Status:

Table: event_by_corp_filing Processed: 2266919 Outstanding: 0
       event_by_corp_filing Process Errors: 38
Table: corp_history_log Processed: 1344306 Outstanding: 0
       corp_history_log Process Errors: 0
Table: credential_log Processed: 2539502 Outstanding: 0
       credential_log Process Errors: 0

Error:

/data-pipeline/.venv/bin/python3 -u "bcreg/process-corps-generate-creds.py"

Caching data for parties and events ...
connection for foreign table "corp_party" cannot be established
DETAIL:  ORA-12541: TNS:no listener
None
connection for foreign table "corp_party" cannot be established
DETAIL:  ORA-12541: TNS:no listener
None
Error during caching operation, switching to smaller cache size
Processing: 0.32070502103306353
Caching data for parties and events ...
connection for foreign table "corp_party" cannot be established
DETAIL:  ORA-12541: TNS:no listener
None
connection for foreign table "corp_party" cannot be established
DETAIL:  ORA-12541: TNS:no listener
None
Error during caching operation, switching to non-cached mode
Processing: 0.48929149005562067
Processing: 0.5713557900162414
>>> Processing 1 of 38 corporations.
connection for foreign table "corporation" cannot be established
DETAIL:  ORA-12541: TNS:no listener
None
connection for foreign table "corporation" cannot be established
DETAIL:  ORA-12541: TNS:no listener
None
connection for foreign table "corporation" cannot be established
DETAIL:  ORA-12541: TNS:no listener
None
current transaction is aborted, commands ignored until end of transaction block

...

Didn't complete any activity this loop, so bail
Restoring cache mode

Traceback (most recent call last):
  File "/data-pipeline/bcreg/bcregistries.py", line 626, in get_bcreg_sql
    cursor.execute(sql)
psycopg2.OperationalError: connection for foreign table "corp_party" cannot be established
DETAIL:  ORA-12541: TNS:no listener
Traceback (most recent call last):
  File "/data-pipeline/bcreg/eventprocessor.py", line 970, in process_corp_event_queue_internal
    bc_registries.cache_bcreg_corps(specific_corps)
  File "/data-pipeline/bcreg/bcregistries.py", line 526, in cache_bcreg_corps
    self.cache_bcreg_corp_tables(specific_corps, generate_individual_sql)
  File "/data-pipeline/bcreg/bcregistries.py", line 543, in cache_bcreg_corp_tables
    party_rows = self.get_bcreg_table(self.other_tables[0], corp_party_where, '', True, generate
_individual_sql)
  File "/data-pipeline/bcreg/bcregistries.py", line 654, in get_bcreg_table
    return self.get_bcreg_sql(table, sql, cache, generate_individual_sql)
  File "/data-pipeline/bcreg/bcregistries.py", line 626, in get_bcreg_sql
    cursor.execute(sql)
psycopg2.OperationalError: connection for foreign table "corp_party" cannot be established
DETAIL:  ORA-12541: TNS:no listener
Traceback (most recent call last):
  File "/data-pipeline/bcreg/bcregistries.py", line 1314, in get_basic_corp_info
    cur.execute(sql_corp, (corp_num,))
psycopg2.OperationalError: connection for foreign table "corporation" cannot be established
DETAIL:  ORA-12541: TNS:no listener
Traceback (most recent call last):
  File "/data-pipeline/bcreg/bcregistries.py", line 1398, in get_bc_reg_corp_info
    corp = self.get_basic_corp_info(corp_num)
  File "/data-pipeline/bcreg/bcregistries.py", line 1314, in get_basic_corp_info
    cur.execute(sql_corp, (corp_num,))
psycopg2.OperationalError: connection for foreign table "corporation" cannot be established
DETAIL:  ORA-12541: TNS:no listener
Traceback (most recent call last):
  File "/data-pipeline/bcreg/eventprocessor.py", line 996, in process_corp_event_queue_internal
    corp_info = bc_registries.get_bc_reg_corp_info(corp['CORP_NUM'])
  File "/data-pipeline/bcreg/bcregistries.py", line 1398, in get_bc_reg_corp_info
    corp = self.get_basic_corp_info(corp_num)
  File "/data-pipeline/bcreg/bcregistries.py", line 1314, in get_basic_corp_info
    cur.execute(sql_corp, (corp_num,))
psycopg2.OperationalError: connection for foreign table "corporation" cannot be established
DETAIL:  ORA-12541: TNS:no listener
Traceback (most recent call last):
  File "/data-pipeline/bcreg/bcregistries.py", line 1314, in get_basic_corp_info
    cur.execute(sql_corp, (corp_num,))
psycopg2.InternalError: current transaction is aborted, commands ignored until end of transaction block
Traceback (most recent call last):
  File "/data-pipeline/bcreg/bcregistries.py", line 1398, in get_bc_reg_corp_info
    corp = self.get_basic_corp_info(corp_num)
  File "/data-pipeline/bcreg/bcregistries.py", line 1314, in get_basic_corp_info
    cur.execute(sql_corp, (corp_num,))
psycopg2.InternalError: current transaction is aborted, commands ignored until end of transaction block

...

Found this example organization: https://theorgbook.pathfinder.gov.bc.ca/en/topic/registration/BC0430308/cred/559582

It was a company registered in 1992 and appears to have been shutdown in 2004.

According to TheOrgBook page linked above, the "Name Effective Date" is 2004, which is unlikely. Chances are the name was effective starting in 1992, or perhaps later if there was a name change. But it should not be a name change at the same time the company was dissolved.

This could make setting the Name Effective date more difficult - search back through all events to find a name change, if none, use Registration Effective Date.

Please investigate - or have

Remove the TOB_INDY_DID, and TOB_INDY_SEED from the agent deployment configurations. They are not used.

Environments to Update

• devex-von-bc-registries-agent-dev

• devex-von-bc-registries-agent-test

• devex-von-bc-registries-agent-prod

• devex-bcgov-dap-dev

• ontvon-von-ont-registries-agent-dev

• ontvon-von-ont-registries-agent-test

• ontvon-von-ont-registries-agent-prod

Integrate SonarQube Scanning, starting with static scanning.

Reference; bcgov/TheOrgBook#602

Create relationships for partner organizations

Logic t.b.d.

Update manage, docker scripts etc to add new Aries agent and controller
Migrate issuer and schema configuration
Test with mara data pipeline

For the initial release of BC TheOrgBook we will not be issuing address credentials.

On occasion we get errors while processing or posting credentials. Typically these are due to transient issues (database maintenance, services restarting, etc), and all that is needed is to re-queue the affected record.

Error notification would allow us to deal with this issues when they occur, rather than polling the pipeline status occasionally.

Please add email and/or webhook error notification support to the pipeline.

The following query used in von-bc-registries-agent/data-pipeline/bcreg/eventprocessor.py is performing poorly and slowing the credential staging process to the point where is can't keep the unprocessed credential_log queue full for the credential posting process.

Look into adding indexes to increase query performance.

SELECT CREDENTIAL_JSON FROM CREDENTIAL_LOG 
                 WHERE SYSTEM_TYPE_CD = %s
                   AND CORP_NUM = %s
                   AND CORP_STATE = %s
                   AND CREDENTIAL_TYPE_CD = %s
                   AND CREDENTIAL_ID = %s
                 ORDER BY RECORD_ID DESC

This is what the current registration look like;

The URL is the route back to the BC Registries Agent, and there are no web pages implemented there, and the proxy providing access into the environment is slated for removal.

A set of tickets/issues related to Key Handling and Backups in preparation for the launch.

Remove the bcreg-x-proxy from the BC Registries Agent environments. The component is not needed, none of the BC Registries Agent services or endpoints should be exposed to the outside.

• Adds support for credential dependency look-up.
• Includes updated versions of von_anchor and von-x

Environments to update:

• devex-von-bc-registries-agent-dev
• devex-von-bc-registries-agent-test
• devex-von-bc-registries-agent-prod

This is not related to the upgrade. We have recently copied entities from the mainframe to CPRD. These entities are imaginatively called “Others” or “Coops etc”. Many of them have unknown dates hence the 2099-12-31. Yes you should put in a rule for them. Here is a summary of the use of the 2099-12-31 date:

Cemetary

28

Cooperative

1051

Extra Pro Coop

0

Financial Institutions

1

Libraries

101

Parishes

49

Pension Funded Society

2

Private Act

364

Railways

5

Society Branch

2

Tramways

11

There are some corps missing in production that existed in prod previously.

Add additional business types to BC Reg data pipeline

Add relationships for partnerships and amalgamations

The latest bcgovimages/von-image set (-9 series) use a single wallet for all roles if more than one is being used. Update deployment configurations to align with this new approach. One one, key, seed, and did are needed for the wallet credentials.

Refs:

• bcgov/von#106
• bcgov/TheOrgBook#638

Update Environments:

• devex-von-bc-registries-agent-dev
• devex-von-bc-registries-agent-test
• devex-von-bc-registries-agent-prod

As a part of tidy-up ...
Remove "Category" as a model type in settings.yml
Remove Category table in ToB

Ian has completed the first cut of the BC Registries Event process - including a BCReg-X based on VON-X and a Data Pipeline based on Mara. There is also a Postgres database for tracking credential processing/publishing.

Please use Slack including Stephen and Ian for questions. Ian may or may not reply since he is on vacation.

We continue to see credential posting failures occasionally during sustained high volume processing such as initial data loads.

The issue occurs due to connection/response timeouts between the agent and the tob-api and is directly related to the tob-api's ability to effectively handle the volume of requests from the agent at any given time.

When the tob-api is unable to effectively handle the load it cannot process the requests fast enough, causing some of the agent requests to time out, which results in the following error message being logged for the process failure:

Bad response from post_json: (504) <html><body><h1>504 Gateway Time-out</h1>
The server didn't respond in time.
</body></html>

Under normal initial load conditions we pre-configure the following resources (for BC Registries);

• 2 mara containers;
  • One dedicated to processing the corporate records.
  • One dedicated to posting credentials.
• 8 agent containers to service the credential post requests.
• 5 tob-api pods to process the posted credentials.

The mara and agent containers are scaled manually, the tob-api containers are configured to auto-scale, however for initial data loads we ensure the tob-api containers are pre-scaled to 5 instances. Under these conditions there are enough tob-api containers to easily handle the requests from the 8 agent containers.

The failures start to occur when there are fewer tob-api containers servicing the high volume of requests. The problem is most notable when there is only a single tob-api container servicing all of the requests. The one container simply cannot keep up, and many agent requests timeout before the tob-api can respond and/or the number of containers is scaled up enough to be able to handle the sustained volume.

This condition can occur for a number of different reasons, including:

• The pod containing the tob-api pod gets moved to different compute node; due to evacuation, scheduling, or load balancing.
• The number of tob-api containers is scaled down; due to decreased resource utilization which can be caused by lulls in the volume of credentials being processed.

Review the REST API for the bcreg-x-agent and determine what services are OK to be open and what needs to be locked down - e.g. the issuance of credentials (/bcreg/issue-credential).

Once defined - document the items to be locked down and add a ticket to make it so.