@stuartc , i'm interested in your thoughts on this (https://github.com/revelrylabs/elixir-nodejs) for use in dispatcher: NodeJS.call(module, args \\ [], opts \\ [])

Options
:name - (optional) The name used for supervisor registration. Defaults to Elixir.NodeJS.Supervisor.
:path - (required) The path to your Node.js code's root directory.
:pool_size - (optional) The number of workers. Defaults to 4.

I wonder if we could save on the CPU and memory usage by having each microservice manage its own pool of NodeJS workers. If core will be available both with and without the CLI, does this make more sense going forward?

Getting a "sh: cd: line 1: can't cd to /opt/app/project/node_modules: No such file or directory" when attempting to run a project that triggers 3 jobs simultaneously from an inbound message:

web_1  | [info] POST /inbox
web_1  | [debug] Processing with MicroserviceWeb.Receiver.receive/2
web_1  |   Parameters: %{"__query_params" => %{}, "a" => 1, "baz" => "qux", "foo" => "other thing", "form" => "COVID_case_registration", "formId" => "Tier_2_Needs_Assessment_v4", "from" => "OpenFn", "ju_ju" => "bean", "nüumber" => 2, "surveyId" => 37479}
web_1  |   Pipelines: [:api]
web_1  | [debug] Receiver.receive/2 called
web_1  | [debug] env: NODE_PATH=/opt/app/project/node_modules
web_1  | cmd:   (cd $NODE_PATH && ./.bin/core execute   -e /opt/app/tmp/expression-1619185429-1-cz7ixx.js   -l @openfn/[email protected]   -s /opt/app/tmp/state-1619185429-1-1kp57ip.json   -o /opt/app/tmp/final_state-1619185429-1-ezb5td.json 
web_1  |   
web_1  |   
web_1  |   )
web_1  | 
web_1  | 
web_1  | [debug] env: NODE_PATH=/opt/app/project/node_modules
web_1  | cmd:   (cd $NODE_PATH && ./.bin/core execute   -e /opt/app/tmp/expression-1619185429-1-13qlsku.js   -l @openfn/[email protected]   -s /opt/app/tmp/state-1619185429-1-1igdd2z.json   -o /opt/app/tmp/final_state-1619185429-1-nnmxoe.json 
web_1  |   
web_1  |   
web_1  |   )
web_1  | 
web_1  | 
web_1  | [debug] env: NODE_PATH=/opt/app/project/node_modules
web_1  | cmd:   (cd $NODE_PATH && ./.bin/core execute   -e /opt/app/tmp/expression-1619185429-1-1tww7ug.js   -l @openfn/[email protected]   -s /opt/app/tmp/state-1619185429-1-14cmllp.json   -o /opt/app/tmp/final_state-1619185429-1-1kzuq7v.json 
web_1  |   
web_1  |   
web_1  |   )
web_1  | 
web_1  | 
web_1  | [info] Sent 202 in 4ms
web_1  | [debug] "sh: cd: line 1: can't cd to /opt/app/project/node_modules: No such file or directory"
web_1  | [debug] "sh: cd: line 1: can't cd to /opt/app/project/node_modules: No such file or directory"
web_1  | [debug] "sh: cd: line 1: can't cd to /opt/app/project/node_modules: No such file or directory"

Using the following project.yaml:

credentials:
  dhis2-play-covid-action:
    hostUrl: '******'
    password: '******'
    username: '******'
  google:
    accessToken: '******'
    expiresIn: '******'
    familyName: '******'
    givenName: '******'
    id: '******'
    idToken: '******'
    locale: '******'
    name: '******'
    picture: '******'
    refreshToken: '******'
    scope: '******'
    tokenType: '******'
  salesforce:
    loginUrl: '******'
    password: '******'
    securityToken: '******'
    username: '******'
jobs:
  covid-19-tei-(dhis2):
    adaptor: '@openfn/[email protected]'
    credential: 'dhis2-play-covid-action'
    expression: >
      // See list of cases here: https://covid19.dhis2.org/demo/dhis-web-tracker-capture/index.html#/?program=DM9n1bUw8W8
      createTEI({
        orgUnit: 'GD7TowwI46c',
        trackedEntityType: 'MCPQUTHX1Ze',
        attributes: [
          {
            attribute: 'he05i8FUwu3', // case id
            value: dataValue('body._id'),
          },
          {
            attribute: 'sB1IHYu2xQT', // first name
            value: dataValue('body.Patient_name'),
          },
          {
            attribute: 'ENRjVGxVL6l', // last name
            value: dataValue('body.Last_Name_of_Patient'),
          },
          {
            attribute: 'Rv8WM2mTuS5', // age
            value: dataValue('body.Age'),
          },
        ],
        enrollments: [
          {
            orgUnit: 'GD7TowwI46c',
            program: 'DM9n1bUw8W8',
            programState: 'sAV9jAajr8x',
            enrollmentDate: dataValue('body.Date'),
            incidentDate: dataValue('body.Covid_19_suspected_criteria/Speciman_Collection_date')
          },
        ],
      });
    trigger: 'covid-case-submitted'
  covid-case-registration-(gs):
    adaptor: '@openfn/[email protected]'
    credential: 'google'
    expression: >
      appendValues({
        spreadsheetId: '1EFkY4zD4qqxnJdH-QaeasKd1zXC-1sNKpEg08W-3sT0',
        range: 'COVID-19 Cases!A2',
        values: state => {
          const kobo = state.data.body;
          
          console.log('Submission data: ' + JSON.stringify(kobo, null, 2));
          
          return [
            [
              kobo['National_ID'],
              kobo['Patient_name'],
              kobo['Last_Name_of_Patient'],
              kobo['Sex'],
              kobo['Age'],
              kobo['Comments'],
              kobo['Date']
            ],
          ];
        },
      });
    trigger: 'covid-case-submitted'
  create-sf-contact:
    adaptor: '@openfn/[email protected]'
    credential: 'salesforce'
    expression: >
      create('Contact', {
        FirstName: dataValue("body.Patient_name"),
        LastName: dataValue('body.Last_Name_of_Patient'),
        Age__c: dataValue("body.Age"),
        Sex__c: dataValue("body.Sex"),	
        Case_ID__c: dataValue("body.National_ID"),	
        Comments__c: dataValue("body.Comments")
      });
    trigger: 'covid-case-submitted'
triggers:
  covid-case-submitted:
    criteria: '{"form":"COVID_case_registration"}'

To debug:

(node:56) UnhandledPromiseRejectionWarning: Error: EACCES: permission denied, mkdir '/opt/app/assets/node_modules/.cache/hard-source'
    at Object.mkdirSync (fs.js:921:3)
    at sync (/opt/app/assets/node_modules/mkdirp/index.js:72:13)
    at sync (/opt/app/assets/node_modules/mkdirp/index.js:78:24)
    at Function.sync (/opt/app/assets/node_modules/mkdirp/index.js:78:24)
    at runReadOrReset (/opt/app/assets/node_modules/hard-source-webpack-plugin/index.js:263:16)
    at _next0 (eval at create (/opt/app/assets/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:58:17)
    at eval (eval at create (/opt/app/assets/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:76:1)
(node:56) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:56) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

I get an error about not being able to find DATABASE_URL even though Ecto doesn't appear at first glance to be present.

Sections to add to the docs to better qualify best-fit implementations...

1. Prerequisites (specs & resource requirements to share with interested teams)
2. Example use cases supported in the current state (vs. roadmap use cases... ideally every technical feature is linked to a real-world requirement)
3. Ongoing management (resource requirements - technical & human)

It's really easy to incorrectly copy paste an expression into a project.yaml file.
The jobs fail silently (if there is no stdout logging), and you have to exec into the container and run the job by hand to figure this out.

This is likely an engine issue as well, but we need to make sure that env variables are properly passed to engine at runtime, not just during (for example) docker build. @stuartc , do you know why the commented out lines don't work in config/config.exs in Microservice during docker build?

config :microservice, Microservice.Engine,
  # TODO: allow dynamic configuration of project path. (this is currently set during docker build.)
  # project_config: "file://" <> System.get_env("PROJECT_PATH") <> "/project.yaml"
  project_config: "file://project/project.yaml"

Specifically, we get a "PROJECT_PATH" is nil type of error, because docker build is being run without a PROJECT_PATH env. I'm wondering why this is even being evaluated during docker build. The offending docker build command is mix do compile.

Ideally, we'd let users specify the path they want to keep their project.yaml in... some might keep it in the root, some might keep it in /my-project along with a bunch of job.js files or even credential.json files.

I know some deps allow for runtime configuration like this:

Thoughts?

Currently, the docs for Docker usage are based on pre-engine microservices. Update these once engine usage is confirmed.

MVP, supported by the DIAL OSC

1. An open source server application (this application, microservice.)
2. The open source inbox application, which will be used by both platform
   and microservice to handle HTTP requests
3. The open source dispatcher application which will be used by both
   platform and microservice to execute jobs using OpenFn/core.
4. A Dockerfile that, given a job, an adaptor, and a credential, is able
   to automatically generate a container that runs the microservice for that
   job.

Roadmap for this application

• mix phx.server receives receipts and sends 200
• Chain jobs together
• Timer jobs can keep state
• tmp files are deleted after job is run
• bring core out of package.json
• endpoint gets URL and PORT from .env
• ShellWorker picks up config from .env
• ShellWorker executes it, given preloaded job, cred, adaptor, and core.
• ShellWorker can pipe to stdout.
• Write tests for everything.

Could we provide a config option that held the HTTP request open (no 202/accepted) until the job is done running?

Bug

See related Community post with a full description of the issue: https://community.openfn.org/t/in-microservice-the-job-using-flow-trigger-always-use-the-configuration-from-the-previous-job-even-though-it-has-its-own-credential/234

Job A, assigned with Credential A, the trigger is cron every 5 minutes
Job B, assigned with Credential B, the trigger is “Flow” trigger of Job A

Job B is always using the Credential A as its state.configuration, instead of Credential B.

@stuartc It works on platform, so it sounds like there is a Flow Jobs bug here. Can you please write a test to replicate and troubleshoot the issue?

Under WP1 of the DS grant, we need to make some modifications to our Docker/Kubernetes setup to acheive compliance with Instant OpenHIE. Pending updates from a discussion with Jembi, we expect this to consist of three things:

1. an instant.json file
2. docker compose scripts
3. kubernetes yaml files

See full details here: https://openhie.github.io/instant/docs/how-to/creating-packages/

And the github for the Instant project: https://github.com/openhie/instant

From Jembi: Example Instant OpenHIE packages that we're currently working on, to give an idea of the structure/content of a package:
https://github.com/jembi/who-covid19-surveillance-package
https://github.com/jembi/covid19-immunization-tracking-package

• Instant OpenHIE Overview slide deck
• Instant OpenHIE Zulip Chat link: https://chat.fhir.org/#narrow/stream/274603-OpenHIE/topic/Instant.20OpenHIE/ (edited)

Configuration via containers in a non-local (k8s, remote docker) environment is very difficult as there is no local volume mount (at least not without significant effort).

How about the ability for the app to boot without config, and have uploaded via http - storing it locally (or in a pvc/volume) for subsequent startups.