• nodejs
  Add initial support for nodejs 8 starting with LTS 8.9.1

• npm packages
  Start with single npm package openwhisk and then as we move forward review which packages to add to the image.

Environment details:

IBM Cloud Functions

Steps to reproduce the issue:

Create a function using the Node.js 8 runtime with this code snippet.

const main = () => {
    return a.b.c
}

Invoking this action, the following activation response is returned.

{
  "error": "An error has occurred: ReferenceError: a is not defined"
}

Stack trace is included in the logs.

"2018-06-19T16:29:11.783438495Z stderr: ReferenceError: a is not defined"
"2018-06-19T16:29:11.783496031Z stderr: at NodeActionRunner.main [as userScriptMain] (eval at NodeActionRunner.init (/nodejsAction/runner.js:76:45), <anonymous>:13:5)"
"2018-06-19T16:29:11.783515019Z stderr: at /nodejsAction/runner.js:95:45"
...

Change the action code to the following source.

const main = async () => {
    return a.b.c
}

Invoking the updated action, the following activation response is returned.

{
  "error": {}
}

Nothing is logged to stdout or stderr for this activation.

This also affect all error handling for async functions, where errors are thrown to signal errors from user-land code.

cause

Looking at the code, when an error is thrown from async functions, it ends up in catch callback handler @ https://github.com/apache/incubator-openwhisk-runtime-nodejs/blob/master/core/nodejsActionBase/runner.js#L107-L117

It returns the Error object thrown from the async function as the child property of the error key in the normal response body.

Error objects serialise to an empty object when converting to JSON, which explains the empty object returned in the activation response.

> JSON.stringify(new Error('hello'))
'{}'

fix

Async error should mirror response & log values from normal errors, return err.message in result and log stack track.

next steps

If people agree with this diagnosis and the suggested resolution, I can try to submit a PR to resolve this?

Following Apache license header guideline, all human-readable Apache-developed files that are included within a distribution must include the header text with few exceptions. You can find few exceptions here: which files do not require a license header.

I used Apache Rat to check this repository after excluding a few files, and I got this report. We need to add Apache licensing header to those files.

Unapproved licenses:

  openwhisk-runtime-nodejs/core/nodejsActionBase/Dockerfile
  openwhisk-runtime-nodejs/core/nodejs6Action/CHANGELOG.md
  openwhisk-runtime-nodejs/core/nodejs6Action/Dockerfile
  openwhisk-runtime-nodejs/core/nodejs8Action/CHANGELOG.md
  openwhisk-runtime-nodejs/core/nodejs8Action/Dockerfile
  openwhisk-runtime-nodejs/tests/src/test/resources/application.conf
  openwhisk-runtime-nodejs/ansible/environments/local/group_vars/all
  openwhisk-runtime-nodejs/ansible/environments/local/hosts

The excluded files are:

**/*.json
**/**.gradle
**/gradlew
**/gradle/**
**/.**
**/templates/**
**/*.j2.*
**/.github/**
**/auth.whisk.system
**/auth.guest
**/i18n_resources.go

@cfjedimaster commented on Fri Jan 06 2017

OpenWhisk is using a 1.* version of the watson-developer-cloud package, which is now at 2.*. Please see if you can upgrade to that version.

there are three official actions which use the request package in https://github.com/apache/incubator-openwhisk-catalog

• packages/github/webhook.js
• packages/slack/post.js
• packages/weather/forecast.js

but cannot found such package while invoke these actions:

    "response": {
        "result": {
            "error": "Initialization has failed due to: Error: Cannot find module 'request'\n    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:582:15)\n    at Function.Module._load (internal/modules/cjs/loader.js:508:25)\n    at Module.require (internal/modules/cjs/loader.js:637:17)\n    at require (internal/modules/cjs/helpers.js:22:18)\n    at eval (eval at evalScript (/nodejsAction/runner.js:180:16), <anonymous>:4:15)\n    at eval (eval at evalScript (/nodejsAction/runner.js:180:16), <anonymous>:175:14)\n    at evalScript (/nodejsAction/runner.js:180:16)\n    at NodeActionRunner.init (/nodejsAction/runner.js:67:35)\n    at doInit (/nodejsAction/src/service.js:166:14)\n    at initCode (/nodejsAction/src/service.js:93:24)"
        },
        "status": "action developer error",
        "success": false
    }

New releases with security patches came out yesterday https://nodejs.org/en/blog/vulnerability/november-2018-security-releases/

Environment details:

• local deployment
• version of docker: 20.10.17, gradle: 6.9.1.

Steps to reproduce the issue:

1. Run the distDocker command to generate the local Docker image for the desired runtime version.

Provide the expected results and outputs:

Create the runtime image 

Provide the actual results and outputs:

Build successful, though with warning "Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0."

This is now released: https://nodejs.org/en/blog/release/v12.0.0/

have the ability to take a source in typescript format.
This is useful for development time, but as always is recommended to compile into a bundle js, no transpile require on /init for production workloads.

This nodejs runtime does not permit a re-init while running a function but responds with an error that could be confusing.

{"error":"Internal system error: system not ready, status: running"}

As part of normalizing all the runtimes to handle re-init consistently, change the error message matching the Java runtime (this will be adopted in the upstream runtime tests which will be extended to include a test for re-init, where the first init is expected to succeed but the second should fail. The action proxy will reject the request with status code other than 200 OK).

[1] https://github.com/apache/incubator-openwhisk-runtime-java/blob/3657b3889fd1ce03581184fcb07766a8c897a4d8/java8/proxy/src/main/java/openwhisk/java/action/Proxy.java#L72-L75

Problem when your action wants to reference another file in your zip, locally this is easy because the node process usually get's started from the root of the project where the package.json is located.

For example the following code works fine in my local computer.
Since I'm running the node process locally

const fs = require('fs')
exports.main = () => {
    const cwd = process.cwd()
    const dirname = __dirname
    //in openwhisk need to find the root of the zip
    console.log(`dirname= ${dirname}`)
    console.log(`cwd= ${cwd}`)
    const file = fs.readFileSync('secondfile.txt',{encoding:'utf8'})
    console.log(file)
    return {file, cwd, dirname}
}

I run this code in openwhisk I get an error about not found

In openwhisk currently the current working directory (cwd) is set to where the nodejs process started /nodejsAction

This is different from using for example dockerskeleton or python action, where the cwd is changed to the tmp directory where the the zip action get's unzip.

For now the workaround is to use __dirname like ${dirname}/secondfile.txt instead of secondfile.txt

const fs = require('fs')
exports.main = () => {
    const cwd = process.cwd()
    const dirname = __dirname
    //in openwhisk need to find the root of the zip
    console.log(`dirname= ${dirname}`)
    console.log(`cwd= ${cwd}`)
    const file = fs.readFileSync(`${dirname}/secondfile.txt`,{encoding:'utf8'})
    console.log(file)
    return {file, cwd, dirname}
}

The fix is in runner.js to change the process directory process.chdir() to the directory where the zip get's extracted

https://github.com/apache/incubator-openwhisk-runtime-nodejs/blob/master/core/nodejsActionBase/runner.js#L59

process.chdir(moduleDir)
 thisRunner.userScriptMain = eval('require("' + moduleDir + '").' + message.main);

cc @rabbah

It would be good to start building the image for nodejs:10 even if its to get the image posted to be use by folks and get feedback.
Would not recommend to add to core repo default deployment until LTS in Oct 2018.

node-cache is a very popular simple/fast internal caching package. Right now I don't think there are any reasonable caching options for actions. I think this would be a really useful addition, especially for actions that can stay warm...

the latest version deals with wsk list sugaring for the pending removal of the api from the backend.

@jberstler commented on Wed Jan 04 2017

In trying to use the NPM package node-rdkafka to post messages to a Message Hub instance from a NodeJS action, user @glikson was unable to successfully authenticate, likely due to the sasl system libraries not being available in the NodeJS runtime.

The next most reasonable course of action was to use message-hub-rest which relies on a soon-to-be-deprecated REST interface.

Please consider adding the appropriate sasl libraries to the NodeJS runtime in order to support this scenario.

Update the Travis build to push a tag "master" to Docker when a the master branch is updated.
This will make it easier to pull the image that aligns with the latest changes on the Master branch.

Reference can be taken from: apache/openwhisk-runtime-java#37

Update node https://nodejs.org/en/blog/release/v6.12.0/
Update openwhisk 3.10.0

Making the web proxy leaner avoids the delay to load module tree for express and body-parser.
Have the web proxy use http server from nodejs

New nodejs8 released: https://nodejs.org/en/blog/release/v8.12.0/

On the first run of non-stem cell (environmentally initialized containers), return the initialization response rather than the return of the run.

Replication steps:
service.yaml:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: greetings-builtin-code
  namespace: default
spec:
  template:
    spec:
      containers:
        - image: openwhisk/action-nodejs-v12
          env:
            - name: __OW_RUNTIME_PLATFORM
              value: knative
            - name: __OW_ACTION_NAME
              value: greetings-builtin-code
            - name: __OW_ACTION_CODE
              value: "function main(params) {return {payload: 'Hello ' + params.name + ' from ' + params.place +  '!'};}"

After deployment,

kubectl apply -f service.yaml

service.serving.knative.dev/greetings-builtin-code configured

kubectl get ksvc

NAME                     URL                                                 LATESTCREATED                  LATESTREADY                    READY   REASON
greetings-builtin-code   http://greetings-builtin-code.default.example.com   greetings-builtin-code-fv6cv   greetings-builtin-code-fv6cv   True    

value-data.json

{
    "value": {
        "name" : "Will",
        "place" : "Austin"
    }
}

Now when curl'ing the endpoint, we expect to see the function run

 curl -d @value-data.json http://greetings-builtin-code.default.example.com

instead we see

{"OK":true}

I can potentially look into the fix, but I definitely wanted to write down the replication steps before I forgot.

Environment details:

• local deployment, vagrant, native ubuntu, Mac OS, Bluemix, ...
• version of docker, vagrant, ubuntu, ...

Steps to reproduce the issue:

Provide the expected results and outputs:

output comes here

Provide the actual results and outputs:

output comes here

Additional information you deem important:

• issue happens only occasionally or under certain circumstances
• changes you did or observed in the environment

bump nodejs versions 10.15, 8.15, 6.16

As pointed out by @alexkli in this comment, it would be great to support tar.gz as an alternative package format. The main reason is that npm has a built-in pack command that creates a tar.gz file, embedding all required dependencies (but no devDependencies) and that this would eliminate the need for scripting workarounds.

The recent PR #95
cause problems in the travis pipeline in core repo: https://travis-ci.org/apache/incubator-openwhisk/builds/447588143

The user should be allowed to on Error return an error object or simple value, that's the contract today
The PR changes this to always make it a string { error: <String>}

Another problem I see this PR always do a console.error() if it contains error.message, this is not ideal as it can leak privacy data into logs, and in some production system eat over their quota.
The user should be in control on what to log or not. we should not force the console.error

I understand that the PR was trying to address #63 I think the real problem is that error native objects were not being serialize.

I propose:

1. Do not change the current contract, do not constrain the output of the Function to a string.
2. Handle properly Event type errors object that it serializes and is not empty
3. Do not console.error, this is user land responsibility domain on what they want to log or not.

This is a feature request...

Looking at the recent knative integration (#119) - this works great for actions listening to HTTP requests as it proxies between the HTTP request and the expected Web Action parameters/response.

This work could be extended to support listening to Cloud Events on Knative. This would allow those actions to be connected to knative event sources.

Digging around in how Cloud Event receivers work - they use an SDK which just listens on a HTTP port for the incoming events and decodes to an object, e.g.
https://github.com/knative/eventing-sources/blob/master/cmd/event_display/main.go

We can just import the CloudEvents JS SDK to handle the HTTP transport and proxy the events to the handler functions. https://github.com/cloudevents/sdk-javascript

If the npm package that is defined by https://github.com/apache/incubator-openwhisk-runtime-nodejs/blob/master/core/nodejs8Action/package.json was published to npm then projects would be able to depend on it and get the current set of js dependencies present in OW. Those projects building with webpack would also be able to exclude those packages from their builds, making their images smaller.

https://nodejs.org/en/blog/vulnerability/march-2018-security-releases/

The 'openwhisk' Package has been updated to 3.16.0. This should be updated in our NodeJS runtime's packages.

Using local deployment of OW through docker compose on mac. All has been wonderful till a couple days ago when I did a fresh install and my actions started returning:

"error": "Initialization has failed due to: Action entrypoint 'main' is not a function."

My actions are built using typescript and webpack based on the pattern found in IBM Cloud Function Typescript Starter Kit.

It looks like this change to the global scope eval is responsible.

Here is a small bundled action that should reproduce the issue:

!function(e){var t={};function n(r){if(t[r])return t[r].exports;var o=t[r]={i:r,l:!1,exports:{}};return e[r].call(o.exports,o,o.exports,n),o.l=!0,o.exports}n.m=e,n.c=t,n.d=function(e,t,r){n.o(e,t)||Object.defineProperty(e,t,{enumerable:!0,get:r})},n.r=function(e){"undefined"!=typeof Symbol&&Symbol.toStringTag&&Object.defineProperty(e,Symbol.toStringTag,{value:"Module"}),Object.defineProperty(e,"__esModule",{value:!0})},n.t=function(e,t){if(1&t&&(e=n(e)),8&t)return e;if(4&t&&"object"==typeof e&&e&&e.__esModule)return e;var r=Object.create(null);if(n.r(r),Object.defineProperty(r,"default",{enumerable:!0,value:e}),2&t&&"string"!=typeof e)for(var o in e)n.d(r,o,function(t){return e[t]}.bind(null,o));return r},n.n=function(e){var t=e&&e.__esModule?function(){return e.default}:function(){return e};return n.d(t,"a",t),t},n.o=function(e,t){return Object.prototype.hasOwnProperty.call(e,t)},n.p="",n(n.s=407)}({407:function(e,t,n){"use strict";var r=this&&this.__awaiter||function(e,t,n,r){return new(n||(n=Promise))(function(o,u){function i(e){try{f(r.next(e))}catch(e){u(e)}}function c(e){try{f(r.throw(e))}catch(e){u(e)}}function f(e){e.done?o(e.value):new n(function(t){t(e.value)}).then(i,c)}f((r=r.apply(e,t||[])).next())})};function o(){return r(this,void 0,void 0,function*(){return{statusCode:200,body:{status:"ok"}}})}Object.defineProperty(t,"__esModule",{value:!0}),t.default=o,global.main=o}});

Why nodejs has following limit when parse action runner,

app.use(bodyParser.json({ limit: "48mb" }));

while python and java8 runtime does not have such limit?

Given that I have a package.json that I want to use to extend the nodejs runtime
Let's say I want to use a different version of npm package openwhisk and some of my own.

package.json:

{
  "dependencies": {
    "cowsay": "^1.3.0",
    "openwhisk": "2.0.0"
  }
}

I have the following Dockerfile

FROM openwhisk/action-nodejs-v8
COPY package.json .
RUN npm i --production

Running this will not work. This is how I want it to work.
This will wipe the already present /nodejsAction/node_modules directory and install my new npm packages openwhisk and cowsay

A 1/2 workaround is to install my dependcies in the parent directory of /nodejsAction/ which /
This way nodejs ref 1 will not find it in /nodejsAction/tmpXX or /nodejsAction/ and fall back to /

FROM openwhisk/action-nodejs-v8
COPY package.json /
RUN cd / && npm i --production

This will only to find cowsay in the /node_modules but will find the 1.x openwhisk in /nodejsAction/node_modules and never use the version 2.0.0 where I installed it /node_modules/openwhisk

My proposed solution is that we installed embedded npm packages in in the root / this way they act as fallback global modules in the search path, and user is always able to declare the npm packages he wants to use in the action.

cc @rabbah

Environment details:

• local deployment, docker compose in dev-tools

Steps to reproduce the issue:

1. compose workflow
2. deploy with default option (kind = nodejs:10)
3. invoke

Provide the expected results and outputs:

start workflow

Provide the actual results and outputs:

{ "error": "Initialization has failed due to: Error: Cannot find module 'redis'\n at Function.Module._resolveFilename (internal/modules/cjs/loader.js:582:15)\n at Function.Module._load (internal/modules/cjs/loader.js:508:25)\n at Module.require (internal/modules/cjs/loader.js:637:17)\n at require (internal/modules/cjs/helpers.js:22:18)\n at eval (eval at initializeActionHandler (/nodejsAction/runner.js:57:23), :361:55)\n at eval (eval at initializeActionHandler (/nodejsAction/runner.js:57:23), :446:3)\n at eval (eval at initializeActionHandler (/nodejsAction/runner.js:57:23), :447:14)\n at initializeActionHandler (/nodejsAction/runner.js:57:23)\n at doInit (/nodejsAction/src/service.js:163:16)\n at initCode (/nodejsAction/src/service.js:93:24)"
}

Additional information you deem important:

the workflow starts when deploying it with --kind=nodejs:6

Environment details:

• all deployments
• all versions

Steps to reproduce the issue:

1. Create a JS action following Package an action as a single bundle instructions.
2. Include a package like request-promise that's already included in the OpenWhisk container.
3. Build the action

Provide the expected results and outputs:

It would be great if right out of the box your JS action would exclude any packages in the container. For instance when I exclude request-promise the action is 2kb.

webpack v4.13.0

1e71fd7ad002a3195f86
  size     name  module                      status
  167 B    0     ./src/config.js             built
  42 B     1     external "request-promise"  built
  1.42 kB  2     ./src/stockAPI.js           built
  992 B    3     ./src/index.js              built

  size     name  asset                       status
  2.05 kB  main  bundle.js                   emitted

  Δt 106ms

Provide the actual results and outputs:

But when I don't specify request-promise as an external the action is 770kb:

webpack v4.13.0

2dfdb6782288d2371fc3
  size     name  module                  status
  42 B     1     external "util"         built
  42 B     3     external "crypto"       built
  42 B     9     external "url"          built
  42 B     16    external "stream"       built
  42 B     20    external "http"         built
  42 B     31    external "querystring"  built
  42 B     35    external "https"        built
  42 B     47    external "net"          built
  42 B     63    external "fs"           built
  42 B     65    external "tls"          built
  167 B    87    ./src/config.js         built
  42 B     88    external "os"           built
  42 B     174   external "zlib"         built
  1.42 kB  224   ./src/stockAPI.js       built
  992 B    225   ./src/index.js          built

  size     name  asset                   status
  733 kB   main  bundle.js               emitted

  Δt 6940ms (211 modules hidden)

Additional information you deem important:

Yes, I know I can go into webpack and list the pre-installed node packages as externals like so:

module.exports = {
  entry: './src/index.js',
  output: {
    path: path.resolve(__dirname, '..', 'dist'),
    filename: 'bundle.js'
  },
  target: 'node',
  externals: [ 'request-promise' ]
};

but wouldn't life be so much easier if all the packages that are installed in the node container were in a single npm package? This way you could easily exclude all the dependent packages in webpack and it would make maintenance of the packages in the container easier.

For instance I created a package simon-runtime-deps which includes all 55 dependencies from the node 6 container. This allows me to create a quick script to externalize those dependencies in webpack:

var path = require('path');
var nodeExternals = require('simon-runtime-deps');

module.exports = {
  entry: './src/index.js',
  output: {
    path: path.resolve(__dirname, '..', 'dist'),
    filename: 'bundle.js'
  },
  target: 'node',
  externals: [nodeExternals()]
};

This approach could also have the added benefit of simplifying the package.json or Dockerfile so you wouldn't have some many dependencies to maintain.

We could have an openwhisk-node8-dependencies package as a base and vendors can add their own packages bluemix-node8-dependencies for example.

This is to discuss replacing use of environment variables in runtimes with some other object ( e.g. "context") that is explicitly provided to actions, to avoid issues when runtimes support concurrent activation processing, and simplify the per-activation metadata provided to action developers.

This requires no change in OW core, but does require changes in all runtimes (assuming consistency is important) so rather than create many issues across all repos, I'm creating this one issue to start with...

Some discussion here:
https://lists.apache.org/thread.html/4b75db671e5f52c11bad6e3073c4cb4170d2ef0fdfb0cd73bed259b0@%3Cdev.openwhisk.apache.org%3E

In general, during concurrent activation processing, referring to (e.g. in nodejs case) process.env will be inaccurate, if that is set during each activation as is done now:
https://github.com/csantanapr/incubator-openwhisk-runtime-nodejs/blob/master/core/nodejsActionBase/src/service.js#L163

However, disabling process.env usage will be a breaking change.

A general plan for deprecating env var usage and allowing migration to new main signatures should be provided.

For nodejs, we can:

• enhance process.env with a dynamic (getter) property a la Object.defineProperty(process.env, key) where value is dynamically calculated based on the current activation
• also capture keys into a new context arg
• change the main function invocation to send the context arg (as well as params)
• generate a warning log for usages of process.env[key] (to signal to devs that they will need to migrate to the new signature)

For java (and others), I think similar will work, although I'm not sure how simple it would be to intercept the env getter to inject a warning log. Each runtime will need to have some way to disambiguate the env var access in order to support concurrency without changing to use the new main function signature (main(params, context)), but additionally, each runtime should also change to support structured logging (activation id included with each log event), if it wants to support concurrency, so it should be considered to do BOTH of these at the same time, in each runtime.

Currently the action running for zip files requires either an index.js or package.json with a main entry module declared. I propose expanding the support for specifying "main" (e.g., wsk action create --main module.function) so that one can override the module to be loaded too.

Nodejs8 LTS got released last night
https://nodejs.org/en/blog/release/v8.11.2/

@rabbah commented on Tue Oct 31 2017

if you declare a node.js action with a function foo and use exports.main = foo, the action will fail to run Initialization has failed due to: ReferenceError: main is not defined. Currently we're only supporting exports.main for zip node.js action. Should we allow it for non-zip actions as well?

The alternative is using --main for the CLI or correspondingly from the API.
This could be an issue for the nodejs runtimes library - @csantanapr wdyt?

@jthomas commented on Thu Nov 09 2017

I would definitely like this. Would be a nice feature for consistency.

When using package action the CWD should be the root of the extracted zip

Currently users need to use __dirname to reference a file inside they zip, this creates problems when developing local as usually the node process is started on the root of the nodejs module.

The proposed solution is to apply same technique from the python package actions, for the proxy to change the current process CWD to be the root of the directory where the package action is extracted.

The change should be done in this area to add process.chdir():
https://github.com/apache/incubator-openwhisk-runtime-nodejs/blob/master/core/nodejsActionBase/runner.js#L55

return unzipInTmpDir(message.code).then(function (moduleDir) {
   process.chdir(moduleDir)

A test case should be added, it could be a simple one that a zip action includes a txt file that the action will read from the current directory and return back the content.

cc @eweiter

Update nodejs 6 and 8 to pickup latest security patches
https://nodejs.org/en/blog/release/v6.14.4/
https://nodejs.org/en/blog/release/v8.11.4/

It would be useful when debugging docker images, to have the hash commit from a previous point in time of the image to compare images or extend a specific commit/nightly in another repo.

I propose we take same approach we do in core repo with the controller and invoker images, using the first 7 characters of the hash commit, which seems to be a standard practice.