Integration tests are still intermittently failing on authorisation.

Periodically a unit test will fail with 401: UNAUTHORISED.

If I "re-run failed jobs" is will reliably pass on the second run.

My hypothesis is that the API client involved in the test is attempting to make its calls with an expired token.

Token expiry is handled in the SDK in the OAuth component. This component encapsulates retrieving tokens from the token endpoint, caching them in memory and on disk, and providing a token to an API client when the API client wants to make an API call.

The OAuth client should be checking if it has a token in-memory or on-disk (the on-disk caching is for when applications are restarted), then checking if the token is expired or is likely to expire soon (there is a threshold setting that represents "this might expire before the call makes the roundtrip") and either requesting a new token from the endpoint to pass on or passing on the cached token.

Some tests - notably the Tasklist ones - will fail multiple calls when they do fail.

Otherwise, I've noticed that it happens later in the test suite, which leads me to think that it happens when a 300 second validity token expires and there is some race condition or logic error that means it is not correctly refreshed before being passed to the API client.

This is difficult to reproduce reliably.

Maybe it needs some specific test of the token refresh timing logic? Or maybe there is something obvious in the code that I am missing.

Tests of Modeler API on Self-Managed currently fail due to not being authorized.

This could be because the OAuth token for Modeler on SaaS needs an audience of api.cloud.camunda.io and Self-Managed, apparently, it should be requested with no audience. See here: https://docs.camunda.io/docs/next/apis-tools/web-modeler-api/authentication/?authentication=self-managed

Getting an OAuth token to use the Modeler API has some significant differences in ergonomics between Self-Managed and SaaS that have an impact on the design of the SDK.

Issue 1 - Different credential sets between SaaS and Self-Managed

• On SaaS, the Modeler API client can be same client credentials as the Admin Console client credential. This is scoped to the organisation, not to a cluster. This means that you cannot have an id/secret pair that accesses both Zeebe and the Modeler.
• On Self-Managed, the Modeler API client can be the same credential set as a Zeebe client. That client can access both Zeebe and the Modeler.

On SaaS, the env var for an admin console client credential id is CAMUNDA_CONSOLE_CLIENT_ID.
On SaaS, the env var for a cluster application client credential id is ZEEBE_CLIENT_ID.

If the user is accessing Modeler on SaaS, we need to use the CAMUNDA_CONSOLE_CLIENT_ID. We can determine if the application is talking to SaaS by the OAuth URL, which is a known value for SaaS.

If the user is accessing Modeler on Self-Managed (ie: using an OAuth URL other than SaaS), we will use the ZEEBE_CLIENT_ID to request a token. If Modeler is moved out of the application credential pool on Self-Managed in the future, we will deal with it at that point.

Issue 2 - Different audience for token request between SaaS and Self-Managed

• On SaaS, a token for use with the Modeler API needs to be requested with an audience of api.cloud.camunda.io.
• On Self-Managed, the audience field should be omitted.

On SaaS, again detected via the OAuth endpoint URL, we will set the audience. On Self-Managed, we will omit the audience field, unless an explicit CAMUNDA_OAUTH_MODELER_AUDIENCE value is set.

Hi ! I know this version is in alpha, but I wanted to submit my issue in case of.
An error is thrown when starting a NestJS project with camunda8/[email protected] or camunda8/[email protected]

Error: Cannot find module 'lib'
Require stack:
- /home/xxx/workspace/projects/edl/poc/camunda/node_modules/.pnpm/@[email protected]/node_modules/@camunda8/sdk/dist/admin/lib/AdminApiClient.js
- /home/xxx/workspace/projects/edl/poc/camunda/node_modules/.pnpm/@[email protected]/node_modules/@camunda8/sdk/dist/admin/index.js
- /home/xxx/workspace/projects/edl/poc/camunda/node_modules/.pnpm/@[email protected]/node_modules/@camunda8/sdk/dist/index.js
- /home/xxx/workspace/projects/edl/poc/camunda/dist/process.service.js
- /home/xxx/workspace/projects/edl/poc/camunda/dist/app.controller.js
- /home/xxx/workspace/projects/edl/poc/camunda/dist/app.module.js
- /home/xxx/workspace/projects/edl/poc/camunda/dist/main.js

SDK Component

All

Expected Behavior

The 'lib' is found

Current Behavior

The 'lib' is not found

Possible Solution

the path of 'lib' (or any other require() in the compiled js) seems to target a file in a same folder.

Steps to Reproduce

1. nest new my-project
2. Call the camunda8 constructor anywhere (ex: const camunda = new Camunda8() in the controller)
3. npm run start:dev

Context (Environment)

NodeJS 20
NestJS@10 project
PNPM

The SDK uses the debug library for debugging. Users can set a DEBUG environment variable to enable debugging information to help trace issues.

Make all the values normalized and document them.

camunda:tasklist
camunda:oauth
camunda:grpc
camunda:worker
camunda:modeler
camunda:adminconsole
camunda:optimize
camunda:zeebeclient
camunda:operate

This is a smoke test for the release.

Replace the zeebe-node dependency in the Desktop Modeler with the new SDK, and raise a PR for Desktop Modeler.

On a push to alpha and main branches, we only need to run the publish workflow.

For PRs, create a single parallelised test without the publish step in it.

Something to think about: we can only run one SaaS test at a time.

Create a complete integration test scenario using all the things in a coordinated way.

See here for details: camunda-community-hub/camunda-saas-oauth-nodejs#21.

Remove expiry timer and reimplement as an expiry check on token request.

Tests in GitHub CI for Self-Managed are failing.

The cause seems to be a race condition in token refresh. At the moment, the OAuth component caches the token and compares the expiry time to the current time, and requests a new token if the expiry time is greater than or equal to the current time.

If the token expires in 1ms it will be used for a call, but this will probably result in it expiring before it hits the service.

To deal with this, I am adding a new configuration field: CAMUNDA_OAUTH_TOKEN_REFRESH_THRESHOLD_MS. It defaults to 1000 (1 second).

This represents the lead time to refresh the token. So, by default a cached token will be refreshed 1 second before it expires, and this can be tuned by the user depending on their environment.

The Node.js client provides a BroadcastSignalCommand for broadcasting tenant-aware signals in Zeebe. These commands should support multi-tenancy by exposing an optional tenantId property/method.

The following error codes may be returned:

PERMISSION_DENIED (code: 7)
when a user attempts to broadcast a signal of a tenant they are not authorized for, when multi-tenancy is enabled.

INVALID_ARGUMENT (code: 3)
For a provided tenant id, when multi-tenancy is disabled
For a missing tenant id, when multi-tenancy is enabled
For an invalid tenant id (i.e. doesn't match the pre-defined format), when multi-tenancy is enabled.

Java client issue: camunda/camunda#13558

At the moment all tests are running on the TypeScript source. The package entry point and any misconfigurations in the transpilation are not tested. This means that a package can be released via automation that doesn't work (See #114).

To make sure that this doesn't happen, we need a test that transpiles the code and then runs a smoke test on it using the package entry point.

What does "supported" mean?

This is the official supported-by-Camunda Nodejs SDK for Camunda Platform 8.

The Node.js SDK will not always support all features of Camunda Platform 8 immediately upon their release. Complete API coverage for a platform release will lag behind the platform release.

Prioritisation of implementing features is influenced by customer demand.

DeleteResource is in the gRPC API protocol, and support for deleting deployed Forms is coming.

Add ZBClient.deleteResource and test.

SDK Component

Operate

Expected Behavior

The following methods work

• searchFlownodeInstances
• getFlownodeInstance

Current Behavior

It looks like methods don't call correct endpoints

• searchFlownodeInstances calls the flownodes/search but should call the flownode-instances/search
• getFlownodeInstance calls the flownodes/${key} but should call the flownode-instances/${key}

Context (Environment)

"@camunda8/operate": "^8.4.0"

The Tasklist API is moving to the Zeebe Gateway. See here: camunda/camunda#15622

How this is implemented in the SDK will depend on the API surface area that we expose to developers.

It may be possible to do this with no change to the API surface area.

This will be targeted to post-8.5.

At the moment there is some sideways configuration of the grpc channel in the Zeebe client. This means that configuration cannot be overridden via the constructor.

This needs to be threaded down into the components.

The following repositories need to be archived:

• https://github.com/camunda-community-hub/camunda-8-sdk-node-js
• https://github.com/camunda-community-hub/tasklist-client-node-js
• https://github.com/camunda-community-hub/zeebe-client-node-js
• https://github.com/camunda-community-hub/console-client-node-js
• https://github.com/camunda-community-hub/modeler-client-node-js
• https://github.com/camunda-community-hub/camunda-8-credentials-from-env
• https://github.com/camunda-community-hub/camunda-saas-oauth-nodejs

Packages that need to be deprecated with a notice:

• @camunda8/console
• @camunda8/tasklist
• @camunda8/oauth
• @camunda8/operate
• @camunda8/optimize
• @camunda8/modeler
• @camunda8/zeebe
• camunda-8-credentials-from-env
• camunda-8-sdk
• camunda-saas-oauth
• zeebe-node

deployProcess is deprecated and has been replaced by deployResource.

This will impact the tests, which use a lot of deployProcess.

Similar to #80, the Zeebe worker variable payload may contain int64 number values.

Without a user-provided Dto class to leverage #78, we can't know the variable type - but we can detect an unsafe number value if we parse with lossless-json.

An option could be to specify via configuration how variables with a number value that is unsafe are handled.

If there is no Dto provided, we could default to throwing during the parse operation. This stops the application from operating with bad data.

And we could allow the user to set a default behaviour, "coerce to BigInt" or "coerce to string".

A Self-Managed Multi-tenancy stack, running locally or in CI, should function as a test environment for multi-tenancy tests that prove the operation as expected of the SDK in the multi-tenancy environment.

• Get Web Modeler running in a multi-tenant environment
• SDK functions fail when run in multi-tenant without tenantId configured
• SDK functions succeed when run in multi-tenant with tenantId configured

Here are the things that need to be dealt with:

• The web browser does not support gRPC
• Web browsers cannot easily support custom SSL certificates
• Secrets cannot be secured in the browser

These three mean that the web browser environment requires distinct strategies from the server-side environment.

Creating and maintaining differential strategies for the two environments (client- and server-side) is a significant engineering effort.

The ZBClient component supports a lot of backward-compatible constructor options, including multiple constructor signatures.

This makes modification and maintenance a challenge. While adding multi-tenancy support to the Zeebe client, this is causing a lot of friction and accidental complexity.

For the 8.5 release, I will simplify the constructor signature to be a Partial<ZBClientConfig>, and then make the configuration hydrator do the following:

Read all environment variables into a config map, then overwrite that map with any explicit configuration passed to the constructor.

This makes the configuration explicit and simple. The zero-conf constructor is default. All configuration from environment variables is applied, then anything that is explicitly passed to a ZBClient constructor overrides it.

SDK Component

Zeebe

The Zeebe Gateway has a new RPC StreamJobs to open a long-lived stream that essentially "pushes" jobs to the worker.

This obviates polling loops.

camunda/camunda#14152

Environment variable to enable it:

ZEEBE_CLIENT_WORKER_STREAMENABLED

See also the documentation about the general concept, including how back pressure works and can be implemented for your custom client: https://stage.docs.camunda.io/docs/next/components/concepts/job-workers/#job-streaming

A future release of Zeebe will support migrating a running process instance to a newer version of the process model. Simple process migration is targeted for 8.4.

The client API for this has been added in stub form with this PR: camunda/camunda#15199.

Add this API stub to the Zeebe Node client.

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Rate-Limited

These updates are currently rate-limited. Click on a checkbox below to force their creation now.

• fix(deps): update dependency @grpc/proto-loader to v0.7.12
• fix(deps): update dependency reflect-metadata to v0.2.2
• chore(deps): update dependency typescript to v5.4.3
• chore(deps): update typescript-eslint monorepo to v6.21.0 (@typescript-eslint/eslint-plugin, @typescript-eslint/parser)
• fix(deps): update dependency @grpc/grpc-js to v1.10.6
• fix(deps): update dependency neon-env to ^0.2.0
• chore(deps): update actions/checkout action to v4
• chore(deps): update actions/setup-node action to v4
• chore(deps): update commitlint monorepo to v19 (major) (@commitlint/cli, @commitlint/config-conventional)
• chore(deps): update dependency @types/debug to v4
• chore(deps): update dependency @types/node to v20
• chore(deps): update dependency delay to v6
• chore(deps): update dependency husky to v9
• chore(deps): update dependency jest to v29
• chore(deps): update dependency semantic-release to v23
• chore(deps): update docker.elastic.co/elasticsearch/elasticsearch docker tag to v8
• chore(deps): update typescript-eslint monorepo to v7 (major) (@typescript-eslint/eslint-plugin, @typescript-eslint/parser)
• fix(deps): update dependency chalk to v5
• fix(deps): update dependency got to v14
• fix(deps): update dependency long to v5
• fix(deps): update dependency node-fetch to v3
• fix(deps): update dependency promise-retry to v2
• fix(deps): update dependency typed-duration to v2
• fix(deps): update dependency uuid to v9 (uuid, @types/uuid)
• 🔐 Create all rate-limited PRs at once 🔐

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• chore(deps): update dependency @commitlint/config-conventional to v18.6.3
• chore(deps): update dependency typedoc to v0.25.12
• chore(deps): update docker.elastic.co/elasticsearch/elasticsearch docker tag to v7.17.19
• fix(deps): update dependency fast-xml-parser to v4.3.6
• chore(deps): update camunda/zeebe docker tag to v8.5.0
• chore(deps): update dependency @types/node to v20.12.3
• chore(deps): update dependency eslint to v8.57.0
• chore(deps): update dependency eslint-plugin-prettier to v5.1.3
• chore(deps): update dependency prettier to v3.2.5
• chore(deps): update dependency typedoc to ^0.25.0
• Click on this checkbox to rebase all open PRs at once

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

This one is a stretch goal for the release.

The new clients have a clean structure with all interfaces in a Dto namespace.

The ZeebeGrpcClient has interfaces all over the place. It will be a significant documentation win to coalesce them all into a Dto namespace.

The SDK API docs task needs to run in CI and automatically update the docs.

We'll run it on main, and tag the methods to indicate what release they were implemented in.

The lossless Json parser from #78 needs to be used for the convenience method getJSONVariablesforProcess.

User variables may contain int64 values.

We should allow the user to optionally pass in a class that extends LosslessDto to be able to get the variable data back with no loss of precision.

As a fall-back (when no user Dto is passed in), we could parse it using lossless-json, and then convert any unsafe numbers into BigInt or string.

Making it a string would make it consistent with how we handle int64 in the SDK - however, without metadata about the payload typing, we are reduced to examining the actual payload and we can only detect int64 when the value exceeds the range of the JS number type.

This means that the type of the field would vary depending on the value - which is strange behaviour. This could lead to application errors where string concatenation could occur when arithmetic addition was expected.

Workflow fragment:

  Post-Failure:
    needs: Build_nightly
    if: failure()
    runs-on: ubuntu-latest
    steps:
    - name: Post to a Slack channel
      uses: slackapi/[email protected]
      with:
        channel-id: ${{ secrets.SLACK_CHANNEL_ID }}
        slack-message: "Nightly build failed. <https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}|Go to the build.>"
      env:
        SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}

Installation and Configuration docs:
https://docs.camunda.io/docs/next/self-managed/console-deployment/configuration/

Docker Image:
https://registry.camunda.cloud/harbor/projects/28/repositories

This will not happen until after the 8.5 release.

Write a quickstart for the Camunda docs and raise a PR.

Check out the guide here: https://docs.github.com/en/code-security/getting-started/dependabot-quickstart-guide#enabling-dependabot-for-your-repository

Then, for code scanning GitHub Advanced Security. Check out the guide at https://docs.github.com/en/code-security/code-scanning/enabling-code-scanning

Desktop Modeler CODE_SCANNING action for reference as well: https://github.com/camunda/camunda-modeler/blob/develop/.github/workflows/CODE_SCANNING.yml

The JSON number value passed by the REST API for processInstanceKey (for example) is of type $int64.

This can't be reliably represented by the JavaScript number type, which is a 2^53 range floating point value.

This means that systems that are running for a long time can generate keys that cannot be represented by the SDK.

The gRPC library deals with this by parsing int64 to a string representation. Since we don't do arithmetic on keys, it might as well be a string.

The REST library that I am currently using, however, (got) represents these as number, leading to two issues:

1. The imprecision issue that will become an L1 in production when a system hits key values that cannot be represented as the JS number type.

2. The impedance mismatch that ZeebeGrpcClient returns long keys as type string, and the other APIs expect a number type as input; and vice versa.

• https://stackoverflow.com/questions/64402932/why-java-long-max-is-not-same-with-vue-axios-data-how-to-fix-that
• https://stackoverflow.com/questions/13502398/json-integers-limit-on-size
• https://jsoneditoronline.org/indepth/parse/why-does-json-parse-corrupt-large-numbers/

I'm looking for alternative REST clients that can serialise long integer types as JavaScript string.

This method should take a more generic signature.

npm ci respects the package-lock.json file, resulting in deterministic installs.

However, users will install the package using npm i, which does not.

So, running npm ci in automated tests is not going to replicate the user's experience.

Create a top-level configuration key for a custom useragent string. Add this to the standard string in a centralized place.

Unit tests can no longer be run without an integration environment. The global setup and teardown should have code in them to skip the Operate and Zeebe client creation if they are not running in an integration test.

The SDK package tracks Camunda Platform 8 minor versioning. Feature releases to support current Platform minor version features result in a patch release of the SDK.

The repository uses []semantic-release](https://github.com/semantic-release/semantic-release) to create releases. Because we track the Camunda 8 Platform minor version, we treat feature implementation during a minor release cycle as a patch release rather than a minor release.

Creating a commit with a feat commit message will cause the package version patch release number to increment. To update the minor version, a commit with the type minor is needed.

A commit with the type release will trigger a patch release even if there are no features or fixes. This can be used on the alpha branch to test automation if needed.

Expected Behavior

No error should pop up

Current Behavior

A cluster is created, multi-tenancy enabled. 2 tenants are created, blue and green.

A script is created:

• deploying the same simple process on both tenants. This process contains a service task
• Starting a worker, with a filter on each tenant (so two workers are started)
• Every 5 seconds, a new process instance is created on the green tenant
• every 20 seconds, a new process instance is created on the red tenant

Process instances are created, and workers execute jobs, but from time to time, an Error: 16 UNAUTHENTICATED: is visible in the log. Even so
, all service tasks are completed.

Possible Solution

No solution

Steps to Reproduce

Create a cluster, with two tenants red and green
create a client ID / Client Secret
Run the script

Context (Environment)

Multi-tenancy usage

Detailed Description

Log visible:

 ##### Started/Job RED: 0/0, GREEN: 1/2
------  Green process instance started: [2251799813686010]
>>>> Execute GreenJob PI=[2251799813686010] tenant=[green]
15:42:07.412 | zeebe |  [service-task] ERROR: Grpc Stream Error: 16 UNAUTHENTICATED: Expected Identity to provide authorized tenants, see cause for details


 ##### Started/Job RED: 0/0, GREEN: 2/3
------  Green process instance started: [4503599627371180]
>>>> Execute GreenJob PI=[4503599627371180] tenant=[green]
15:42:12.524 | zeebe |  [service-task] ERROR: Grpc Stream Error: 16 UNAUTHENTICATED: Expected Identity to provide authorized tenants, see cause for details


 ##### Started/Job RED: 0/0, GREEN: 3/4
(node:226) UnhandledPromiseRejectionWarning: Error: 16 UNAUTHENTICATED: Expected Identity to provide authorized tenants, see cause for details
    at callErrorFromStatus (/mnt/d/pym/CamundaDrive/Support & Consulting/S-19554 Avertra NodeJS/avertra-test/node_modules/@grpc/grpc-js/build/src/call.js:31:19)
    at Object.onReceiveStatus (/mnt/d/pym/CamundaDrive/Support & Consulting/S-19554 Avertra NodeJS/avertra-test/node_modules/@grpc/grpc-js/build/src/client.js:192:76)
    at listener.onReceiveStatus.processedStatus (/mnt/d/pym/CamundaDrive/Support & Consulting/S-19554 Avertra NodeJS/avertra-test/node_modules/@grpc/grpc-js/build/src/call-interface.js:78:35)
    at Object.onReceiveStatus (/mnt/d/pym/CamundaDrive/Support & Consulting/S-19554 Avertra NodeJS/avertra-test/node_modules/zeebe-node/dist/lib/GrpcClient.js:97:36)
    at InterceptingListenerImpl.onReceiveStatus (/mnt/d/pym/CamundaDrive/Support & Consulting/S-19554 Avertra NodeJS/avertra-test/node_modules/@grpc/grpc-js/build/src/call-interface.js:73:23)
    at Object.onReceiveStatus (/mnt/d/pym/CamundaDrive/Support & Consulting/S-19554 Avertra NodeJS/avertra-test/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:360:141)
    at Object.onReceiveStatus (/mnt/d/pym/CamundaDrive/Support & Consulting/S-19554 Avertra NodeJS/avertra-test/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:323:181)
    at process.nextTick (/mnt/d/pym/CamundaDrive/Support & Consulting/S-19554 Avertra NodeJS/avertra-test/node_modules/@grpc/grpc-js/build/src/resolving-call.js:99:78)
    at process._tickCallback (internal/process/next_tick.js:61:11)
for call at
    at ServiceClientImpl.makeUnaryRequest (/mnt/d/pym/CamundaDrive/Support & Consulting/S-19554 Avertra NodeJS/avertra-test/node_modules/@grpc/grpc-js/build/src/client.js:160:32)
    at ServiceClientImpl.<anonymous> (/mnt/d/pym/CamundaDrive/Support & Consulting/S-19554 Avertra NodeJS/avertra-test/node_modules/@grpc/grpc-js/build/src/make-client.js:105:19)
    at Promise (/mnt/d/pym/CamundaDrive/Support & Consulting/S-19554 Avertra NodeJS/avertra-test/node_modules/zeebe-node/dist/lib/GrpcClient.js:272:47)
    at process._tickCallback (internal/process/next_tick.js:68:7)
(node:226) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:226) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
--

Possible Implementation

red-green.txt
simple-bpmn.txt

In the published package, type information is missing.

This is because the path mapping in the project is not transpiled.

Adding these two fixes it:

https://github.com/nonara/ts-patch
https://github.com/LeDDGroup/typescript-transform-paths

SDK Component

OAuth component, and Zeebe client (if it doesn't use the OAuth component)

Expected Behavior

Has the ability to specify the token scope via env var and constructor to support custom identity providers. See: camunda/camunda-modeler#4102

Current Behavior

OAuth hardcodes the scopes per component.

The repository needs to be moved to the Camunda GitHub organisation, and all the references and links need to be updated.

• Move repo to Camunda org
• Make sure I have all necessary permissions on the new repo
• Update links to API docs
• Update package.json
• Update README.md

"Randomly", a test will fail in the suite with a 401 Unauthorised response from an API.

Example:

 console.log
    Failed to search for process instances for 2251799813685806

      at src/zeebe/lib/cancelProcesses.ts:16:12

  console.log
    HTTPError: Response code 401 (Unauthorized) (request to http://localhost:8081/v1/process-instances/search)
        at Request.<anonymous> (/Users/jwulf/workspace/c8-sdk/node_modules/got/dist/source/as-promise/index.js:118:42)

This is probably due to an expired token being used for the request.

To debug this, I'll need to write a lifecycle unit test for the OAuth token expiry strategy.

The workflow is like this:

All development work should be done via PRs against the alpha branch.

When a PR is merged to alpha, the tests are run, then semantic-release runs to determine if a new release is required. If it is, then an alpha package is published to NPM.

Production releases are accomplished by opening a PR from alpha to main.

When a PR is merged into main, semantic-release runs and if a new package release is required, a package is published to NPM.