ptarmiganlabs / butler-cw Goto Github PK

Add config options and associated code to Butler CW:

  # Uptime monitor
  uptimeMonitor:
    enabled: true                   # Should uptime messages be written to the console and log files?
    frequency: every 15 seconds     # https://bunkat.github.io/later/parsers.html
    logLevel: verbose               # Starting at what log level should uptime messages be shown?

Hello,
I get this error whenever I run "node index.js". Can you provide any insight? And please let me know if you need more information.

PS D:\butler-cw-master> node index.js
2019-03-29T12:40:07.301Z info: Starting Qlik Sense cache warmer.
2019-03-29T12:40:10.337Z verbose: Starting loading of appid a099cb6b-53fb-4e70-b213-2832073b0c02
(node:7508) UnhandledPromiseRejectionWarning: TypeError: Converting circular structure to JSON
at JSON.stringify ()
at loadAppIntoCache (D:\butler-cw-master\index.js:211:57)
at process._tickCallback (internal/process/next_tick.js:68:7)
(node:7508) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a
promise which was not handled with .catch(). (rejection id: 1)
(node:7508) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a
non-zero exit code.

Describe the bug
With the switch to GitHub Actions used for CI, everything relating to the previous CI solution should be removed.

Describe the bug
When running in a multi-arch (amd64 and arm64) k8s cluster, containers fail to start on arm64 nodes.

Expected behavior
Containers should automatically start on amd64, arm64 or arm nodes.

Solution idea
The issue might be caused by the variant field in the manifest. It seems that the manifests of other (working) multi-arch images Docker images don't have that variant field. Might be worth removing it as a first attempt at resolving the issue.

Is your feature request related to a problem? Please describe.
Restify is dated while Fastify is actively maintained and has a much larger surrounding ecosystem.
All Butler tools are migrated to Fastify.

Describe the bug
I don't know, maybe it is not a bug.

Screenshots

Describe environment:

• OS: Debian 10
• Containerisation no
• Version of Butler CW used 2.3.12

Expose health check endpoints from container, to allow container supervisors to better monitor the state of the container.

Is your feature request related to a problem? Please describe.
Improvement.

Currently the automatically created release packages includes quite a few files that are only used during development of Butler CW.
These files probably more than anything else confuse first-time users of Butler CW.

Suggestion is to only include files needed for standalone execution of Butler CW in release ZIP artefact.

Is your feature request related to a problem? Please describe.
The later library (which is used by Butler CW) is no longer maintained.
There's however a drop-in replacement in the form of @breejs/later.

Switch Butler CW to use the latter instead of the former.

Reducing the number of external dependencies is a good thing.

Improve CI/CD process to be more automated, including multi architecture support.

Is your feature request related to a problem? Please describe.
New feature idea:

Make it possible to trigger cache warm runs by calling a REST API that's exposed by Butler CW.
Settings in the main and schedule config files should be respect, the API call would just be a way to temporarily override schedules defined in the schedule config file.

Hello,
First of all, thanks so much for this work, it's very helpful. I am running into an issue shown in the attached image. Basically, when an app is scheduled to run, at every specified time it runs, there is an enigmaOpen error, but with no message:

error: enigmaOpen error: {}

Can you help me or at least explain to me why this is happening / if you've seen this before? The error is thrown at line 56:

 try {
        g = await s.open();
    } catch (err) {
        globals.logger.log('error', `enigmaOpen error: ${JSON.stringify(err)}`);
        return;
    }

The strange thing is that if doInitialLoad is set to "true" - it runs the first time successfully, but everything on the scheduler does not with the above message.

I have tried running natively as well as in docker, but the same issue happens.

Describe the bug
If heartbeats are enabled and sent to a https endpoint, the heartbeat call will fail with a "...unable to verify the first certificate" error.

Expected behavior
It should be possible to send heartbeats to either http or https endpoints.

Describe environment:
Bug verified in Node.js 14.5 on mac OS Big Sur, Docker and in Kubernetes.

Is your feature request related to a problem? Please describe.
On/off flag in config file to control whether or not to show when each app's first few runs will take place.

Addition to main config file:

scheduler:
  startup: 
    showPerAppSchedule: 
      enable: true        # Should the first itemCount scheduled runs be shown for each app, on startup?
      itemCount: 10

Add config option and corresponding code in Butler CW:

  # Docker health checks are used when running Butler CW as a Docker container. 
  # The Docker engine will call the container's health check REST endpoint with a set interval to determine
  # whether the container is alive/well or not.
  # If you are not running Butler CW in Docker you can safely disable this feature. 
  dockerHealthCheck:
    enabled: true         # Control whether a REST endpoint will be set up to serve Docker health check messages
    port: 12398           # Port the Docker health check service runs on (if enabled)

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Rate-Limited

These updates are currently rate-limited. Click on a checkbox below to force their creation now.

• fix(deps): update dependency mqtt to v4.3.8
• fix(deps): update dependency @breejs/later to v4.2.0
• fix(deps): update dependency commander to v11.1.0
• fix(deps): update dependency enigma.js to v2.14.0
• fix(deps): update dependency fs-extra to v11.2.0
• fix(deps): update dependency moment to v2.30.1
• fix(deps): update dependency ws to v8.17.0
• chore(deps): update actions/checkout action to v4
• chore(deps): update actions/setup-node action to v4
• chore(deps): update crazy-max/ghaction-virustotal action to v4
• chore(deps): update docker/build-push-action action to v5
• chore(deps): update docker/login-action action to v3
• chore(deps): update docker/metadata-action action to v5
• chore(deps): update docker/setup-buildx-action action to v3
• chore(deps): update docker/setup-qemu-action action to v3
• chore(deps): update github artifact actions to v4 (major) (actions/download-artifact, actions/upload-artifact)
• chore(deps): update github/codeql-action action to v3
• chore(deps): update googlecloudplatform/release-please-action action to v4
• chore(deps): update node.js to v20
• fix(deps): update dependency commander to v12
• fix(deps): update dependency eslint to v9
• fix(deps): update dependency eslint-config-prettier to v9
• fix(deps): update dependency winston-daily-rotate-file to v5
• 🔐 Create all rate-limited PRs at once 🔐

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• fix(deps): update dependency axios to v1.6.0 [security]
• fix(deps): update dependency config to v3.3.11
• fix(deps): update dependency eslint to v8.57.0
• fix(deps): update dependency eslint-config-prettier to v8.10.0
• fix(deps): update dependency eslint-plugin-import to v2.29.1
• fix(deps): update dependency fastify to v4.26.2
• fix(deps): update dependency winston to v3.13.0
• chore(deps): update dependency prettier to v3
• fix(deps): update dependency @octokit/rest to v20
• fix(deps): update dependency eslint-plugin-prettier to v5
• fix(deps): update dependency mqtt to v5
• Click on this checkbox to rebase all open PRs at once

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

A config file setting could be be used to control whether schedules are defined in UTC or local time of the computer where Butler CW is running.

Describe the bug
The default config file should have all non-critical features turned off by default.
These features can then be enabled as needed when installing Butler CW.

Specifically, disable (set to false) the following features in the template config file:
mqttConfig.out.enable: false
uptimeMonitor.enabled: false

This works perfectly on Apps without Section Access. I have few apps where section access in enabled. Can this work on apps enabled with Section Access?

The logging module of Butler CW does not currently handle error logging as well as it should.
Specifically, in some cases there is no stack trace or error message when errors occur.

This can be easily fixed by adding the following to the Winston.createLogger call:

winston.format.errors({ stack: true })

Describe the bug
In the spirit of keeping the Docker image nice and small, it should only include files needed for running Butler CW.
Specifically, files only needed during development should not be included.

Hi Göran I try to install the butler-cw on my Server and have a Problem with connectivity please check the image

Upgrading Sense Enterprise to Feb 2020 or beyond means that the Sense certificates will be recreated. The new certs must be exported and made available to Butler CW.
Failing to do so will cause Butler CW to stop working.

Update docs to highlight the above.

Switch to using release-please manifests, as done in the Butler project.

Describe the bug
I got the following Warning when running Butler-CW with MQTT disabled. I looked into it, and it looks in loadAppIntoCache it doesn't check for whether or not MQTT is enabled when it runs mqttClient.publish() on line 184. I think it needs to have the sort of if statement checking if mqttConfig.out.enable is set that we see on line 331.

(node:15104) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'publish' of undefined
    at [REDACTED]\index.js:184:24
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
(node:15104) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:15104) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

To Reproduce
Disable MQTT and don't put a valid URL in mqttconfig.broker.uri, then try to cache-warm an app.

Expected behavior
Ideally with the MQTT server disabled it wouldn't attempt to publish to it and we wouldn't get this warning.

Describe environment:

• OS: Windows Server 2019
• Containerisation - No
• Version of Butler CW used - 4.0.2
• Command used to start Butler CW - node index.js

Config file(s)

mqttConfig:
  out:
    enable: false
    tzFormat: UTC
    baseTopic: butler-cw/
  broker:
    uri: mqtt://0.0.0.0

Would offer user a more structured way of examining the REST API of Butler CW.

Having the same principles wrt source code formatting will greatly simplify things when it comes to maintaining the various Butler tools.
Update Butler CW's linting rules so they match for example Butler Sheet Icons' ditto.

Describe the bug
Having a schedule of every 55 seconds results in this execution plan:

2021-10-21T14:53:55.945Z
2021-10-21T14:54:00.945Z
2021-10-21T14:54:55.945Z
2021-10-21T14:55:00.945Z
2021-10-21T14:55:55.945Z
2021-10-21T14:56:00.945Z
2021-10-21T14:56:55.945Z
2021-10-21T14:57:00.945Z
2021-10-21T14:57:55.945Z
2021-10-21T14:58:00.945Z

Which is not what's expected.

This is however happening due to how the underlying scheduling library works (discussion here).
For now this will not be fixed in Butler CW. A proper/better implementation may be added in future versions of Butler CW.

I am trying to install butler-cw on a Windows 2016 server and Qlik Sense June 2018. The Qix engine is 12.120.7 (edited js file accordingly). When I go to run it, I get:
C:\butler-cw-master>node index.js
2018-08-31T23:22:48.919Z - info: Starting Qlik Sense cache warmer.
module.js:478
throw err;
^

Error: Cannot find module 'enigma.js/schemas/qix/12.170.2/schema.json'
at Function.Module._resolveFilename (module.js:476:15)
at Function.Module._load (module.js:424:25)
at Module.require (module.js:504:17)
at require (internal/module.js:20:19)
at Object. (C:\butler-cw-master\index.js:45:19)
at Module._compile (module.js:577:32)
at Object.Module._extensions..js (module.js:586:10)
at Module.load (module.js:494:32)
at tryModuleLoad (module.js:453:12)
at Function.Module._load (module.js:445:3)
at Module.runMain (module.js:611:10)
at run (bootstrap_node.js:394:7)
at startup (bootstrap_node.js:160:9)
at bootstrap_node.js:507:3

Is your feature request related to a problem? Please describe.
Some apps are updated every day at different times

Describe the solution you'd like
Is there any way to warm up the app after it reload?

Is your feature request related to a problem? Please describe.
New feature idea:

Make it possible to trigger cache warm runs by sending MQTT messages to Butler CW.
Settings in the main and schedule config files should be respect, the MQTT message would just be a way to temporarily override schedules defined in the schedule config file.

https://www.repostatus.org/

It would be nice to get some basic info about the underlying Qlik Sense environment written to the Butler logs, whenever Butler is started.
Things such as overall Sense version, QIX engine version, info about the hardware Sense is running on etc.

Discussed in #186

Store Butler CW's memory usage to InfluxDB, similar to how Butler SOS does this.

This will require a fair amount of new code in Butler CW, as well as new options in the main config file.
Something like this:

  # Uptime monitor
  uptimeMonitor:
    enabled: true                   # Should uptime messages be written to the console and log files?
    frequency: every 15 seconds     # https://bunkat.github.io/later/parsers.html
    logLevel: verbose               # Starting at what log level should uptime messages be shown?
    influxdb: 
      enabled: true    # Should data on Butler CW's own memory use be stored in Influxdb?
      hostIP: <IP or FQDN of Influxdb server>
      hostPort: <Port where Influxdb is listening>    # Optional. Default value=8086
      auth:
        enable: false                 # Does influxdb instance require authentication (true/false)?
        username: <username>          # Username for Influxdb authentication. Mandatory if auth.enable=true
        password: <password>          # Password for Influxdb authentication. Mandatory if auth.enable=true
      dbName: SenseOps
      # Default retention policy that should be created in InfluxDB when Butler CW creates a new database there. 
      # Any data older than retention policy threshold will be purged from InfluxDB.
      retentionPolicy:
        name: 2_weeks
        duration: 2w
        instanceTag: DEV              # Tag that can be used to differentiate data from multiple Butler CW instances

Describe the bug
I started butler-cw with parameter "freq: at 11:15 am" in config.
Bug 1 - Warming cache started immediately
Bug 2 - It use UTC time(in logs too), not local (maybe it is feature?)

To Reproduce
Use parameter "freq: at 11:15 am" in config

Expected behavior

1. Start according to schedule.
2. Use local time(?)

Describe environment:

• OS: Debian 10
• Containerisation no
• Version of Butler CW 3.1.1

Config file(s)
...
freq: at 11:15 am

The current way of doing Docker health checks in Butler CW uses a soon-to-be-deprecated feature in the Fastify library.

Instead do things the same way as in the Butler SOS project, using the fastify-healthcheck npm library.
Code examples available in Butler SOS repo.

FROM ….
LABEL maintainer="Göran Sander [email protected]"
LABEL description="lorem ipsum"

Is your feature request related to a problem? Please describe.
Make it possible, per app, to control whether the app should be loaded or not when Butler CW is started.
I.e. whether or not to do an initial cache warming run for the app in question.

Is your feature request related to a problem? Please describe.
Feature request.
Would be nice to get visual feedback on when the next cache warming will be done for the app that's being loaded right now.
Make it optional via setting in main config file.

Is your feature request related to a problem? Please describe.
New feature idea.
Currently the apps.yaml schedule file uses app IDs to determine which apps to cache warm.
That could be done using tags and/or custom properties instead.
This would be good as you would no longer need to keep track of app IDs, but rather control cache warming by more easy to remember tags and custom properties.

For tags, the solution could be:

• Let's assume there is a tag (available for apps) called "cachewarm-sales".
• Assign that tag to the sales apps that should be cache warmed.
• Enter that tag name in Butler CW's schedule file instead of the app's IDs.
• Butler CW will cache warm those apps according to the schedule in the config file.

For custom properties, the solution could look like this:

• In the schedule config file, for an entry you enter a) a custom property name, and b) the value which Butler CW should look for.
• If Butler CW finds any apps with the correct custom property/value combo, those apps will be cache warmed according to the schedule in the config file.

Hi,
I configured everything as per the instruction. but getting some error while trying to run node index.Js

these two i didnt fill

"owner": "",
"repo": "<repo name, e.g. qliksense-cache-warming>",

Copy the concept used in the Butler project.

Hello,

Receiving follow error, Could you please check it for me

C:\butler-cw-master>node index.js
WARNING: No configurations found in configuration directory:C:\butler-cw-master\config
WARNING: To disable this warning set SUPPRESS_NO_CONFIG_WARNING in the environment.
C:\butler-cw-master\node_modules\config\lib\config.js:203
throw new Error('Configuration property "' + property + '" is not defined');
^

Error: Configuration property "defaultLogLevel" is not defined
at Config.get (C:\butler-cw-master\node_modules\config\lib\config.js:203:11)
at Object. (C:\butler-cw-master\index.js:19:27)
at Module._compile (internal/modules/cjs/loader.js:688:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:699:10)
at Module.load (internal/modules/cjs/loader.js:598:32)
at tryModuleLoad (internal/modules/cjs/loader.js:537:12)
at Function.Module._load (internal/modules/cjs/loader.js:529:3)
at Function.Module.runMain (internal/modules/cjs/loader.js:741:12)
at startup (internal/bootstrap/node.js:285:19)
at bootstrapNodeJSCore (internal/bootstrap/node.js:739:3)

Is your feature request related to a problem? Please describe.
Feature.

MQTT is incredibly useful as an integration tool. Thus:
When an app is loaded into a server, send a MQTT message to a configurable MQTT broker and topic.

Change to node:16-bullseye-slim in Dockerfile.

Discussed in #184

After upgrading QS server april 2020 , cache warmer stopped working and showing up this error
2020-05-21T22:31:05.910Z error: enigmaOpen error: {"target":{"_events":{},"_eventsCount":4,"readyState":3,"protocol":"","_binaryType":"nodebuffer","_closeFrameReceived":false,"_closeFrameSent":false,"_closeMessage":"","_closeTimer":null,"_closeCode":1006,"_extensions":{},"_isServer":false,"_receiver":null,"_sender":null,"_socket":null,"url":"wss://server.domain.corp:4747/app/ba4ca9f6-1935-4509-8a8e-501b3d19d44d","_req":null},"type":"error","message":"read ECONNRESET","error":{"errno":"ECONNRESET","code":"ECONNRESET","syscall":"read"}}