I was trying to evaluate Zenko by using the helm chart. i tried v1.1.0 and latest 1.1.3 but both show the same issue on kubernates 1.15 deployed via rancher 2.3.2:
the helm is deployed and several containers are up and running, for instance redis and mongodb are both "green" while zenko specific ones will never complete, they are all from zenko/backbeat:8.1.14 image and show the same error.
Any hints?
thank you.
kubectl logs zenko-backbeat-api-5566bdfd68-mfsgt
Log level has been modified to info
[email protected] start /usr/src/app
node index.js
(node:26) DeprecationWarning: current URL string parser is deprecated, and will be removed in a future version. To use the new parser, pass option { useNewUrlParser: true } to MongoClient.connect.
{"name":"mdManagement","time":1572445217762,"level":"info","message":"connected to mongodb","hostname":"zenko-backbeat-api-5566bdfd68-mfsgt","pid":26}
(node:26) DeprecationWarning: collection.update is deprecated. Use updateOne, updateMany, or bulkWrite instead.
{"name":"BackbeatServer:index","time":1572445217903,"error":{"message":"NoSuchKey","code":404,"stack":"Error: NoSuchKey\n at Object.keys.filter.forEach.index (/usr/src/app/node_modules/arsenal/lib/errors.js:29:31)\n at Array.forEach ()\n at errorsGen (/usr/src/app/node_modules/arsenal/lib/errors.js:28:12)\n at Object. (/usr/src/app/node_modules/arsenal/lib/errors.js:35:18)\n at Module._compile (internal/modules/cjs/loader.js:776:30)\n at Object.Module._extensions..js (internal/modules/cjs/loader.js:787:10)\n at Module.load (internal/modules/cjs/loader.js:653:32)\n at tryModuleLoad (internal/modules/cjs/loader.js:593:12)\n at Function.Module._load (internal/modules/cjs/loader.js:585:3)\n at Module.require (internal/modules/cjs/loader.js:690:17)","name":"Error"},"level":"error","message":"could not load management db","hostname":"zenko-backbeat-api-5566bdfd68-mfsgt","pid":26}
{"name":"mdManagement","time":1572445222966,"level":"info","message":"connected to mongodb","hostname":"zenko-backbeat-api-5566bdfd68-mfsgt","pid":26}
{"name":"BackbeatServer:index","time":1572445223057,"error":{"message":"NoSuchKey","code":404,"stack":"Error: NoSuchKey\n at Object.keys.filter.forEach.index (/usr/src/app/node_modules/arsenal/lib/errors.js:29:31)\n at Array.forEach ()\n at errorsGen (/usr/src/app/node_modules/arsenal/lib/errors.js:28:12)\n at Object. (/usr/src/app/node_modules/arsenal/lib/errors.js:35:18)\n at Module._compile (internal/modules/cjs/loader.js:776:30)\n at Object.Module._extensions..js (internal/modules/cjs/loader.js:787:10)\n at Module.load (internal/modules/cjs/loader.js:653:32)\n at tryModuleLoad (internal/modules/cjs/loader.js:593:12)\n at Function.Module._load (internal/modules/cjs/loader.js:585:3)\n at Module.require (internal/modules/cjs/loader.js:690:17)","name":"Error"},"level":"error","message":"could not load management db","hostname":"zenko-backbeat-api-5566bdfd68-mfsgt","pid":26}
{"name":"mdManagement","time":1572445228118,"level":"info","message":"connected to mongodb","hostname":"zenko-backbeat-api-5566bdfd68-mfsgt","pid":26}
{"name":"BackbeatServer:index","time":1572445228203,"error":{"message":"NoSuchKey","code":404,"stack":"Error: NoSuchKey\n at Object.keys.filter.forEach.index (/usr/src/app/node_modules/arsenal/lib/errors.js:29:31)\n at Array.forEach ()\n at errorsGen (/usr/src/app/node_modules/arsenal/lib/errors.js:28:12)\n at Object. (/usr/src/app/node_modules/arsenal/lib/errors.js:35:18)\n at Module._compile (internal/modules/cjs/loader.js:776:30)\n at Object.Module._extensions..js (internal/modules/cjs/loader.js:787:10)\n at Module.load (internal/modules/cjs/loader.js:653:32)\n at tryModuleLoad (internal/modules/cjs/loader.js:593:12)\n at Function.Module._load (internal/modules/cjs/loader.js:585:3)\n at Module.require (internal/modules/cjs/loader.js:690:17)","name":"Error"},"level":"error","message":"could not load management db","hostname":"zenko-backbeat-api-5566bdfd68-mfsgt","pid":26}
{"name":"mdManagement","time":1572445233246,"level":"info","message":"connected to mongodb","hostname":"zenko-backbeat-api-5566bdfd68-mfsgt","pid":26}
{"name":"BackbeatServer:index","time":1572445234168,"error":{"message":"NoSuchKey","code":404,"stack":"Error: NoSuchKey\n at Object.keys.filter.forEach.index (/usr/src/app/node_modules/arsenal/lib/errors.js:29:31)\n at Array.forEach ()\n at errorsGen (/usr/src/app/node_modules/arsenal/lib/errors.js:28:12)\n at Object. (/usr/src/app/node_modules/arsenal/lib/errors.js:35:18)\n at Module._compile (internal/modules/cjs/loader.js:776:30)\n at Object.Module._extensions..js (internal/modules/cjs/loader.js:787:10)\n at Module.load (internal/modules/cjs/loader.js:653:32)\n at tryModuleLoad (internal/modules/cjs/loader.js:593:12)\n at Function.Module._load (internal/modules/cjs/loader.js:585:3)\n at Module.require (internal/modules/cjs/loader.js:690:17)","name":"Error"},"level":"error","message":"could not load management db","hostname":"zenko-backbeat-api-5566bdfd68-mfsgt","pid":26}
{"name":"mdManagement","time":1572445239220,"level":"info","message":"connected to mongodb","hostname":"zenko-backbeat-api-5566bdfd68-mfsgt","pid":26}
{"name":"BackbeatServer:index","time":1572445239764,"error":{"message":"NoSuchKey","code":404,"stack":"Error: NoSuchKey\n at Object.keys.filter.forEach.index (/usr/src/app/node_modules/arsenal/lib/errors.js:29:31)\n at Array.forEach ()\n at errorsGen (/usr/src/app/node_modules/arsenal/lib/errors.js:28:12)\n at Object. (/usr/src/app/node_modules/arsenal/lib/errors.js:35:18)\n at Module._compile (internal/modules/cjs/loader.js:776:30)\n at Object.Module._extensions..js (internal/modules/cjs/loader.js:787:10)\n at Module.load (internal/modules/cjs/loader.js:653:32)\n at tryModuleLoad (internal/modules/cjs/loader.js:593:12)\n at Function.Module._load (internal/modules/cjs/loader.js:585:3)\n at Module.require (internal/modules/cjs/loader.js:690:17)","name":"Error"},"level":"error","message":"could not load management db","hostname":"zenko-backbeat-api-5566bdfd68-mfsgt","pid":26}
helm install output
NAME: zenko
LAST DEPLOYED: Wed Oct 30 12:32:10 2019
NAMESPACE: default
STATUS: DEPLOYED
RESOURCES:
==> v1/ClusterRole
NAME AGE
zenko-cosmos-operator 88s
zenko-grafana-clusterrole 88s
==> v1/ClusterRoleBinding
NAME AGE
zenko-cosmos-operator 88s
zenko-grafana-clusterrolebinding 88s
==> v1/ConfigMap
NAME DATA AGE
zenko-backbeat-grafana-dashboard 1 89s
zenko-cloudserver-grafana-dashboard 1 89s
zenko-grafana 1 89s
zenko-grafana-config-dashboards 1 89s
zenko-grafana-dashboards-json 0 89s
zenko-jmx-exporter 1 89s
zenko-mongodb-replicaset-dashboard 1 89s
zenko-mongodb-replicaset-init 1 89s
zenko-mongodb-replicaset-mongodb 1 89s
zenko-mongodb-replicaset-tests 1 89s
zenko-prometheus-server 3 89s
zenko-redis-ha-configmap 3 89s
zenko-redis-ha-probes 2 89s
zenko-zenko-grafana-datasource 1 89s
zenko-zenko-queue-config 1 89s
zenko-zenko-queue-manager-bootstrap 1 89s
zenko-zenko-queue-metrics 1 89s
==> v1/Deployment
NAME READY UP-TO-DATE AVAILABLE AGE
zenko-cosmos-operator 0/1 1 0 87s
zenko-cosmos-scheduler 0/1 1 0 87s
zenko-zenko-queue-manager 0/1 1 0 87s
==> v1/HorizontalPodAutoscaler
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
zenko-cloudserver Deployment/zenko-cloudserver /80% 2 4 4 86s
==> v1/Job
NAME COMPLETIONS DURATION AGE
zenko-zenko-queue-config-ab364206 0/1 85s 86s
zenko-zenko-queue-manager-bootstrap-0c3fbfe6 0/1 85s 86s
==> v1/PersistentVolumeClaim
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
zenko-s3-data Bound pvc-8c92c229-c36a-4dce-b1f3-719f18979802 90Gi RWO nfs-client 88s Filesystem
==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
zenko-backbeat-api-5566bdfd68-mfsgt 0/1 ContainerCreating 0 84s
zenko-backbeat-gc-consumer-574d5d69cf-9mk94 0/1 ContainerCreating 0 86s
zenko-backbeat-gc-consumer-574d5d69cf-bhxlp 0/1 ContainerCreating 0 86s
zenko-backbeat-gc-consumer-574d5d69cf-gmn9h 0/1 ContainerCreating 0 86s
zenko-backbeat-ingestion-consumer-db4b76459-222d5 0/1 ContainerCreating 0 84s
zenko-backbeat-ingestion-consumer-db4b76459-hkfvm 0/1 ContainerCreating 0 85s
zenko-backbeat-ingestion-consumer-db4b76459-hwdbr 0/1 ContainerCreating 0 84s
zenko-backbeat-ingestion-producer-69589dfdc6-n6r49 0/1 Pending 0 82s
zenko-backbeat-lifecycle-bucket-processor-78bdf96fc7-8qqmg 0/1 ContainerCreating 0 82s
zenko-backbeat-lifecycle-bucket-processor-78bdf96fc7-jk54t 0/1 ContainerCreating 0 83s
zenko-backbeat-lifecycle-bucket-processor-78bdf96fc7-zmp8j 0/1 ContainerCreating 0 82s
zenko-backbeat-lifecycle-conductor-65d5ccb9f6-99k6k 0/1 ContainerCreating 0 84s
zenko-backbeat-lifecycle-object-processor-7cc4545cfc-2g5m2 0/1 ContainerCreating 0 84s
zenko-backbeat-lifecycle-object-processor-7cc4545cfc-8twnk 0/1 ContainerCreating 0 83s
zenko-backbeat-lifecycle-object-processor-7cc4545cfc-phxxt 0/1 ContainerCreating 0 83s
zenko-backbeat-replication-data-processor-86f5dbcd9-5hlzn 0/1 ContainerCreating 0 86s
zenko-backbeat-replication-data-processor-86f5dbcd9-tsbhl 0/1 ContainerCreating 0 86s
zenko-backbeat-replication-data-processor-86f5dbcd9-vk7lv 0/1 ContainerCreating 0 86s
zenko-backbeat-replication-populator-749cfd8d59-5rs9r 0/1 ContainerCreating 0 82s
zenko-backbeat-replication-status-processor-b8cfd5785-b468l 0/1 ContainerCreating 0 82s
zenko-backbeat-replication-status-processor-b8cfd5785-gkrvt 0/1 ContainerCreating 0 82s
zenko-backbeat-replication-status-processor-b8cfd5785-k68p9 0/1 ContainerCreating 0 82s
zenko-cloudserver-66b4d45b4c-4j9tw 0/1 Terminating 0 85s
zenko-cloudserver-66b4d45b4c-cpghv 0/1 Terminating 0 85s
zenko-cloudserver-66b4d45b4c-d8xj9 0/1 Terminating 0 85s
zenko-cloudserver-66b4d45b4c-dj2lv 0/1 Terminating 0 85s
zenko-cloudserver-66b4d45b4c-djhvx 0/1 Terminating 0 85s
zenko-cloudserver-66b4d45b4c-kkz6q 0/1 ContainerCreating 0 86s
zenko-cloudserver-66b4d45b4c-mq4z6 0/1 Terminating 0 85s
zenko-cloudserver-66b4d45b4c-n9zdh 0/1 Terminating 0 85s
zenko-cloudserver-66b4d45b4c-qxw4l 0/1 ContainerCreating 0 86s
zenko-cloudserver-66b4d45b4c-qzj7d 0/1 Terminating 0 85s
zenko-cloudserver-66b4d45b4c-t7259 0/1 ContainerCreating 0 86s
zenko-cloudserver-66b4d45b4c-vxkc9 0/1 ContainerCreating 0 85s
zenko-cloudserver-66b4d45b4c-zn6vk 0/1 Terminating 0 85s
zenko-cloudserver-manager-7dc886bf55-wj4mw 0/1 ContainerCreating 0 83s
zenko-cosmos-operator-5b86756ff7-tvrhb 0/1 ContainerCreating 0 86s
zenko-cosmos-scheduler-5c96f7ff9d-29zhp 0/1 ContainerCreating 0 86s
zenko-grafana-56f8f4f559-jtbnd 0/3 ContainerCreating 0 82s
zenko-mongodb-replicaset-0 0/2 Init:0/3 0 85s
zenko-prometheus-server-0 0/2 Init:0/1 0 79s
zenko-redis-ha-server-0 0/2 Init:0/1 0 83s
zenko-s3-data-78f46ccd8d-sg8zt 0/1 Init:0/1 0 85s
zenko-zenko-queue-0 0/2 ContainerCreating 0 80s
zenko-zenko-queue-config-ab364206-s2r5s 0/1 ContainerCreating 0 85s
zenko-zenko-queue-exporter-6b995f54fc-xl4jp 0/1 ContainerCreating 0 85s
zenko-zenko-queue-manager-5475c68d5c-6th4h 0/1 ContainerCreating 0 83s
zenko-zenko-queue-manager-bootstrap-0c3fbfe6-s2j25 0/1 ContainerCreating 0 85s
zenko-zenko-quorum-0 0/2 ContainerCreating 0 82s
==> v1/Role
NAME AGE
zenko-cosmos-operator 87s
zenko-cosmos-scheduler 87s
zenko-redis-ha 87s
==> v1/RoleBinding
NAME AGE
zenko-cosmos-operator 87s
zenko-cosmos-scheduler 87s
zenko-redis-ha 87s
==> v1/Secret
NAME TYPE DATA AGE
zenko-grafana Opaque 3 89s
zenko-zenko-queue-manager Opaque 3 89s
==> v1/Service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
zenko-backbeat-api ClusterIP 10.43.139.10 8900/TCP 87s
zenko-cloudserver ClusterIP 10.43.123.152 80/TCP 87s
zenko-grafana ClusterIP 10.43.145.174 80/TCP 87s
zenko-mongodb-replicaset ClusterIP None 27017/TCP 87s
zenko-mongodb-replicaset-client ClusterIP None 27017/TCP,9216/TCP 87s
zenko-prometheus-server ClusterIP 10.43.65.181 80/TCP 87s
zenko-prometheus-server-headless ClusterIP None 80/TCP 87s
zenko-redis-ha ClusterIP None 6379/TCP,26379/TCP 87s
zenko-redis-ha-announce-0 ClusterIP 10.43.3.137 6379/TCP,26379/TCP 87s
zenko-redis-ha-announce-1 ClusterIP 10.43.188.83 6379/TCP,26379/TCP 87s
zenko-redis-ha-announce-2 ClusterIP 10.43.232.95 6379/TCP,26379/TCP 87s
zenko-s3-data ClusterIP 10.43.189.105 9991/TCP 87s
zenko-zenko-queue ClusterIP 10.43.0.137 9092/TCP 87s
zenko-zenko-queue-headless ClusterIP None 9092/TCP 87s
zenko-zenko-queue-manager ClusterIP 10.43.99.245 9000/TCP 87s
zenko-zenko-quorum ClusterIP 10.43.17.141 2181/TCP 87s
zenko-zenko-quorum-headless ClusterIP None 2181/TCP,3888/TCP,2888/TCP 87s
==> v1/ServiceAccount
NAME SECRETS AGE
zenko 1 88s
zenko-cosmos-operator 1 88s
zenko-cosmos-scheduler 1 88s
zenko-grafana 1 88s
zenko-prometheus-alertmanager 1 88s
zenko-prometheus-kube-state-metrics 1 88s
zenko-prometheus-node-exporter 1 88s
zenko-prometheus-pushgateway 1 88s
zenko-prometheus-server 1 88s
zenko-redis-ha 1 88s
zenko-zenko-queue-manager 1 88s
==> v1/StatefulSet
NAME READY AGE
zenko-mongodb-replicaset 0/3 86s
zenko-redis-ha-server 0/3 86s
==> v1/StorageClass
NAME PROVISIONER AGE
zenko-cosmos-operator-remote-storage kubernetes.io/no-provisioner 88s
==> v1beta1/ClusterRole
NAME AGE
zenko-prometheus-kube-state-metrics 88s
zenko-prometheus-server 88s
==> v1beta1/ClusterRoleBinding
NAME AGE
zenko-prometheus-kube-state-metrics 88s
zenko-prometheus-server 88s
==> v1beta1/CronJob
NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE
zenko-zenko-reporting-count-items @hourly False 0 86s
==> v1beta1/CustomResourceDefinition
NAME AGE
cosmoses.zenko.io 88s
==> v1beta1/Deployment
NAME READY UP-TO-DATE AVAILABLE AGE
zenko-backbeat-api 0/1 1 0 87s
zenko-backbeat-gc-consumer 0/3 3 0 87s
zenko-backbeat-ingestion-consumer 0/3 3 0 87s
zenko-backbeat-ingestion-producer 0/1 1 0 87s
zenko-backbeat-lifecycle-bucket-processor 0/3 3 0 87s
zenko-backbeat-lifecycle-conductor 0/1 1 0 87s
zenko-backbeat-lifecycle-object-processor 0/3 3 0 87s
zenko-backbeat-replication-data-processor 0/3 3 0 87s
zenko-backbeat-replication-populator 0/1 1 0 87s
zenko-backbeat-replication-status-processor 0/3 3 0 87s
zenko-cloudserver 0/4 30 0 87s
zenko-cloudserver-manager 0/1 1 0 87s
zenko-s3-data 0/1 1 0 87s
zenko-zenko-queue-exporter 0/1 1 0 87s
==> v1beta1/Ingress
NAME HOSTS ADDRESS PORTS AGE
zenko-grafana s3.xxx.xxx.xxxx.it 10.xx.xx.29,... 80 86s
zenko-prometheus-server s3.xxx.xxx.xxxx.it 10.xx.xx.29,... 80 86s
zenko-zenko s3.xxx..xxxx.it 10.xx.xx.29,... 80 86s
==> v1beta1/PodDisruptionBudget
NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE
zenko-mongodb-replicaset N/A 1 0 89s
zenko-zenko-quorum N/A 1 0 89s
==> v1beta1/PodSecurityPolicy
NAME PRIV CAPS SELINUX RUNASUSER FSGROUP SUPGROUP READONLYROOTFS VOLUMES
zenko-grafana false RunAsAny RunAsAny RunAsAny RunAsAny false configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim
==> v1beta1/Role
NAME AGE
zenko 87s
zenko-grafana 87s
==> v1beta1/RoleBinding
NAME AGE
zenko 87s
zenko-grafana 87s
==> v1beta1/StatefulSet
NAME READY AGE
zenko-zenko-queue 0/3 86s
zenko-zenko-quorum 0/3 86s
==> v1beta2/Deployment
NAME READY UP-TO-DATE AVAILABLE AGE
zenko-grafana 0/1 1 0 87s
==> v1beta2/StatefulSet
NAME READY AGE
zenko-prometheus-server 0/2 86s
NOTES:
Access the Zenko endpoint from this URL:
http://s3.lxxxx.xxxxx.xxxxx.it