Comments (8)
You've cut off the stacktrace. In your previous post, you have java.rmi.ConnectException: Connection refused to host
.
You don't need to deploy Prometheus itself to get the JMX exporter working. Once it's working, you will have Service definitions that you can call /metrics endpoint over HTTP on the 5555 pod ports
from cp-helm-charts.
As I've written in other issues, these Helm Charts are no longer maintained.
If you want to setup monitoring on Confluent components with Prometheus and Grafana using CFK, that's available in this repo https://github.com/confluentinc/confluent-kubernetes-examples/tree/master/monitoring/grafana-dashboard#monitoring
from cp-helm-charts.
DNS resolution failed for kafka-cp-kafka-headless
Your image shows no healthy Kafka pods.
Therefore, everything will fail to start because Kafka is not running, as a dependency.
from cp-helm-charts.
[Update]
Thanks @OneCricketeer, in the values.yaml
file I fullfilled the bootstrapServers
of all artifacts as "http://confluentinc-cp-kafka:9092".
root@master:~/cp-helm-charts# kubectl get all -n kafka
NAME READY STATUS RESTARTS AGE
pod/confluentinc-cp-control-center-5cd976b498-7lkn6 1/1 Running 0 131m
pod/confluentinc-cp-zookeeper-0 2/2 Running 0 131m
pod/confluentinc-cp-ksql-server-5bb5d679d9-5jvjk 2/2 Running 0 131m
pod/confluentinc-cp-kafka-0 2/2 Running 0 131m
pod/confluentinc-cp-kafka-rest-58bd995bf4-f8kpf 2/2 Running 1 (129m ago) 131m
pod/confluentinc-cp-schema-registry-d458757c9-z229g 1/2 CrashLoopBackOff 28 (31s ago) 131m
pod/confluentinc-cp-kafka-connect-69f74b8665-jvnxh 1/2 Error 25 (6m25s ago) 131m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/confluentinc-cp-kafka-headless ClusterIP None <none> 9092/TCP 131m
service/confluentinc-cp-zookeeper-headless ClusterIP None <none> 2888/TCP,3888/TCP 131m
service/confluentinc-cp-zookeeper ClusterIP 10.152.183.188 <none> 2181/TCP,5556/TCP 131m
service/confluentinc-cp-control-center ClusterIP 10.152.183.32 <none> 9021/TCP 131m
service/confluentinc-cp-kafka-connect ClusterIP 10.152.183.167 <none> 8083/TCP,5556/TCP 131m
service/confluentinc-cp-kafka-rest ClusterIP 10.152.183.85 <none> 8082/TCP,5556/TCP 131m
service/confluentinc-cp-ksql-server ClusterIP 10.152.183.142 <none> 8088/TCP,5556/TCP 131m
service/confluentinc-cp-kafka ClusterIP 10.152.183.169 <none> 9092/TCP,5556/TCP 131m
service/confluentinc-cp-schema-registry ClusterIP 10.152.183.119 <none> 8081/TCP,5556/TCP 131m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/confluentinc-cp-control-center 1/1 1 1 131m
deployment.apps/confluentinc-cp-ksql-server 1/1 1 1 131m
deployment.apps/confluentinc-cp-kafka-rest 1/1 1 1 131m
deployment.apps/confluentinc-cp-schema-registry 0/1 1 0 131m
deployment.apps/confluentinc-cp-kafka-connect 0/1 1 0 131m
NAME DESIRED CURRENT READY AGE
replicaset.apps/confluentinc-cp-control-center-5cd976b498 1 1 1 131m
replicaset.apps/confluentinc-cp-ksql-server-5bb5d679d9 1 1 1 131m
replicaset.apps/confluentinc-cp-kafka-rest-58bd995bf4 1 1 1 131m
replicaset.apps/confluentinc-cp-schema-registry-d458757c9 1 1 0 131m
replicaset.apps/confluentinc-cp-kafka-connect-69f74b8665 1 1 0 131m
NAME READY AGE
statefulset.apps/confluentinc-cp-zookeeper 1/1 131m
statefulset.apps/confluentinc-cp-kafka 1/1 131m
Right now, these are my 2 issues:
For cp-schema-registry-server
:
(io.confluent.kafka.schemaregistry.rest.SchemaRegistryConfig)
[2023-01-12 17:54:57,122] INFO Logging initialized @1102ms to org.eclipse.jetty.util.log.Slf4jLog (org.eclipse.jetty.util.log)
[2023-01-12 17:54:57,132] INFO Initial capacity 128, increased by 64, maximum capacity 2147483647. (io.confluent.rest.ApplicationServer)
[2023-01-12 17:54:57,273] INFO Adding listener: http://0.0.0.0:8081 (io.confluent.rest.ApplicationServer)
[2023-01-12 17:54:57,600] WARN Ignoring Kafka broker endpoint http://confluentinc-cp-kafka:9092 that does not match the setting for kafkastore.security.protocol=PLAINTEXT (io.confluent.kafka.schemaregistry.rest.SchemaRegistryConfig)
[2023-01-12 17:54:57,601] ERROR Server died unexpectedly: (io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain)
org.apache.kafka.common.config.ConfigException: No supported Kafka endpoints are configured. Either kafkastore.bootstrap.servers must have at least one endpoint matching kafkastore.security.protocol or broker endpoints loaded from ZooKeeper must have at least one endpoint matching kafkastore.security.protocol.
at io.confluent.kafka.schemaregistry.rest.SchemaRegistryConfig.endpointsToBootstrapServers(SchemaRegistryConfig.java:716)
at io.confluent.kafka.schemaregistry.rest.SchemaRegistryConfig.bootstrapBrokers(SchemaRegistryConfig.java:651)
at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.kafkaClusterId(KafkaSchemaRegistry.java:1280)
at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.<init>(KafkaSchemaRegistry.java:158)
at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.initSchemaRegistry(SchemaRegistryRestApplication.java:69)
at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.configureBaseApplication(SchemaRegistryRestApplication.java:88)
at io.confluent.rest.Application.configureHandler(Application.java:255)
at io.confluent.rest.ApplicationServer.doStart(ApplicationServer.java:227)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
at io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain.main(SchemaRegistryMain.java:43)
For prometheus-jmx-exporter
:
root@master:~/cp-helm-charts# kubectl logs pod/confluentinc-cp-kafka-connect-69f74b8665-jvnxh prometheus-jmx-exporter -n kafka
Caused by: java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
java.net.ConnectException: Connection refused (Connection refused)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:338)
at sun.rmi.registry.RegistryImpl_Stub.lookup(RegistryImpl_Stub.java:112)
at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:132)
... 21 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at java.net.Socket.<init>(Socket.java:434)
at java.net.Socket.<init>(Socket.java:211)
at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:148)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
... 26 more
from cp-helm-charts.
You can ignore Prometheus since the Registry server isn't started.
Error here - WARN Ignoring Kafka broker endpoint http://
; Kafka isn't an HTTP service, so remove the protocol
from cp-helm-charts.
Thanks again @OneCricketeer.
After changing the broker endpoint to confluentinc-cp-kafka:9092
the cluster works properly. Only, Prometheus containers are restarted due to the issue you mention.
root@master:~/cp-helm-charts# kubectl get all -n kafka
NAME READY STATUS RESTARTS AGE
pod/confluentinc-cp-control-center-68b5f664d-wbg65 1/1 Running 0 6m10s
pod/confluentinc-cp-zookeeper-0 2/2 Running 0 6m10s
pod/confluentinc-cp-kafka-0 2/2 Running 0 6m10s
pod/confluentinc-cp-kafka-rest-7c5887d5fb-b7zzr 2/2 Running 0 6m10s
pod/confluentinc-cp-ksql-server-78487b764c-7mwt8 2/2 Running 1 6m10s
pod/confluentinc-cp-kafka-connect-5d68fc9947-xzzlr 2/2 Running 2 (79s ago) 6m10s
pod/confluentinc-cp-schema-registry-7747f7fd9c-wsl25 2/2 Running 2 (36s ago) 6m10s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/confluentinc-cp-kafka-headless ClusterIP None <none> 9092/TCP 6m10s
service/confluentinc-cp-zookeeper-headless ClusterIP None <none> 2888/TCP,3888/TCP 6m10s
service/confluentinc-cp-kafka-rest ClusterIP 10.152.183.169 <none> 8082/TCP,5556/TCP 6m10s
service/confluentinc-cp-zookeeper ClusterIP 10.152.183.39 <none> 2181/TCP,5556/TCP 6m10s
service/confluentinc-cp-kafka-connect ClusterIP 10.152.183.25 <none> 8083/TCP,5556/TCP 6m10s
service/confluentinc-cp-control-center ClusterIP 10.152.183.23 <none> 9021/TCP 6m10s
service/confluentinc-cp-kafka ClusterIP 10.152.183.28 <none> 9092/TCP,5556/TCP 6m10s
service/confluentinc-cp-ksql-server ClusterIP 10.152.183.91 <none> 8088/TCP,5556/TCP 6m10s
service/confluentinc-cp-schema-registry ClusterIP 10.152.183.247 <none> 8081/TCP,5556/TCP 6m10s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/confluentinc-cp-control-center 1/1 1 1 6m10s
deployment.apps/confluentinc-cp-kafka-rest 1/1 1 1 6m10s
deployment.apps/confluentinc-cp-ksql-server 1/1 1 1 6m10s
deployment.apps/confluentinc-cp-kafka-connect 1/1 1 1 6m10s
deployment.apps/confluentinc-cp-schema-registry 1/1 1 1 6m10s
NAME DESIRED CURRENT READY AGE
replicaset.apps/confluentinc-cp-control-center-68b5f664d 1 1 1 6m10s
replicaset.apps/confluentinc-cp-kafka-rest-7c5887d5fb 1 1 1 6m10s
replicaset.apps/confluentinc-cp-ksql-server-78487b764c 1 1 1 6m10s
replicaset.apps/confluentinc-cp-kafka-connect-5d68fc9947 1 1 1 6m10s
replicaset.apps/confluentinc-cp-schema-registry-7747f7fd9c 1 1 1 6m10s
NAME READY AGE
statefulset.apps/confluentinc-cp-zookeeper 1/1 6m10s
statefulset.apps/confluentinc-cp-kafka 1/1 6m10s
This is the configuration provided in the values.yml
file:
## ------------------------------------------------------
## Zookeeper
## ------------------------------------------------------
cp-zookeeper:
enabled: true
servers: 1
image: confluentinc/cp-zookeeper
imageTag: 6.1.0
## Optionally specify an array of imagePullSecrets. Secrets must be manually created in the namespace.
## https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
imagePullSecrets:
# - name: "regcred"
heapOptions: "-Xms512M -Xmx512M"
persistence:
enabled: false
## The size of the PersistentVolume to allocate to each Zookeeper Pod in the StatefulSet. For
## production servers this number should likely be much larger.
##
## Size for Data dir, where ZooKeeper will store the in-memory database snapshots.
dataDirSize: 10Gi
# dataDirStorageClass: ""
## Size for data log dir, which is a dedicated log device to be used, and helps avoid competition between logging and snaphots.
dataLogDirSize: 10Gi
# dataLogDirStorageClass: ""
# TODO: find correct security context for user in this zk-image
securityContext:
runAsUser: 0
resources: {}
## If you do want to specify resources, uncomment the following lines, adjust them as necessary,
## and remove the curly braces after 'resources:'
# limits:
# cpu: 100m
# memory: 128Mi
# requests:
# cpu: 100m
# memory: 128Mi
## ------------------------------------------------------
## Kafka
## ------------------------------------------------------
cp-kafka:
enabled: true
brokers: 1
image: confluentinc/cp-enterprise-kafka
imageTag: 6.1.0
## Optionally specify an array of imagePullSecrets. Secrets must be manually created in the namespace.
## https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
imagePullSecrets:
# - name: "regcred"
heapOptions: "-Xms512M -Xmx512M"
persistence:
enabled: false
#storageClass: ""
size: 5Gi
disksPerBroker: 1
resources: {}
## If you do want to specify resources, uncomment the following lines, adjust them as necessary,
## and remove the curly braces after 'resources:'
# limits:
# cpu: 100m
# memory: 128Mi
# requests:
# cpu: 100m
# memory: 128Mi
securityContext:
runAsUser: 0
## ------------------------------------------------------
## Schema Registry
## ------------------------------------------------------
cp-schema-registry:
enabled: true
image: confluentinc/cp-schema-registry
imageTag: 6.1.0
## Optionally specify an array of imagePullSecrets. Secrets must be manually created in the namespace.
## https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
imagePullSecrets:
# - name: "regcred"
heapOptions: "-Xms512M -Xmx512M"
resources: {}
## If you do want to specify resources, uncomment the following lines, adjust them as necessary,
## and remove the curly braces after 'resources:'
# limits:
# cpu: 100m
# memory: 128Mi
# requests:
# cpu: 100m
# memory: 128Mi
kafka:
bootstrapServers: "confluentinc-cp-kafka:9092"
## ------------------------------------------------------
## REST Proxy
## ------------------------------------------------------
cp-kafka-rest:
enabled: true
image: confluentinc/cp-kafka-rest
imageTag: 6.1.0
## Optionally specify an array of imagePullSecrets. Secrets must be manually created in the namespace.
## https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
imagePullSecrets:
# - name: "regcred"
heapOptions: "-Xms512M -Xmx512M"
resources: {}
## If you do want to specify resources, uncomment the following lines, adjust them as necessary,
## and remove the curly braces after 'resources:'
# limits:
# cpu: 100m
# memory: 128Mi
# requests:
# cpu: 100m
# memory: 1 8Mi
cp-kafka:
bootstrapServers: "confluentinc-cp-kafka:9092"
## ------------------------------------------------------
## Kafka Connect
## ------------------------------------------------------
cp-kafka-connect:
enabled: true
image: confluentinc/cp-kafka-connect
imageTag: 6.1.0
## Optionally specify an array of imagePullSecrets. Secrets must be manually created in the namespace.
## https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
imagePullSecrets:
# - name: "regcred"
heapOptions: "-Xms512M -Xmx512M"
resources: {}
## If you do want to specify resources, uncomment the following lines, adjust them as necessary,
## and remove the curly braces after 'resources:'
# limits:
# cpu: 100m
# memory: 128Mi
# requests:
# cpu: 100m
# memory: 128Mi
kafka:
bootstrapServers: "confluentinc-cp-kafka:9092"
## ------------------------------------------------------
## KSQL Server
## ------------------------------------------------------
cp-ksql-server:
enabled: true
image: confluentinc/cp-ksqldb-server
imageTag: 6.1.0
## Optionally specify an array of imagePullSecrets. Secrets must be manually created in the namespace.
## https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
imagePullSecrets:
# - name: "regcred"
heapOptions: "-Xms512M -Xmx512M"
ksql:
headless: false
kafka:
bootstrapServers: "confluentinc-cp-kafka:9092"
## ------------------------------------------------------
## Control Center
## ------------------------------------------------------
cp-control-center:
enabled: true
image: confluentinc/cp-enterprise-control-center
imageTag: 6.1.0
## Optionally specify an array of imagePullSecrets. Secrets must be manually created in the namespace.
## https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
imagePullSecrets:
# - name: "regcred"
heapOptions: "-Xms512M -Xmx512M"
resources: {}
## If you do want to specify resources, uncomment the following lines, adjust them as necessary,
## and remove the curly braces after 'resources:'
# limits:
# cpu: 100m
# memory: 128Mi
# requests:
# cpu: 100m
# memory: 128Mi
configurationOverrides:
"replication.factor": "1"
kafka:
bootstrapServers: "confluentinc-cp-kafka:9092"
cp-zookeeper:
## If the Zookeeper Chart is disabled a URL and port are required to connect
url: "http://confluentinc-cp-zookeeper:2181"
Could you please recommend me a way to enable Prometheus to activate monitoring?
from cp-helm-charts.
enable Prometheus to activate monitoring?
There's a separate project called prom-operator or kube-prometheus from CoreOS that defines ServiceMonitor resource types that you'd want
from cp-helm-charts.
Thanks, @OneCricketeer.
In the same values.yaml
I have declared Prometheus like this for each entity:
## Monitoring
## Kafka JMX Settings
## ref: https://docs.confluent.io/current/kafka/monitoring.html
jmx:
port: 5555
## Prometheus Exporter Configuration
## ref: https://prometheus.io/docs/instrumenting/exporters/
prometheus:
## JMX Exporter Configuration
## ref: https://github.com/prometheus/jmx_exporter
jmx:
enabled: true
image: solsson/kafka-prometheus-jmx-exporter@sha256
imageTag: 6f82e2b0464f50da8104acd7363fb9b995001ddff77d248379f8788e78946143
port: 5556
## Resources configuration for the JMX exporter container.
## See the `resources` documentation above for details.
resources: {}
However, the connection refuse error persist for the prometheus agents:
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at java.net.Socket.<init>(Socket.java:434)
at java.net.Socket.<init>(Socket.java:211)
at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:148)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
... 26 more
To whom are these agents trying to connect? This entity is not enabled by default when deploying the whole cluster.
from cp-helm-charts.
Related Issues (20)
- cp-helm-charts for production HOT 3
- Kafka broker to Kafka Client encryption using TLS and authentcation HOT 1
- mirror maker for multicluster replication HOT 1
- k8s-sts-pod-env invalid
- Chart does not work in Kubernetes 1.25. `PodDisruptionBudget` `v1beta` is _removed_ HOT 15
- no option for schedulerName
- Enquiry .... HOT 1
- zookeeper is not scraping metrics
- Is the values.heapOptions in cp-ksql-server doing anything? HOT 4
- Need clarity on licensing HOT 1
- Clarity on Externally exposing connect and kafka broker HOT 3
- Kafka Kraft Mode Helm Charts HOT 1
- jmx exporter HOT 1
- Helm chart deployment is failing when my cluster-domain is not cluster.local HOT 1
- Rack Awareness for AWS multiple availability zones On EKS Cluster HOT 4
- when will there be an update ? HOT 1
- prometheus jmx-exporter fails with UseCGroupMemoryLimitForHeap error
- Helm chart not working with kubernetes 1.25+ HOT 3
- Issue with Kafka Pod Communication Across Clusters in Bitnami HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cp-helm-charts.