Giter Club home page Giter Club logo

Comments (8)

OneCricketeer avatar OneCricketeer commented on July 22, 2024 1

You've cut off the stacktrace. In your previous post, you have java.rmi.ConnectException: Connection refused to host.

You don't need to deploy Prometheus itself to get the JMX exporter working. Once it's working, you will have Service definitions that you can call /metrics endpoint over HTTP on the 5555 pod ports

from cp-helm-charts.

OneCricketeer avatar OneCricketeer commented on July 22, 2024 1

As I've written in other issues, these Helm Charts are no longer maintained.

If you want to setup monitoring on Confluent components with Prometheus and Grafana using CFK, that's available in this repo https://github.com/confluentinc/confluent-kubernetes-examples/tree/master/monitoring/grafana-dashboard#monitoring

from cp-helm-charts.

OneCricketeer avatar OneCricketeer commented on July 22, 2024

DNS resolution failed for kafka-cp-kafka-headless

Your image shows no healthy Kafka pods.

Therefore, everything will fail to start because Kafka is not running, as a dependency.

from cp-helm-charts.

sfl0r3nz05 avatar sfl0r3nz05 commented on July 22, 2024

[Update]

Thanks @OneCricketeer, in the values.yaml file I fullfilled the bootstrapServers of all artifacts as "http://confluentinc-cp-kafka:9092".

root@master:~/cp-helm-charts# kubectl get all -n kafka
NAME                                                  READY   STATUS             RESTARTS         AGE
pod/confluentinc-cp-control-center-5cd976b498-7lkn6   1/1     Running            0                131m
pod/confluentinc-cp-zookeeper-0                       2/2     Running            0                131m
pod/confluentinc-cp-ksql-server-5bb5d679d9-5jvjk      2/2     Running            0                131m
pod/confluentinc-cp-kafka-0                           2/2     Running            0                131m
pod/confluentinc-cp-kafka-rest-58bd995bf4-f8kpf       2/2     Running            1 (129m ago)     131m
pod/confluentinc-cp-schema-registry-d458757c9-z229g   1/2     CrashLoopBackOff   28 (31s ago)     131m
pod/confluentinc-cp-kafka-connect-69f74b8665-jvnxh    1/2     Error              25 (6m25s ago)   131m

NAME                                         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
service/confluentinc-cp-kafka-headless       ClusterIP   None             <none>        9092/TCP            131m
service/confluentinc-cp-zookeeper-headless   ClusterIP   None             <none>        2888/TCP,3888/TCP   131m
service/confluentinc-cp-zookeeper            ClusterIP   10.152.183.188   <none>        2181/TCP,5556/TCP   131m
service/confluentinc-cp-control-center       ClusterIP   10.152.183.32    <none>        9021/TCP            131m
service/confluentinc-cp-kafka-connect        ClusterIP   10.152.183.167   <none>        8083/TCP,5556/TCP   131m
service/confluentinc-cp-kafka-rest           ClusterIP   10.152.183.85    <none>        8082/TCP,5556/TCP   131m
service/confluentinc-cp-ksql-server          ClusterIP   10.152.183.142   <none>        8088/TCP,5556/TCP   131m
service/confluentinc-cp-kafka                ClusterIP   10.152.183.169   <none>        9092/TCP,5556/TCP   131m
service/confluentinc-cp-schema-registry      ClusterIP   10.152.183.119   <none>        8081/TCP,5556/TCP   131m

NAME                                              READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/confluentinc-cp-control-center    1/1     1            1           131m
deployment.apps/confluentinc-cp-ksql-server       1/1     1            1           131m
deployment.apps/confluentinc-cp-kafka-rest        1/1     1            1           131m
deployment.apps/confluentinc-cp-schema-registry   0/1     1            0           131m
deployment.apps/confluentinc-cp-kafka-connect     0/1     1            0           131m

NAME                                                        DESIRED   CURRENT   READY   AGE
replicaset.apps/confluentinc-cp-control-center-5cd976b498   1         1         1       131m
replicaset.apps/confluentinc-cp-ksql-server-5bb5d679d9      1         1         1       131m
replicaset.apps/confluentinc-cp-kafka-rest-58bd995bf4       1         1         1       131m
replicaset.apps/confluentinc-cp-schema-registry-d458757c9   1         1         0       131m
replicaset.apps/confluentinc-cp-kafka-connect-69f74b8665    1         1         0       131m

NAME                                         READY   AGE
statefulset.apps/confluentinc-cp-zookeeper   1/1     131m
statefulset.apps/confluentinc-cp-kafka       1/1     131m

Right now, these are my 2 issues:

For cp-schema-registry-server:

(io.confluent.kafka.schemaregistry.rest.SchemaRegistryConfig)
[2023-01-12 17:54:57,122] INFO Logging initialized @1102ms to org.eclipse.jetty.util.log.Slf4jLog (org.eclipse.jetty.util.log)
[2023-01-12 17:54:57,132] INFO Initial capacity 128, increased by 64, maximum capacity 2147483647. (io.confluent.rest.ApplicationServer)
[2023-01-12 17:54:57,273] INFO Adding listener: http://0.0.0.0:8081 (io.confluent.rest.ApplicationServer)
[2023-01-12 17:54:57,600] WARN Ignoring Kafka broker endpoint http://confluentinc-cp-kafka:9092 that does not match the setting for kafkastore.security.protocol=PLAINTEXT (io.confluent.kafka.schemaregistry.rest.SchemaRegistryConfig)
[2023-01-12 17:54:57,601] ERROR Server died unexpectedly:  (io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain)
org.apache.kafka.common.config.ConfigException: No supported Kafka endpoints are configured. Either kafkastore.bootstrap.servers must have at least one endpoint matching kafkastore.security.protocol or broker endpoints loaded from ZooKeeper must have at least one endpoint matching kafkastore.security.protocol.
        at io.confluent.kafka.schemaregistry.rest.SchemaRegistryConfig.endpointsToBootstrapServers(SchemaRegistryConfig.java:716)
        at io.confluent.kafka.schemaregistry.rest.SchemaRegistryConfig.bootstrapBrokers(SchemaRegistryConfig.java:651)
        at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.kafkaClusterId(KafkaSchemaRegistry.java:1280)
        at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.<init>(KafkaSchemaRegistry.java:158)
        at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.initSchemaRegistry(SchemaRegistryRestApplication.java:69)
        at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.configureBaseApplication(SchemaRegistryRestApplication.java:88)
        at io.confluent.rest.Application.configureHandler(Application.java:255)
        at io.confluent.rest.ApplicationServer.doStart(ApplicationServer.java:227)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
        at io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain.main(SchemaRegistryMain.java:43)

For prometheus-jmx-exporter:

root@master:~/cp-helm-charts# kubectl logs pod/confluentinc-cp-kafka-connect-69f74b8665-jvnxh prometheus-jmx-exporter  -n kafka

Caused by: java.rmi.ConnectException: Connection refused to host: localhost; nested exception is: 
        java.net.ConnectException: Connection refused (Connection refused)
        at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
        at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
        at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
        at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:338)
        at sun.rmi.registry.RegistryImpl_Stub.lookup(RegistryImpl_Stub.java:112)
        at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:132)
        ... 21 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:589)
        at java.net.Socket.connect(Socket.java:538)
        at java.net.Socket.<init>(Socket.java:434)
        at java.net.Socket.<init>(Socket.java:211)
        at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
        at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:148)
        at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
        ... 26 more

from cp-helm-charts.

OneCricketeer avatar OneCricketeer commented on July 22, 2024

You can ignore Prometheus since the Registry server isn't started.

Error here - WARN Ignoring Kafka broker endpoint http:// ; Kafka isn't an HTTP service, so remove the protocol

from cp-helm-charts.

sfl0r3nz05 avatar sfl0r3nz05 commented on July 22, 2024

Thanks again @OneCricketeer.

After changing the broker endpoint to confluentinc-cp-kafka:9092 the cluster works properly. Only, Prometheus containers are restarted due to the issue you mention.

root@master:~/cp-helm-charts# kubectl get all -n kafka
NAME                                                   READY   STATUS    RESTARTS      AGE
pod/confluentinc-cp-control-center-68b5f664d-wbg65     1/1     Running   0             6m10s
pod/confluentinc-cp-zookeeper-0                        2/2     Running   0             6m10s
pod/confluentinc-cp-kafka-0                            2/2     Running   0             6m10s
pod/confluentinc-cp-kafka-rest-7c5887d5fb-b7zzr        2/2     Running   0             6m10s
pod/confluentinc-cp-ksql-server-78487b764c-7mwt8       2/2     Running   1             6m10s
pod/confluentinc-cp-kafka-connect-5d68fc9947-xzzlr     2/2     Running   2 (79s ago)   6m10s
pod/confluentinc-cp-schema-registry-7747f7fd9c-wsl25   2/2     Running   2 (36s ago)   6m10s

NAME                                         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
service/confluentinc-cp-kafka-headless       ClusterIP   None             <none>        9092/TCP            6m10s
service/confluentinc-cp-zookeeper-headless   ClusterIP   None             <none>        2888/TCP,3888/TCP   6m10s
service/confluentinc-cp-kafka-rest           ClusterIP   10.152.183.169   <none>        8082/TCP,5556/TCP   6m10s
service/confluentinc-cp-zookeeper            ClusterIP   10.152.183.39    <none>        2181/TCP,5556/TCP   6m10s
service/confluentinc-cp-kafka-connect        ClusterIP   10.152.183.25    <none>        8083/TCP,5556/TCP   6m10s
service/confluentinc-cp-control-center       ClusterIP   10.152.183.23    <none>        9021/TCP            6m10s
service/confluentinc-cp-kafka                ClusterIP   10.152.183.28    <none>        9092/TCP,5556/TCP   6m10s
service/confluentinc-cp-ksql-server          ClusterIP   10.152.183.91    <none>        8088/TCP,5556/TCP   6m10s
service/confluentinc-cp-schema-registry      ClusterIP   10.152.183.247   <none>        8081/TCP,5556/TCP   6m10s

NAME                                              READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/confluentinc-cp-control-center    1/1     1            1           6m10s
deployment.apps/confluentinc-cp-kafka-rest        1/1     1            1           6m10s
deployment.apps/confluentinc-cp-ksql-server       1/1     1            1           6m10s
deployment.apps/confluentinc-cp-kafka-connect     1/1     1            1           6m10s
deployment.apps/confluentinc-cp-schema-registry   1/1     1            1           6m10s

NAME                                                         DESIRED   CURRENT   READY   AGE
replicaset.apps/confluentinc-cp-control-center-68b5f664d     1         1         1       6m10s
replicaset.apps/confluentinc-cp-kafka-rest-7c5887d5fb        1         1         1       6m10s
replicaset.apps/confluentinc-cp-ksql-server-78487b764c       1         1         1       6m10s
replicaset.apps/confluentinc-cp-kafka-connect-5d68fc9947     1         1         1       6m10s
replicaset.apps/confluentinc-cp-schema-registry-7747f7fd9c   1         1         1       6m10s

NAME                                         READY   AGE
statefulset.apps/confluentinc-cp-zookeeper   1/1     6m10s
statefulset.apps/confluentinc-cp-kafka       1/1     6m10s

This is the configuration provided in the values.yml file:

## ------------------------------------------------------
## Zookeeper
## ------------------------------------------------------
cp-zookeeper:
  enabled: true
  servers: 1
  image: confluentinc/cp-zookeeper
  imageTag: 6.1.0
  ## Optionally specify an array of imagePullSecrets. Secrets must be manually created in the namespace.
  ## https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
  imagePullSecrets:
  #  - name: "regcred"
  heapOptions: "-Xms512M -Xmx512M"
  persistence:
    enabled: false
    ## The size of the PersistentVolume to allocate to each Zookeeper Pod in the StatefulSet. For
    ## production servers this number should likely be much larger.
    ##
    ## Size for Data dir, where ZooKeeper will store the in-memory database snapshots.
    dataDirSize: 10Gi
    # dataDirStorageClass: ""
    ## Size for data log dir, which is a dedicated log device to be used, and helps avoid competition between logging and snaphots.
    dataLogDirSize: 10Gi
    # dataLogDirStorageClass: ""
    
  # TODO: find correct security context for user in this zk-image  
  securityContext: 
    runAsUser: 0

  resources: {}
  ## If you do want to specify resources, uncomment the following lines, adjust them as necessary,
  ## and remove the curly braces after 'resources:'
  #  limits:
  #   cpu: 100m
  #   memory: 128Mi
  #  requests:
  #   cpu: 100m
  #   memory: 128Mi

## ------------------------------------------------------
## Kafka
## ------------------------------------------------------
cp-kafka:
  enabled: true
  brokers: 1
  image: confluentinc/cp-enterprise-kafka
  imageTag: 6.1.0
  ## Optionally specify an array of imagePullSecrets. Secrets must be manually created in the namespace.
  ## https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
  imagePullSecrets:
  #  - name: "regcred"
  heapOptions: "-Xms512M -Xmx512M"
  persistence:
    enabled: false
    #storageClass: ""
    size: 5Gi
    disksPerBroker: 1
  resources: {}
  ## If you do want to specify resources, uncomment the following lines, adjust them as necessary,
  ## and remove the curly braces after 'resources:'
  #  limits:
  #   cpu: 100m
  #   memory: 128Mi
  #  requests:
  #   cpu: 100m
  #   memory: 128Mi
  securityContext: 
    runAsUser: 0

## ------------------------------------------------------
## Schema Registry
## ------------------------------------------------------
cp-schema-registry:
  enabled: true
  image: confluentinc/cp-schema-registry
  imageTag: 6.1.0
  ## Optionally specify an array of imagePullSecrets. Secrets must be manually created in the namespace.
  ## https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
  imagePullSecrets:
  #  - name: "regcred"
  heapOptions: "-Xms512M -Xmx512M"
  resources: {}
  ## If you do want to specify resources, uncomment the following lines, adjust them as necessary,
  ## and remove the curly braces after 'resources:'
  #  limits:
  #   cpu: 100m
  #   memory: 128Mi
                 #  requests:
  #   cpu: 100m
  #   memory: 128Mi
  kafka:
    bootstrapServers: "confluentinc-cp-kafka:9092"

## ------------------------------------------------------
## REST Proxy
## ------------------------------------------------------
cp-kafka-rest:
  enabled: true
  image: confluentinc/cp-kafka-rest
  imageTag: 6.1.0
  ## Optionally specify an array of imagePullSecrets. Secrets must be manually created in the namespace.
  ## https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
  imagePullSecrets:
  #  - name: "regcred"
  heapOptions: "-Xms512M -Xmx512M"
  resources: {}
  ## If you do want to specify resources, uncomment the following lines, adjust them as necessary,
  ## and remove the curly braces after 'resources:'
  #  limits:
  #   cpu: 100m
  #   memory: 128Mi
  #  requests:
  #   cpu: 100m
  #   memory: 1 8Mi 
  cp-kafka:
    bootstrapServers: "confluentinc-cp-kafka:9092"

## ------------------------------------------------------
## Kafka Connect
## ------------------------------------------------------
cp-kafka-connect:
  enabled: true
  image: confluentinc/cp-kafka-connect
  imageTag: 6.1.0
  ## Optionally specify an array of imagePullSecrets. Secrets must be manually created in the namespace.
  ## https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
  imagePullSecrets:
  #  - name: "regcred"
  heapOptions: "-Xms512M -Xmx512M"
  resources: {}
  ## If you do want to specify resources, uncomment the following lines, adjust them as necessary,
  ## and remove the curly braces after 'resources:'
  #  limits:
  #   cpu: 100m
  #   memory: 128Mi
  #  requests:
  #   cpu: 100m
  #   memory: 128Mi
  kafka:
    bootstrapServers: "confluentinc-cp-kafka:9092"

## ------------------------------------------------------
## KSQL Server
## ------------------------------------------------------
cp-ksql-server:
  enabled: true
  image: confluentinc/cp-ksqldb-server
  imageTag: 6.1.0
  ## Optionally specify an array of imagePullSecrets. Secrets must be manually created in the namespace.
  ## https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
  imagePullSecrets:
  #  - name: "regcred"
  heapOptions: "-Xms512M -Xmx512M"
  ksql:
    headless: false
  kafka:
    bootstrapServers: "confluentinc-cp-kafka:9092"

## ------------------------------------------------------
## Control Center
## ------------------------------------------------------
cp-control-center:
  enabled: true
  image: confluentinc/cp-enterprise-control-center
  imageTag: 6.1.0
  ## Optionally specify an array of imagePullSecrets. Secrets must be manually created in the namespace.
  ## https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
  imagePullSecrets:
  #  - name: "regcred"
  heapOptions: "-Xms512M -Xmx512M"
  resources: {}
  ## If you do want to specify resources, uncomment the following lines, adjust them as necessary,
  ## and remove the curly braces after 'resources:'
  #  limits:
  #   cpu: 100m
  #   memory: 128Mi
  #  requests:
  #   cpu: 100m
  #   memory: 128Mi
  configurationOverrides:
    "replication.factor": "1"
  kafka:
    bootstrapServers: "confluentinc-cp-kafka:9092"
  cp-zookeeper:
  ## If the Zookeeper Chart is disabled a URL and port are required to connect
    url: "http://confluentinc-cp-zookeeper:2181"

Could you please recommend me a way to enable Prometheus to activate monitoring?

from cp-helm-charts.

OneCricketeer avatar OneCricketeer commented on July 22, 2024

enable Prometheus to activate monitoring?

There's a separate project called prom-operator or kube-prometheus from CoreOS that defines ServiceMonitor resource types that you'd want

from cp-helm-charts.

sfl0r3nz05 avatar sfl0r3nz05 commented on July 22, 2024

Thanks, @OneCricketeer.

In the same values.yaml I have declared Prometheus like this for each entity:

  ## Monitoring
  ## Kafka JMX Settings
  ## ref: https://docs.confluent.io/current/kafka/monitoring.html
  jmx:
    port: 5555

  ## Prometheus Exporter Configuration
  ## ref: https://prometheus.io/docs/instrumenting/exporters/
  prometheus:
    ## JMX Exporter Configuration
    ## ref: https://github.com/prometheus/jmx_exporter
    jmx:
      enabled: true
      image: solsson/kafka-prometheus-jmx-exporter@sha256
      imageTag: 6f82e2b0464f50da8104acd7363fb9b995001ddff77d248379f8788e78946143
      port: 5556

      ## Resources configuration for the JMX exporter container.
      ## See the `resources` documentation above for details.
      resources: {}

However, the connection refuse error persist for the prometheus agents:

Caused by: java.net.ConnectException: Connection refused (Connection refused)
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:589)
        at java.net.Socket.connect(Socket.java:538)
        at java.net.Socket.<init>(Socket.java:434)
        at java.net.Socket.<init>(Socket.java:211)
        at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
        at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:148)
        at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
        ... 26 more

To whom are these agents trying to connect? This entity is not enabled by default when deploying the whole cluster.

from cp-helm-charts.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.