Comments (13)
burmanm
commented on May 25, 2024
I don't think the ingress proxy should have any effect to the Reaper<->CassandraDatacenter connection. Would it be possible to get more info from your environment? Such as status for the CassandraDatacenter (to see that Reaper annotations are correct and settings also):
kubectl describe cassdc dc1
logs from the reaper operator to see that there's no anomalities (replace releaseName with the helm install's releaseName you set)
kubectl logs -l name=releaseName-reaper-operator-k8ssandra
Logs from the reaper itself:
kubectl logs -l app.kubernetes.io/managed-by=reaper-operator
And describe for the reaper:
kubectl describe pod -l app.kubernetes.io/managed-by=reaper-operator
from k8ssandra.
shallada
commented on May 25, 2024
Does this help?
$ kubectl describe cassdc dc1
Name: dc1
Namespace: default
Labels: app.kubernetes.io/instance=k8ssandra-cluster-a
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=k8ssandra-cluster
app.kubernetes.io/version=3.11.7
helm.sh/chart=k8ssandra-cluster-0.14.0
Annotations: meta.helm.sh/release-name: k8ssandra-cluster-a
meta.helm.sh/release-namespace: default
reaper.cassandra-reaper.io/instance: k8ssandra-cluster-a-reaper-k8ssandra
API Version: cassandra.datastax.com/v1beta1
Kind: CassandraDatacenter
Metadata:
Creation Timestamp: 2020-12-08T19:46:33Z
Finalizers:
finalizer.cassandra.datastax.com
Generation: 2
Resource Version: 2471
Self Link: /apis/cassandra.datastax.com/v1beta1/namespaces/default/cassandradatacenters/dc1
UID: 6c2f448f-fd8d-4d0c-a172-9a1a444e648e
Spec:
Cluster Name: k8ssandra
Config:
Jvm - Options:
initial_heap_size: 800M
max_heap_size: 800M
Config Builder Resources:
Management API Auth:
Insecure:
Pod Template Spec:
Metadata:
Creation Timestamp: <nil>
Spec:
Containers:
Env:
Name: LOCAL_JMX
Value: no
Name: cassandra
Resources:
Init Containers:
Args:
/bin/sh
-c
echo -n "$JMX_USERNAME $JMX_PASSWORD" > /config/jmxremote.password
Env:
Name: JMX_USERNAME
Value From:
Secret Key Ref:
Key: username
Name: k8ssandra-cluster-a-reaper-secret-k8ssandra
Name: JMX_PASSWORD
Value From:
Secret Key Ref:
Key: password
Name: k8ssandra-cluster-a-reaper-secret-k8ssandra
Image: busybox
Image Pull Policy: IfNotPresent
Name: jmx-credentials
Resources:
Volume Mounts:
Mount Path: /config
Name: server-config
Resources:
Server Image: jsanda/mgmtapi-3_11:v0.1.13-k8c-88
Server Type: cassandra
Server Version: 3.11.7
Size: 1
Storage Config:
Cassandra Data Volume Claim Spec:
Access Modes:
ReadWriteOnce
Resources:
Requests:
Storage: 5Gi
Storage Class Name: standard
System Logger Resources:
Status:
Cassandra Operator Progress: Ready
Conditions:
Last Transition Time: 2020-12-08T19:48:30Z
Message:
Reason:
Status: False
Type: ScalingUp
Last Transition Time: 2020-12-08T19:48:30Z
Message:
Reason:
Status: False
Type: Updating
Last Transition Time: 2020-12-08T19:48:30Z
Message:
Reason:
Status: False
Type: Stopped
Last Transition Time: 2020-12-08T19:48:30Z
Message:
Reason:
Status: False
Type: ReplacingNodes
Last Transition Time: 2020-12-08T19:48:30Z
Message:
Reason:
Status: False
Type: RollingRestart
Last Transition Time: 2020-12-08T19:48:30Z
Message:
Reason:
Status: False
Type: Resuming
Last Transition Time: 2020-12-08T19:48:30Z
Message:
Reason:
Status: False
Type: ScalingDown
Last Transition Time: 2020-12-08T19:48:30Z
Message:
Reason:
Status: True
Type: Valid
Last Transition Time: 2020-12-08T19:48:48Z
Message:
Reason:
Status: True
Type: Initialized
Last Transition Time: 2020-12-08T19:48:48Z
Message:
Reason:
Status: True
Type: Ready
Last Server Node Started: 2020-12-08T19:47:43Z
Node Statuses:
k8ssandra-dc1-default-sts-0:
Host ID: 6477a07d-b26c-4e51-8e3f-5812f0dcbeb6
Observed Generation: 2
Quiet Period: 2020-12-08T19:48:54Z
Super User Upserted: 2020-12-08T19:48:49Z
Users Upserted: 2020-12-08T19:48:49Z
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CreatedResource 35m cass-operator Created service k8ssandra-dc1-service
Normal CreatedResource 35m cass-operator Created service k8ssandra-seed-service
Normal CreatedResource 35m cass-operator Created service k8ssandra-dc1-all-pods-service
Normal CreatedResource 35m cass-operator Created statefulset k8ssandra-dc1-default-sts
Normal ScalingUpRack 35m cass-operator Scaling up rack default
Normal LabeledPodAsSeed 34m cass-operator Labeled pod a seed node k8ssandra-dc1-default-sts-0
Normal StartingCassandra 34m cass-operator Starting Cassandra for pod k8ssandra-dc1-default-sts-0
Normal StartedCassandra 33m cass-operator Started Cassandra for pod k8ssandra-dc1-default-sts-0
Normal CreatedResource 33m cass-operator Created PodDisruptionBudget dc1-pdb
Normal UpdatingRack 33m cass-operator Updating rack default
Normal CreatedUsers 33m (x3 over 33m) cass-operator Created users
Normal CreatedSuperuser 33m (x3 over 33m) cass-operator Created superuser
*********************************************************
$ kubectl logs -l app.kubernetes.io/managed-by=reaper-operator
POST /tasks/log-level (io.dropwizard.servlets.tasks.LogConfigurationTask)
POST /tasks/gc (io.dropwizard.servlets.tasks.GarbageCollectionTask)
INFO [2020-12-08 19:50:04,486] [main] o.e.j.s.h.ContextHandler - Started i.d.j.MutableServletContextHandler@d84418a{/,null,AVAILABLE}
INFO [2020-12-08 19:50:04,506] [main] o.e.j.s.AbstractConnector - Started application@2dd1086{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
INFO [2020-12-08 19:50:04,519] [main] o.e.j.s.AbstractConnector - Started admin@6b8d54da{HTTP/1.1,[http/1.1]}{0.0.0.0:8081}
INFO [2020-12-08 19:50:04,520] [main] o.e.j.s.Server - Started @30337ms
INFO [2020-12-08 19:50:41,854] [dw-29 - PUT /cluster/k8ssandra?seedHost=k8ssandra-dc1-service] i.c.r.ClusterResource - PUT addOrUpdateCluster called with: cluster_name = k8ssandra, seedHost = k8ssandra-dc1-service
INFO [2020-12-08 19:50:42,578] [dw-29 - PUT /cluster/k8ssandra?seedHost=k8ssandra-dc1-service] i.c.r.ClusterResource - creating new cluster based on given seed host: k8ssandra
*********************************************************
$ kubectl describe pod -l app.kubernetes.io/managed-by=reaper-operator
Name: k8ssandra-cluster-a-reaper-k8ssandra-847c99ccd8-8428z
Namespace: default
Priority: 0
Node: kind-worker2/172.18.0.4
Start Time: Tue, 08 Dec 2020 19:49:22 +0000
Labels: app.kubernetes.io/managed-by=reaper-operator
pod-template-hash=847c99ccd8
reaper.cassandra-reaper.io/reaper=k8ssandra-cluster-a-reaper-k8ssandra
Annotations: <none>
Status: Running
IP: 10.244.3.12
IPs:
IP: 10.244.3.12
Controlled By: ReplicaSet/k8ssandra-cluster-a-reaper-k8ssandra-847c99ccd8
Containers:
reaper:
Container ID: containerd://98b186bd55b531b3130ad8b362ec6b3be02e88d36fd918d58d8d1784604d7fc8
Image: thelastpickle/cassandra-reaper:2.0.5
Image ID: docker.io/thelastpickle/cassandra-reaper@sha256:dfec9a0daba645dcaf6480340aafb3ca588fab32fef20f2539089a42d26506ab
Ports: 8080/TCP, 8081/TCP
Host Ports: 0/TCP, 0/TCP
State: Running
Started: Tue, 08 Dec 2020 19:49:34 +0000
Ready: True
Restart Count: 0
Liveness: http-get http://:8081/healthcheck delay=45s timeout=1s period=15s #success=1 #failure=3
Readiness: http-get http://:8081/healthcheck delay=45s timeout=1s period=15s #success=1 #failure=3
Environment:
REAPER_STORAGE_TYPE: cassandra
REAPER_ENABLE_DYNAMIC_SEED_LIST: false
REAPER_CASS_CONTACT_POINTS: [k8ssandra-dc1-service]
REAPER_AUTH_ENABLED: false
REAPER_JMX_AUTH_USERNAME: <set to the key 'username' in secret 'k8ssandra-cluster-a-reaper-secret-k8ssandra'> Optional: false
REAPER_JMX_AUTH_PASSWORD: <set to the key 'password' in secret 'k8ssandra-cluster-a-reaper-secret-k8ssandra'> Optional: false
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-tjbrw (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
default-token-tjbrw:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-tjbrw
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 34m default-scheduler Successfully assigned default/k8ssandra-cluster-a-reaper-k8ssandra-847c99ccd8-8428z to kind-worker2
Normal Pulling 34m kubelet Pulling image "thelastpickle/cassandra-reaper:2.0.5"
Normal Pulled 34m kubelet Successfully pulled image "thelastpickle/cassandra-reaper:2.0.5"
Normal Created 34m kubelet Created container reaper
Normal Started 34m kubelet Started container reaper
from k8ssandra.
burmanm
commented on May 25, 2024
So it looks like it has found the cluster and added it. Just not showing up in the UI for some reason..
from k8ssandra.
jsanda
commented on May 25, 2024
Just not showing up in the UI for some reason..
I suspect the cluster is never actually getting added. @shallada can you run the following command so we can see whether the cluster gets added:
$ kubectl exec -it <cassandra-pod> -c cassandra -- cqlsh -e "select * from reaper_db.cluster"
And the can we see some filtered logs from Reaper:
$ kubectl logs <reaper-pod> | grep ClusterResource
@burmanm I looked into a bit before the ticket was created. I looked at @shallada's reaper-operator logs. I could see that cassandradatacenter_controller.go started but I did not see it processing any reconciliation requests.
from k8ssandra.
shallada
commented on May 25, 2024
$ kubectl exec -it k8ssandra-dc1-default-sts-0 -c cassandra -- cqlsh -e "select * from reaper_db.cluster"
name | last_contact | partitioner | properties | seed_hosts| state
-----------+---------------------------------+---------------------------------------------+------------------+---------------------------+--------
k8ssandra | 2020-12-09 00:00:00.000000+0000 | org.apache.cassandra.dht.Murmur3Partitioner | {"jmxPort":7199} | {'k8ssandra-dc1-service'}| ACTIVE
*************************************************
$ kubectl logs k8ssandra-cluster-a-reaper-k8ssandra-847c99ccd8-lqjzj | grep ClusterResource
GET /cluster (io.cassandrareaper.resources.ClusterResource)
POST /cluster (io.cassandrareaper.resources.ClusterResource)
DELETE /cluster/{cluster_name} (io.cassandrareaper.resources.ClusterResource)
GET /cluster/{cluster_name} (io.cassandrareaper.resources.ClusterResource)
PUT /cluster/{cluster_name} (io.cassandrareaper.resources.ClusterResource)
GET /cluster/{cluster_name}/tables (io.cassandrareaper.resources.ClusterResource)
INFO [2020-12-09 02:43:49,417] [dw-26 - PUT /cluster/k8ssandra?seedHost=k8ssandra-dc1-service] i.c.r.ClusterResource - PUT addOrUpdateCluster called with: cluster_name = k8ssandra, seedHost = k8ssandra-dc1-service
INFO [2020-12-09 02:43:49,854] [dw-26 - PUT /cluster/k8ssandra?seedHost=k8ssandra-dc1-service] i.c.r.ClusterResource - creating new cluster based on given seed host: k8ssandra```
from k8ssandra.
jsanda
commented on May 25, 2024
@shallada do you see the cluster in the Reaper UI?
from k8ssandra.
shallada
commented on May 25, 2024
from k8ssandra.
shallada
commented on May 25, 2024
When I clicked "Add Cluster", I saw this
from k8ssandra.
shallada
commented on May 25, 2024
Chrome dev tools shows this:
from k8ssandra.
shallada
commented on May 25, 2024
BTW - I am accessing the UI using Chrome on my mac running OS version 11.0.1. My Reaper is running on a KinD cluster running on Ubuntu 1804
from k8ssandra.
burmanm
commented on May 25, 2024
Any chance you could test with say Firefox to verify if this UI issue is only related to Chrome?
from k8ssandra.
shallada
commented on May 25, 2024
I tried Safari - same thing
from k8ssandra.
shallada
commented on May 25, 2024
The problem I was having was due to my use of ingress. Here's the configuration that worked for me (I used an Nginix ingress in addition to the Traefik ingress that K8ssandra uses):
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: petclinic-ingress
spec:
rules:
- http:
paths:
- path: /
backend:
serviceName: petclinic-frontend
servicePort: 8081
- path: /petclinic/api
backend:
serviceName: petclinic-backend
servicePort: 9966
- path: /prometheus
backend:
serviceName: k8ssandra-cluster-a-prometheus-k8ssandra
servicePort: 9090
- path: /grafana
backend:
serviceName: grafana-service
servicePort: 3000
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: reaper-ingress
annotations:
nginx.ingress.kubernetes.io/use-regex: "true"
nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
rules:
- http:
paths:
- path: /reaper(/|$)(.*)
backend:
serviceName: k8ssandra-cluster-a-reaper-k8ssandra-reaper-service
servicePort: 8080
---
As I understand it, this ingress configuration (reaper-ingress above) redirects traffic to /reaper to the Reaper service. The service is smart enough to deal with the base of the URL.
from k8ssandra.
Related Issues (20)
- Unable to connect with public ip after exposing K8ssandra with service type LoadBalancer
- getting 404 for EKS doc HOT 1
- When adding new node to an existing cluster, the new node gets deployed by K8s but stays at 1/2 forever HOT 17
- Add nodeSelector and tolerations support to cass-operator helm chart
- The search button in the K8ssandra documentation does not work HOT 1
- Warning CALL list keyspaces system_traces failed on all datacenter dc1 pods
- cass-operator chart does not allow override of certain images in values.yaml
- Need Helm chart to bring cassandra cluster as kind K8ssandraCluster for reaper and Medusa. HOT 4
- kustomize is now integrated into kubectl use this command instead
- Stargate doesn't start and it remains in CrashLoopBackoff state HOT 1
- Getting 404 for https://helm.k8ssandra.io/stable HOT 2
- How to create a single cluster with `containerd` instead of `docker` ?
- k8ssandra repair taking days to complete HOT 1
- Issue with Replication Alteration During K8ssandra v1 to K8ssandra-Operator Migration HOT 2
- cass-operator chart is missing resources
- If metrics are set to :8080, then the containerPort should be set also HOT 1
- Pull cass-operator from cr.k8ssandra.io and cr.dtsx.io
- Trying to set up ingress with nginx HOT 2
- Make it possible to disable the check for cert-manager in the Helm chart
- Certificate Request failure due to missing secret in cert-manager HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from k8ssandra.