Comments (11)
abaghai
commented on August 16, 2024
1
I'm having this exact issue. Can't do anything in kubectl without getting this message:
error: You must be logged in to the server (the server has asked for the client to provide credentials)
from deploy-ibm-cloud-private.
tpouyer
commented on August 16, 2024
@ramonvanstijn I have an internal defect open for this issue and here is what the dev team told me:
"The issue has been fixed in k8s 1.7.5 kubernetes/kubernetes#49495 , and we are planning to upgrade k8s to 1.8.x #3376 in coming release, the problem will be resolved then."
For context the "coming release" is 2.1.0.1 which is not due out for at least a month.
As a work-around you can use vagrant suspend to halt the vm and vagrant up to resume it. This will not shutdown the vm and hence avoid this issue.
I have updated the README.md with this information for others who may run into the same problem. I will also leave this issue open so that others can easily find it.
from deploy-ibm-cloud-private.
ramonvanstijn
commented on August 16, 2024
@tpouyer Thanks for the swift response and work-around.
from deploy-ibm-cloud-private.
cattoire
commented on August 16, 2024
But what to do on a system where VM's will be rebooted once in a while. I have the same problem, in fact the catalog-api server is not starting after a reboot. Result is that the whole master is not starting. How can I reestablish this cluster now that it is in this state?
from deploy-ibm-cloud-private.
khahmed
commented on August 16, 2024
Please try the following workaround and see if it helps:
- Delete apiservices of v1alpha1.servicecatalog.k8s.io
kubectl delete apiservices v1alpha1.servicecatalog.k8s.io
- Delete the dead controller manager
docker rm "k8s controller manager"
-
wait till service catalog got started or just delete the pods and trigger a restart.
-
Recover the service catalog apiservices by re-register the apiservice of v1alpha1.servicecatalog.k8s.io
kubectl apply -f cluster/cfc-components/service-catalog/apiregistration.yaml
from deploy-ibm-cloud-private.
jtbeers
commented on August 16, 2024
kubernetes 1.8.3 is now in ibmcom docker hub
from deploy-ibm-cloud-private.
ramonvanstijn
commented on August 16, 2024
@jtbeers Thanks for the heads up. How to update kubernetes (to version 1.8.3) in a ICP cluster?
from deploy-ibm-cloud-private.
ramonvanstijn
commented on August 16, 2024
I redeployed using the latest and greatest Vagrantfile and rebooted The original issue disappeared but the cluster is not healthy after reboot as you can see on the attached screenshot from the dashboard.
. These is the output of kubectl get pods --all-namespaces:
NAMESPACE NAME READY STATUS RESTARTS AGE
federation-system federation-cluster-apiserver-6cb658d5c-lp5rj 1/1 Running 0 2h
federation-system federation-cluster-controller-manager-7978c96fc7-thk8z 1/1 Running 22 2h
kube-system auth-apikeys-vgt56 1/1 Running 0 32m
kube-system auth-idp-2k6bd 3/3 Running 0 32m
kube-system auth-pap-2tbb8 1/1 Running 0 32m
kube-system auth-pdp-vlq7t 1/1 Running 0 32m
kube-system calico-node-amd64-4wvht 2/2 Running 6 3h
kube-system calico-node-amd64-5tq72 2/2 Running 0 3h
kube-system calico-node-amd64-6xbks 2/2 Running 6 3h
kube-system calico-policy-controller-5997c6c956-9xcn5 1/1 Running 0 2h
kube-system catalog-catalog-apiserver-k5s9g 1/1 Running 0 2h
kube-system catalog-catalog-controller-manager-bd9f49c8c-6gmgz 1/1 Running 0 41m
kube-system catalog-ui-zb5mj 1/1 Running 0 2h
kube-system coredns-8696b8bf4c-xkmlv 1/1 Running 0 33m
kube-system default-http-backend-8448fbc655-gp88d 1/1 Running 0 39m
kube-system elasticsearch-client-6c9fc8b5b6-mkpbm 2/2 Running 0 2h
kube-system elasticsearch-data-0 1/1 Running 0 2h
kube-system elasticsearch-master-667485dfc5-rf4gs 1/1 Running 0 2h
kube-system etcd-0 1/1 Running 0 2h
kube-system filebeat-ds-amd64-9pdwc 1/1 Running 3 3h
kube-system filebeat-ds-amd64-ccmxl 1/1 Running 0 3h
kube-system filebeat-ds-amd64-tbfhx 1/1 Running 3 3h
kube-system heapster-5fd94775d5-zmf2b 2/2 Running 0 2h
kube-system helm-api-6b69694476-9jhdp 1/1 Running 0 33m
kube-system helmrepo-6b78cbc8c6-blq24 1/1 Running 1 35m
kube-system icp-ds-0 1/1 Running 0 42m
kube-system icp-router-6vlv8 1/1 Running 2 2h
kube-system image-manager-0 2/2 Running 0 2h
kube-system k8s-etcd-192.168.27.100 1/1 Running 0 3h
kube-system k8s-mariadb-192.168.27.100 1/1 Running 0 3h
kube-system k8s-master-192.168.27.100 3/3 Running 0 3h
kube-system k8s-proxy-192.168.27.100 1/1 Running 0 3h
kube-system k8s-proxy-192.168.27.101 1/1 Running 3 3h
kube-system k8s-proxy-192.168.27.102 1/1 Running 3 3h
kube-system kube-dns-9494dc977-mncwb 3/3 Running 0 35m
kube-system logstash-5ccb9849d6-6hh6f 1/1 Running 0 2h
kube-system metering-dm-8587b865b4-zjc4p 0/1 Running 2 2h
kube-system metering-reader-amd64-pht7q 1/1 Running 8 3h
kube-system metering-reader-amd64-tjn8l 1/1 Running 2 3h
kube-system metering-reader-amd64-w2wqb 1/1 Running 9 3h
kube-system metering-server-748d8f8f5b-rvx78 0/1 Running 2 2h
kube-system metering-ui-75ff5748c-rhh8w 0/1 Running 2 2h
kube-system monitoring-exporter-76b94fdd94-hm4pp 1/1 Running 0 2h
kube-system monitoring-grafana-5c49f54dd-2vn9d 2/2 Running 0 2h
kube-system monitoring-prometheus-77d4df9dd6-9276r 3/3 Running 0 2h
kube-system monitoring-prometheus-alertmanager-564496655f-hkq64 3/3 Running 0 2h
kube-system monitoring-prometheus-kubestatemetrics-776b5dcb86-d5hg9 1/1 Running 0 2h
kube-system monitoring-prometheus-nodeexporter-amd64-2bnpr 1/1 Running 3 3h
kube-system monitoring-prometheus-nodeexporter-amd64-bm2r6 1/1 Running 0 3h
kube-system monitoring-prometheus-nodeexporter-amd64-dhrlv 1/1 Running 3 3h
kube-system nginx-ingress-lb-amd64-85s27 1/1 Running 0 2h
kube-system platform-api-jflf7 1/1 Running 0 2h
kube-system platform-ui-5d5gm 1/1 Running 0 2h
kube-system rescheduler-xr46x 1/1 Running 0 2h
kube-system tiller-deploy-55fb4d8dcc-44bm6 1/1 Running 0 41m
kube-system unified-router-ms24q 1/1 Running 2 2h
from deploy-ibm-cloud-private.
mc-chris
commented on August 16, 2024
If you are on the master node, you can run kubectl this way
kubectl -s http://127.0.0.1:8888 get pods --all-namespaces
from deploy-ibm-cloud-private.
ramonvanstijn
commented on August 16, 2024
An update on my ICP CE 2.1.0.1 issue(s) after reboot.
I am able to get more predictive results after removing the section below from /usr/local/bin/icp-ce-startup.sh
sudo docker ps -a | grep Exit | cut -d ' ' -f 1 | xargs sudo docker rm > /dev/null || true
sleep 180
kubectl config set-credentials icpadmin --username=admin --password=admin &> /dev/null
kubectl config set-cluster icp --server=http://127.0.0.1:8888 --insecure-skip-tls-verify=true &> /dev/null
kubectl config set-context icp --cluster=icp --user=admin --namespace=default &> /dev/null
kubectl config use-context icp &> /dev/null
kubectl get pods -o wide -n kube-system | grep "icp-ds" | cut -d ' ' -f 1 | xargs kubectl -n kube-system delete pods
sleep 120
while [[ '' != $(kubectl get pods --namespace kube-system | sed -n '1!p' | grep -v Running) ]]
do
kubectl get pods -o wide -n kube-system | grep "CrashLoopBackOff\\|Init" | cut -d ' ' -f 1 | xargs kubectl -n kube-system delete pods
sleep 120
done
It looks to me - at least on my laptops - that more time is needed to start up all the pods. Additionally I am worried about the deletion of pods in the above removed section during startup. I know k8s will start new pods but I found out that deleting the monitoring-grafana-xxxx pod is a bad idea because it will loose its state, the prometheus datasource config, and ultimately the monitoring dashboards stop working. The workaround for this problem is to run the Batch Job that creates the prometheus datasource again:
kubectl replace --force -f /home/vagrant/cluster/cfc-components/monitoring/grafana-set-ds-job.yaml
I identified two additional pods that need attention, helmrepo-xxxxx and helm-api-xxxxx. I edited the corresponding deployments and changed the livenessProbe values to (changes in bold):
livenessProbe:
failureThreshold: 10
httpGet:
path: "/healthcheck"
port: 3000
scheme: HTTP
initialDelaySeconds: 30
periodSeconds: 120
successThreshold: 1
timeoutSeconds: 30
I captured the values from the upcoming version 2.1.0.2.
from deploy-ibm-cloud-private.
ramonvanstijn
commented on August 16, 2024
Closed due to inactivity.
from deploy-ibm-cloud-private.
Related Issues (20)
- ICP 3.1.0 installation fails - referring incorrect version of helm-api, helm-rudder and cert-gen images
- Nodes become inactive after several days of ICP idling around after installation HOT 2
- Unable to access heketi cluster IP after ICP 3.1.1 installed on CentOS 7.2 HOT 2
- Terraform openstack is not correctly populating /etc/hosts
- ICP 3.1.1 web console displays Welcome to OpenResty! HOT 6
- Issues with SSL certificates installing using Vagrant HOT 13
- upgrade-prepare from ICP CE from 3.1.0 to 3.1.2 failing on master node HOT 1
- null_resource.icp-worker-scaler: timeout - last error: ssh: handshake failed: ssh HOT 1
- failed while Deploying in Softlayer VMs using Ansible - Waiting for platform auth service HOT 1
- [Help] Installation of ICP using vagrant failed, possibly due to timeout error HOT 1
- Native LXC Deployment HOT 4
- Unable to login to ICP 3.1.2 console deployed on OpenShift 3.11 HOT 2
- Issue while installating ICP 3.1.1 : FAILED - RETRYING: Waiting for ClusterIssuer CRD
- Unable to connect to the server: dial tcp 192.168.36.100:8001: connect: no route to host HOT 1
- Vagrant install fails with "icp: Error response from server. Status code: 403; message: Error 403 : Access Forbidden" HOT 1
- Vagrant file for installing ICP EE HOT 7
- IBM Cloud Pak for Integration: CWOAU0062E: The OAuth service provider could not redirect HOT 1
- Cloudctl iam team-add-groups fails with an error when the team has an underscore (_) in the name
- Errors when install IBM-Cloud-Private V 3.2.0 HOT 1
- Deployments in IBM Cloud Private 3.2.1 dont launch
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from deploy-ibm-cloud-private.