Comments (11)
Thanks,
For replying back.
The issue got resolved I am able to create the certificate.
But the Certificate which was created not able to access in different namespaces using clusterissuer as Kind.
Do you have any reference yaml file. So that I can create the certificate which can be accessible across namespaces .
from cert-manager.
Hey @minigamkreddy thanks for raising. This is a fairly common error to see and usually it is networking, DNS or cloud provider specific as the issue. We have a guide to help debug if you could try that first?
https://cert-manager.io/docs/troubleshooting/webhook/
Failing that, can you please share your k8s environment details.
In general every cert-manager resource if sent to the cert-manager-webhook
deployment to validate the resource before it is saved to k8s to be actioned. It appears k8s cannot find that service, so please check that component is running.
from cert-manager.
Thanks For replying back
Yes I will Follow the link which you have provided me.
CERT MANAGER DETAILS
root@KmasterVM:/home/manoj/VM1_E810/vcsr-orch/helms# kubectl get pods -n cert-manager
NAME READY STATUS RESTARTS AGE
cert-manager-7ddd8cdb9f-c7kwx 1/1 Running 1 38h
cert-manager-cainjector-57cd76c845-fk77m 1/1 Running 1 38h
cert-manager-webhook-cf8f9f895-n6n6q 1/1 Running 1 38h
Environment Details
Kubernetes version:
kubectl version
Client Version: v1.28.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.1
OS Details
Description: Ubuntu 20.04.6 LTS
cert-manager version: ert-manager-controller:
Image: quay.io/jetstack/cert-manager-controller:v1.14.5
Note : For more Details related to Environment please reply back.
MORE DEATAILS
root@KmasterVM:/home/manoj/VM1_E810/vcsr-orch/helms# kubectl get endpoints -n cert-manager cert-manager-webhook
NAME ENDPOINTS AGE
cert-manager-webhook 192.168.224.104:10250 38h
kubectl get pod -n cert-manager -l app.kubernetes.io/name=webhook
NAME READY STATUS RESTARTS AGE
cert-manager-webhook-cf8f9f895-n6n6q 1/1 Running 1 39h
root@KmasterVM:/home/manoj/VM1_E810/vcsr-orch/helms# kubectl get pod -n cert-manager -l app.kubernetes.io/name=webhook
NAME READY STATUS RESTARTS AGE
cert-manager-webhook-cf8f9f895-n6n6q 1/1 Running 1 39h
root@KmasterVM:/home/manoj/VM1_E810/vcsr-orch/helms# kubectl logs -n cert-manager -l app.kubernetes.io/name=webhook | head -10
W0508 09:55:18.187126 1 client_config.go:618] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0508 09:55:18.273616 1 webhook.go:129] "using dynamic certificate generating using CA stored in Secret resource" logger="cert-manager.webhook" secret_namespace="cert-manager" secret_name="cert-manager-webhook-ca"
I0508 09:55:18.273781 1 server.go:146] "listening for insecure healthz connections" logger="cert-manager" address=":6080"
I0508 09:55:18.274173 1 server.go:206] "listening for secure connections" logger="cert-manager" address=":10250"
I0508 09:55:18.303989 1 reflector.go:351] Caches populated for *v1.Secret from k8s.io/[email protected]/tools/cache/reflector.go:229
I0508 09:55:19.280437 1 dynamic_source.go:255] "Updated cert-manager TLS certificate" logger="cert-manager" DNSNames=["cert-manager-webhook","cert-manager-webhook.cert-manager","cert-manager-webhook.cert-manager.svc"]
root@KmasterVM:/home/manoj/VM1_E810/vcsr-orch/helms#
root@KmasterVM:/home/manoj/VM1_E810/vcsr-orch/helms# kubectl get deploy -n cert-manager cert-manager-webhook -oyaml | grep -A3 ports:
ports:
- containerPort: 10250
name: https
protocol: TCP
from cert-manager.
resource mapping not found for name: "cert-manager" namespace: "" from "test-resources.yaml": no matches for kind "Issuer" in version "v1"
ensure CRDs are installed first
Error from server (InternalError): error when creating "test-resources.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": proxyconnect tcp: dial tcp 10.10.224.60:3128: connect: connection refused
Error from server (InternalError): error when creating "test-resources.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": proxyconnect tcp: dial tcp 10.10.224.60:3128: connect: connection
refused
When I do the kubectl apply -f test-resources.yaml cert-manager is excepted the response from the proxy server.
10.10.224.60:3128: these should not happen cert-manager should except the response from the current cluster.
Below commands are not working
kubectl -n cert-manager port-forward deploy/cert-manager-webhook 6080
curl -sS --dump-header - 127.0.0.1:6080/healthz => These command except the response from there 10.10.224.60:3128. These is not communicated with interval cluster.
How to reslove these issue.
from cert-manager.
If you run kubectl get crd
do you see the cert-manager CRD resources in the cluster?
Just looking at this error:
resource mapping not found for name: "cert-manager" namespace: "" from "test-resources.yaml": no matches for kind "Issuer" in version "v1"
ensure CRDs are installed first
Perhaps the CRDs were not installed, and that could explain it. But i expect they are there otherwise there should be more failing components.
from cert-manager.
root@KmasterVM:/home/manoj/VM1_E810/cert-manager# kubectl get crd
NAME CREATED AT
bgpconfigurations.crd.projectcalico.org 2023-09-05T04:25:42Z
bgppeers.crd.projectcalico.org 2023-09-05T04:25:42Z
blockaffinities.crd.projectcalico.org 2023-09-05T04:25:42Z
caliconodestatuses.crd.projectcalico.org 2023-09-05T04:25:42Z
certificaterequests.cert-manager.io 2024-05-07T14:58:36Z
certificates.cert-manager.io 2024-05-07T14:58:36Z
challenges.acme.cert-manager.io 2024-05-07T14:58:36Z
clusterinformations.crd.projectcalico.org 2023-09-05T04:25:42Z
clusterissuers.cert-manager.io 2024-05-07T14:58:36Z
felixconfigurations.crd.projectcalico.org 2023-09-05T04:25:42Z
globalnetworkpolicies.crd.projectcalico.org 2023-09-05T04:25:42Z
globalnetworksets.crd.projectcalico.org 2023-09-05T04:25:42Z
hostendpoints.crd.projectcalico.org 2023-09-05T04:25:42Z
ipamblocks.crd.projectcalico.org 2023-09-05T04:25:42Z
ipamconfigs.crd.projectcalico.org 2023-09-05T04:25:42Z
ipamhandles.crd.projectcalico.org 2023-09-05T04:25:42Z
ippools.crd.projectcalico.org 2023-09-05T04:25:42Z
ipreservations.crd.projectcalico.org 2023-09-05T04:25:42Z
issuers.cert-manager.io 2024-05-07T14:58:36Z
kubecontrollersconfigurations.crd.projectcalico.org 2023-09-05T04:25:42Z
networkpolicies.crd.projectcalico.org 2023-09-05T04:25:42Z
networksets.crd.projectcalico.org 2023-09-05T04:25:42Z
orders.acme.cert-manager.io 2024-05-07T14:58:36Z
root@KmasterVM:/home/manoj/VM1_E810/cert-manager#
root@KmasterVM:/home/manoj/VM1_E810/cert-manager# kubectl apply -f test-resources.yaml
root@KmasterVM:/home/manoj/VM1_E810/cert-manager# kubectl apply -f test-resources.yaml
namespace/cert-manager unchanged
Error from server (InternalError): error when creating "test-resources.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": Service Unavailable
Error from server (InternalError): error when creating "test-resources.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": Service Unavailable
from cert-manager.
above issue got resloved. But I am seeing another issue
oot@KmasterVM:/home/manoj/VM1_E810/cert-manager# kubectl apply -f test-resources.yaml
namespace/cert-manager-test unchanged
Error from server (InternalError): error when creating "test-resources.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-05-15T17:19:50Z is before 2024-05-15T21:01:12Z
Error from server (InternalError): error when creating "test-resources.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-05-15T17:19:50Z is before 2024-05-15T21:01:12Z
I have change the time to IST but still it doesn't work
Can you suggest me any idea how to reslove these issue.
from cert-manager.
Ok so you have CRDs and you now have a new error. Previously it was connection refused
but now you have "tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-05-15T17:19:50Z is before 2024-05-15T21:01:12Z"
Can you get the cert-manager-webhook-ca certificate out of the secret it is stored in?
k get secret -n cert-manager cert-manager-webhook-ca -o json | jq -r '.data["tls.crt"]' | base64 --decode | openssl x509 -text -noout
Assuming that cert is active (which it should be). I would validate that your server time is synced properly. This seems like your master node you are running on. Maybe try checking your server time compared to actual time to see if there is any drift?
sntp -d pool.ntp.org
For example my output seems to be:
..
+0.022888 +/- 0.025390 pool.ntp.org 162.159.200.1
Which seems very much in sync.
The concerning bit in your error is that the cert seems to be issued for the future in your case. So some time setting appears to be off: certificate has expired or is not yet valid: current time 2024-05-15T17:19:50Z is before 2024-05-15T21:01:12Z
from cert-manager.
The ClusterIssuer
is available to use by Certificate
resources from any namespace.
However the Certificate
resource itself is namespaced and cert-manager will only create a Secret
in the same namespace as the Certificate
.
It very much depends on your use case here. Usually the certificates would be mounted to an app, provided to and Ingress
in the same namespace and other controllers would reference the secret across namesapces, like an ingress controller deployment.
Can you describe the use case?
Do you know what resolved the original issue? Just incase anyone else has a similar problem in the future?
from cert-manager.
Hi @hawksight ,
Thanks for your response.
The original issue got resolved by fixing the proxy.
In our K8s, in /etc/kubernetes/ there were few files in which there was proxy set.
On removing the proxy, the issue got resolved.
Following is our usecase:
We have 2 applications running in Node1 and Node2 of single cluster.
APP1 running in Node1 in namespace node1
APP2 running in Node2 in namespace node2
We have common ClusterIssuer.yaml(currently we are using selfsigned) which should issue the certificate.
And the APP1 and APP2 helm charts will have respective certificate.yaml files which will refer to same ClusterIssuer.
We are able to create secret and mount it in APP1 and APP2.
We can see the ca.crt, tls.crt and tls.key in the mounted path in APP1 and APP2.
Now while doing certificate based authentication between APP1 and APP2, we are facing issue "root ca do not match".
We see that the ca.crt generated is different.
If we are referring same ClusterIssuer then the ca.crt should be same right ?
##############################################
Following is out ClusterIssue.yaml file:
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: selfsigned-clusterissuer
namespace: test
spec:
selfSigned: {}
#####################################################
And following is our certificate in APP1 charts:
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: test-ca
namespace: node1
spec:
isCA: true
commonName: "example1.com"
dnsNames:
- example1.com
secretName: test-ca-secret
issuerRef:
name: selfsigned-clusterissuer
kind: ClusterIssuer
group: cert-manager.io
############################################
And following is our certificate file in APP2 charts:
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: test-ca
namespace: {{ .Values.nameSpace }}
spec:
isCA: true
commonName: "example1.com"
dnsNames:
- example1.com
secretName: test-ca-secret
issuerRef:
name: selfsigned-clusterissuer
kind: ClusterIssuer
group: cert-manager.io
Can you please check why the ca.crt getting generated are different ?
from cert-manager.
The self signed issuer will create self signed certificates, so the two certificates created will not have the same root CA as they are self signed.
For your use case I would set up a CA issuer (which loads the CA from a secret) and use that. This would ensure certificates in both your apps have the same CA.
/priority awaiting-more-evidence
from cert-manager.
Related Issues (20)
- Cert Manager 503 Error for Domain
- Improvement of OpenSSF Scorecard Score
- "propagation check failed" err="wrong status code '404', expected '200'" HOT 1
- Stale/Stuck Challenges should be deleted after a given timeout HOT 3
- Challenges with not permitted zones in the domain are considered as processing HOT 1
- stale certificates present in metrics after certificate update
- Unhelpful log warning message: Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
- Helm chart conditions for CRDs are commented out
- Allow configuring Secret type
- Error after updating to latest version in webhook
- tls.crt doesn't contain the root but on the other side the trust manager recommends to trust only the root HOT 1
- Rename finalizer to conform to k8s requirements HOT 7
- `error finalizing order` message is light on details
- Proposal: define shortName for ClusterIssuer CRD
- Helm chart broken for helm v3.16.0 HOT 2
- Installation on k3s causes the ec2 t2.micro instance to hang
- Allow adding new fields such as unhealthyPodEvictionPolicy to the PDB
- Certificate Status: Issuing certificate as Secret does not exist
- Missing UID in webhook challenge request
- cert manager pods showing unknown flag HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cert-manager.