Giter Club home page Giter Club logo

Comments (17)

nabadger avatar nabadger commented on May 23, 2024

Looking at this now

from dex-k8s-authenticator.

nabadger avatar nabadger commented on May 23, 2024

@praveendhac It looks like you're deploying the Dex chart with the dex-k8s-authenticator values (they are 2 independent charts)

I think you should be using:

helm upgrade \
 --install 
 --namespace <dex-k8s-auth-ns>
 --values <dex-k8s-auth-values.yaml> charts/dex-k8s-authenticator

.

Let me know if this makes sense and fixes the issue for you.

from dex-k8s-authenticator.

praveendhac avatar praveendhac commented on May 23, 2024

dex-server-ns is the namespace where Dex is deployed so deploying dex-k8s-authenticator- in the same namespace.
renamed dex-k8s-authenticator.yaml to dex-k8s-authenticator-dex-server.yaml with values specific to my config.
Helm deployment command used

helm upgrade --install --namespace dex-server-ns --values dex-k8s-authenticator-dex-server.yaml dex-client-app-helm charts/dex-k8s-authenticator

Pod Status and Events

$ kubectl get all -n dex-server-ns
NAME                                                             READY     STATUS             RESTARTS   AGE
pod/dex-client-app-helm-dex-k8s-authenticator-6dfd8f6ddf-rvcjc   0/1       CrashLoopBackOff   3          1m

Events:
  Type     Reason     Age                From                                                              Message
  ----     ------     ----               ----                                                              -------
  Normal   Scheduled  48s                default-scheduler                                                 Successfully assigned dex-server-ns/dex-client-app-helm-dex-k8s-authenticator-6dfd8f6ddf-rvcjc to dex-server-cpu-worker-b5f659cc4-r24ld
  Normal   Pulling    20s (x3 over 47s)  kubelet, dex-server-cpu-worker-b5f659cc4-r24ld  pulling image "mintel/dex-k8s-authenticator:latest"
  Normal   Pulled     20s (x3 over 42s)  kubelet, dex-server-cpu-worker-b5f659cc4-r24ld  Successfully pulled image "mintel/dex-k8s-authenticator:latest"
  Normal   Created    20s (x3 over 42s)  kubelet, dex-server-cpu-worker-b5f659cc4-r24ld  Created container
  Normal   Started    20s (x3 over 42s)  kubelet, dex-server-cpu-worker-b5f659cc4-r24ld  Started container
  Warning  BackOff    3s (x6 over 41s)   kubelet, dex-server-cpu-worker-b5f659cc4-r24ld  Back-off restarting failed container

$ helm ls -a
NAME               	REVISION	UPDATED                 	STATUS  	CHART                      	NAMESPACE
dex-client-app-helm	2       	Thu Jul 12 23:10:38 2018	DEPLOYED	dex-k8s-authenticator-0.1.2	dex-server-ns
dex-server-helm    	1       	Wed Jul 11 17:18:32 2018	DEPLOYED	dex-0.2.2                  	dex-server-ns

from dex-k8s-authenticator.

nabadger avatar nabadger commented on May 23, 2024

@praveendhac can you show the output of kubectl describe pod <pod> and kubectl logs <pod> for the dex-client-app here?

from dex-k8s-authenticator.

praveendhac avatar praveendhac commented on May 23, 2024

Please find below details

$ kubectl get all -n dex-server-ns
NAME                                                             READY     STATUS             RESTARTS   AGE
pod/dex-client-app-helm-dex-k8s-authenticator-6dfd8f6ddf-rvcjc   0/1       CrashLoopBackOff   159        13h

$ kubectl describe pod dex-client-app-helm-dex-k8s-authenticator-6dfd8f6ddf-rvcjc -n dex-server-ns
Name:           dex-client-app-helm-dex-k8s-authenticator-6dfd8f6ddf-rvcjc
Namespace:      dex-server-ns
Node:           dex-server-cpu-worker-b5f659cc4-r24ld/
Start Time:     Thu, 12 Jul 2018 23:07:54 +0100
Labels:         app=dex-k8s-authenticator
                env=dev
                pod-template-hash=2898492889
                release=dex-client-app-helm
Annotations:    cni.projectcalico.org/podIP=REDACTED
Status:         Running
IP:             REDACTED
Controlled By:  ReplicaSet/dex-client-app-helm-dex-k8s-authenticator-6dfd8f6ddf
Containers:
  dex-k8s-authenticator:
    Container ID:  docker://855b707b6c8ae916058153da3d8a4d16f5487d9947f225d3f14b26452f3e3909
    Image:         mintel/dex-k8s-authenticator:latest
    Image ID:      docker-pullable://mintel/dex-k8s-authenticator@sha256:cb1555153df3b589c85ab16b9bb5966caefe630a7c9769cfb942c5b1b3d04b5b
    Port:          5555/TCP
    Host Port:     0/TCP
    Args:
      --config
      config.yaml
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Fri, 13 Jul 2018 12:17:29 +0100
      Finished:     Fri, 13 Jul 2018 12:17:29 +0100
    Ready:          False
    Restart Count:  159
    Liveness:       http-get http://:http/ delay=0s timeout=1s period=10s #success=1 #failure=3
    Readiness:      http-get http://:http/ delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:    <none>
    Mounts:
      /app/config.yaml from config (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-5tllt (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      dex-client-app-helm-dex-k8s-authenticator
    Optional:  false
  default-token-5tllt:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-5tllt
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason   Age                  From                                                              Message
  ----     ------   ----                 ----                                                              -------
  Warning  BackOff  1m (x3660 over 13h)  kubelet, dex-server-cpu-worker-b5f659cc4-r24ld  Back-off restarting failed container

$ kubectl logs pod/dex-client-app-helm-dex-k8s-authenticator-6dfd8f6ddf-rvcjc -n dex-server-ns
Error from server: Get https://dex-server-cpu-worker-b5f659cc4-r24ld:10250/containerLogs/dex-server-ns/dex-client-app-helm-dex-k8s-authenticator-6dfd8f6ddf-rvcjc/dex-k8s-authenticator: dial tcp: lookup dex-server-cpu-worker-b5f659cc4-r24ld on 10.0.0.10:53: server misbehaving

Able to describe the pod but unable to pull logs from pod.

from dex-k8s-authenticator.

nabadger avatar nabadger commented on May 23, 2024

Ok, so if dex-k8s-authenticator cannot contact the the dex-server, it will crash on startup - there's an outstanding issue to resolve this, but it's not actually a problem since kubernetes will retry anyway.

The key issue here is that the pod cannot contact the dex-server instance.

It looks like it can't talk to https://dex-server-cpu-worker-b5f659cc4-r24ld:10250

It should be contacting it via the ingress.

I'm really not sure why it's actually pointing at:

https://dex-server-cpu-worker-b5f659cc4-r24ld:10250/containerLogs/dex-server-ns/dex-client-app-helm-dex-k8s-authenticator-6dfd8f6ddf-rvcjc/dex-k8s-authenticator

This to me looks like your dex-k8s-authenticator config is wrong.

dex-k8s-authenticator will use the issuer field in it's config and attempt to contact that host over http.

Can you confirm what the issuer field is set to - from your other case, it looks like the issuer should be set to https://dex.ingress.dex-server.example.com

You can confirm this by checking the configmap.

from dex-k8s-authenticator.

praveendhac avatar praveendhac commented on May 23, 2024 
$ kubectl get configmaps dex-client-app-helm-dex-k8s-authenticator -n dex-server-ns -o yaml
apiVersion: v1
data:
  config.yaml: |-
    listen: http://0.0.0.0:5555
    debug: true
    clusters:
    - client_id: dex-server-clientid
      client_secret: REDACTED
      description: dex-server-cluster Shoot Cluster Long Description...
      issuer: https://dex.ingress.dex-server.example.com
      k8s_master_uri: https://api.dex-server.example.com
      name: dex-server-cluster
      redirect_uri: https://login.ingress.dex-server.example.com/callback
      short_description: dex-server-cluster Shoot
kind: ConfigMap
metadata:
  creationTimestamp: 2018-07-12T22:07:54Z
  labels:
    app: dex-client-app-helm-dex-k8s-authenticator
    chart: dex-k8s-authenticator-0.1.2
    env: dev
    heritage: Tiller
    release: dex-client-app-helm
  name: dex-client-app-helm-dex-k8s-authenticator
  namespace: dex-server-ns
  resourceVersion: "214012"
  selfLink: /api/v1/namespaces/dex-server-ns/configmaps/dex-client-app-helm-dex-k8s-authenticator
  uid: 069ca682-8620-11e8-80e2-b223bcbc03d4

from dex-k8s-authenticator.

nabadger avatar nabadger commented on May 23, 2024

What if you create a busybox instance (same namespace) and try to wget or curl https://dex.ingress.dex-server.example.com/healthz

Another way is to modify the dex-k8s-authenticator yaml and override the cmd and args option:

command: ["/bin/sh"]
args: ["-c", "while true; do sleep 1000;done"]

This will run a pod using the dex-k8s-auth image, but let you exec into it and debug it further.

from dex-k8s-authenticator.

trinitronx avatar trinitronx commented on May 23, 2024

@praveendhac : Looks like same behavior when Ingress DNS records for dex don't exist or are not connectable as described by @nabadger

You'll probably see errors like the following If you can get logs out of last crashed pod. Looks like your kubectl logs [...SNIP...] command is not able to access logs on the old pod because it's already gone! You might have luck getting logs out if it's in CrashLoopBackoff with old pods that stick around longer rather than in fast CrashLoop state where pods are cycling too fast. Also if you installed a log shipper to ElasticSearch or some similar log aggregation service you could find the logs there.

Here's what you might see:

$ kubectl -n kube-system logs login-dex-k8s-authenticator-76c8dd6488-94gqt
            '/certs/ca-cert2/..2018_07_12_17_29_20.747868164/ca-cert2' -> '/usr/local/share/ca-certificates/ca-cert2.crt'
            '/certs/ca-cert1/..2018_07_12_17_29_20.735577337/ca-cert1' -> '/usr/local/share/ca-certificates/ca-cert1.crt'
            WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
            2018/07/12 22:12:27 Using config file:%!(EXTRA string=/app/config.yaml)
            2018/07/12 22:12:28 Creating new provider https://dex.kops-ldap.prod.local.foo
            2018/07/12 22:12:28 Failed to query provider "https://dex.kops-ldap.prod.local.foo": Get https://dex.kops-ldap.prod.local.foo/.well-known/openid-configuration: dial tcp: lookup dex.kops-ldap.prod.local.foo on 100.64.0.10:53: no such host

If this is the case, it means that dex-k8s-authenticator cannot resolve the DNS name for dex (e.g. dex.kops-ldap.prod.local.foo).

If you are using nginx-ingress-controller, check it's command line args for --ingress-class=. You need to make sure your Ingress object for dex has the matching annotation and your desired DNS names. For example, if you had --ingress-class=nginx-internal the annotation must match kubernetes.io/ingress.class: nginx-internal:

kubectl -n kube-system  get ingress dex -o yaml --export=true
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    ingress.kubernetes.io/force-ssl-redirect: "true"
    kubernetes.io/ingress.class: nginx-internal
  creationTimestamp: null
  generation: 1
  labels:
    app: dex
    chart: dex-0.2.2
    env: prod
    heritage: Tiller
    release: dex
  name: dex
  selfLink: /apis/extensions/v1beta1/namespaces/kube-system/ingresses/dex
spec:
  rules:
  - host: dex.kops-ldap.prod.local.foo
    http:
      paths:
      - backend:
          serviceName: dex
          servicePort: 5556
        path: /
  - host: dex.local.foo
    http:
      paths:
      - backend:
          serviceName: dex
          servicePort: 5556
        path: /
  - host: dex.prod.local.foo
    http:
      paths:
      - backend:
          serviceName: dex
          servicePort: 5556
        path: /
  - host: dex.kube-system
    http:
      paths:
      - backend:
          serviceName: dex
          servicePort: 5556
        path: /
  - host: dex.kube-system.svc
    http:
      paths:
      - backend:
          serviceName: dex
          servicePort: 5556
        path: /
  - host: dex.kube-system.svc.cluster.local
    http:
      paths:
      - backend:
          serviceName: dex
          servicePort: 5556
        path: /
  tls:
  - hosts:
    - dex.kops-ldap.prod.local.foo
    - dex.local.foo
    - dex.prod.local.foo
    - dex.kube-system
    - dex.kube-system.svc
    - dex.kube-system.svc.cluster.local
    secretName: dex-tls
status:
  loadBalancer: {}

Finally, unless you're running dns-controller you also need to point the external name for dex (e.g. dex.kops-ldap.prod.local.foo) to the Load balancer started by nginx-ingress-controller (this assumes you're in AWS using Route53).

Get your ELB DNS name by checking the Service for nginx-ingress-controller:

kubectl get svc nginx-ingress-internal-controller -n kube-system -o json | jq -r '.status.loadBalancer.ingress[0].hostname'

Create a Route53 CNAME to point dex's external name to this ELB DNS name. For example, you might point: dex.kops-ldap.prod.local.foo => internal-abcdef0123456789abcdef0123456789-12345678.us-east-1.elb.amazonaws.com

from dex-k8s-authenticator.

nabadger avatar nabadger commented on May 23, 2024

It might just be that Dex itself isn't running:

The command you pasted above shows:

kubectl get all -n dex-server-ns
NAME                                                             READY     STATUS             RESTARTS   AGE
pod/dex-client-app-helm-dex-k8s-authenticator-6dfd8f6ddf-rvcjc   0/1       CrashLoopBackOff   159        13h

I believe you were intending to run Dex in the same namespace - if so, then you should expect to see Dex running in this above command (unless you stripped out that output)

from dex-k8s-authenticator.

praveendhac avatar praveendhac commented on May 23, 2024

Output of kubectl get all -n dex-server-ns has pods for dex and authenticator, deployments, services etc., just pasted dex-authenticator output to reduce unnecessary verbosity.

from dex-k8s-authenticator.

nabadger avatar nabadger commented on May 23, 2024

This still looks like a networking / routing / dns / ingress issue to me.

Are you running a NetworkPolicy or a service-mesh?

The easiest way of debugging this that I've found is per #47 (comment)

If you modify the deployment to not run dex-k8s-auth on start, that will let you exec into the pod. From there you can debug network connectivity issues, plus you can see the actual configuration that is in use, and you can actually run the command by hand to confirm what happens.

Or just run another Pod with dnsutils and similar installed to debug this (it doesn't have to be dex-k8s-auth).

I've had similar issues before (not dex related) where I could not access Ingress from in-cluster pods - this was down to dns resolution mis-configuration.

from dex-k8s-authenticator.

praveendhac avatar praveendhac commented on May 23, 2024

Used #47 (comment) to debug, there is domain name mismatch error in Dex Authenticator Pod.

/app# bin/dex-k8s-authenticator --config config.yaml --debug
2018/07/16 11:51:51 Using config file:%!(EXTRA string=/app/config.yaml)
2018/07/16 11:51:51 Creating new provider https://dex.ingress.example.com
2018/07/16 11:51:51 GET /.well-known/openid-configuration HTTP/1.1
Host: dex.ingress.example.com

2018/07/16 11:51:51 Failed to query provider "https://dex.ingress.example.com": Get https://dex.ingress.example.com/.well-known/openid-configuration: x509: certificate is valid for ingress.local, not dex.ingress.example.com
/app # command terminated with exit code 137

Can I use below config to bringup Dex Server with custom certificates.

web:
  http: 0.0.0.0:5556
  # Uncomment for HTTPS options.
  # https: 127.0.0.1:5554
  tlsCert: /etc/dex/tls.crt
  tlsKey: /etc/dex/tls.key

Or, get the certs from Dex Server and pass it to Dex Authenticator to fix the error.

from dex-k8s-authenticator.

praveendhac avatar praveendhac commented on May 23, 2024

Also figured out, self-signed certificates will not work.

2018/07/16 14:00:08 Failed to query provider "https://dex.ingress.example.com": Get https://dex.ingress.example.com/.well-known/openid-configuration: x509: certificate signed by unknown authority
/app # command terminated with exit code 137

from dex-k8s-authenticator.

nabadger avatar nabadger commented on May 23, 2024

@praveendhac @trinitronx

You can add certs in the dex-k8s-authenticator helm chart (see values.yaml), but we think we've identified a recent bug.

There's an open case #42 which suggests a bug in how the certs are mounted in dex-k8s-authenticator helm chart. I believe there's a quick fix suggested here #42 (comment)

If that works, I can merge that change, and you should be ok (maybe you can test the fix?)

There's a related issue here: #35

Regarding adding certs on the Dex chart, there's an open case to add that function here: #22

from dex-k8s-authenticator.

praveendhac avatar praveendhac commented on May 23, 2024

@nabadger
I followed below steps which worked for me.
Create Certs (used lets encrypt as self-signed certs are not working). Create secret in the same namespace where helm charts are deployed.

$ certbot certonly --manual -d *.ingress.dex-appser.praveen.com --work-dir=le_wd --config-dir=le_cd --logs-dir=le_ld
$ kubectl create secret tls pd-custom-certs --key ingress.dex-appser.praveen.com.key.pem --cert ingress.dex-appser.praveen.com.crt.pem -n dex-appser-ns

Modify ingress for Dex Server and Dex Authenticator as shown below

----SNIP(Server)----
ingress:
  enabled: true
  path: /
  hosts:
    - dex.ingress.dex-appser.praveen.com
  tls:
    - hosts:
      - dex.ingress.dex-appser.praveen.com
      secretName: pd-custom-certs

----SNIP(Authenticator)----
ingress:
  enabled: true
  path: /
  hosts:
    - login.ingress.dex-appser.praveen.com
  tls:
    - secretName: pd-custom-certs
      hosts:
        - login.ingress.dex-appser.praveen.com

from dex-k8s-authenticator.

nabadger avatar nabadger commented on May 23, 2024

@praveendhac great - glad you got it working.

I'll close this for now, but will prioritise a few things

1 - documentation
2 - fixing dex-k8s-auth helm chart for self-signed certs
3 - updating dex helm chart for self-signed certs

from dex-k8s-authenticator.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.