Implement control loop to implement `kubernetes.io/tls-acme: "true"` behaviour on Ingress resources about cert-manager HOT 19 CLOSED

I've just merged #210 which adds an 'ingress-shim' to cert-manager, that does exactly this. It does not bind explicitly to ACME HTTP01 as discussed in this thread previously.

It's now possible to specify an issuer name (or cluster issuer name) in an annotation on an ingress resource (e.g. certmanager.k8s.io/issuer), and have corresponding certificates created automatically. By setting the --default-issuer-name and --default-issuer-kind flags on ingress-shim, it is also possible to re-implement kube-lego's kubernetes.io/tls-acme annotation.

Documentation needs writing up on how to do this, but in the meantime any feedback would be greatly appreciated!

By running helm install with something like: --set ingressShim.extraArgs='{--default-issuer-name=letsencrypt-staging,--default-issuer-kind=ClusterIssuer}', you should now be able to provision certificates with nothing more than a kubernetes.io/tls-acme annotation on your ingresses!

from cert-manager.

@thearrow We patched cert-manager to get credentials from the local environment.

That patch got caught up in internal processes at $dayjob and in the meantime, another set of patches was merged here:

#363

You may need to upgrade cert-manager to enable that sort of functionality.

from cert-manager.

Correct @munnerz you can't filter by annotations. At the k8s API server end, it creates indexes for efficient label filtering, but does nothing for annotations. So to efficiently watch only the resources you care about, a label should be used for anything that might be a 'watch' or 'get' filter.

Many early k8s services like nginx-ingress and originally kcm made the mistake of using annotations for everything, which makes them less scalable with resource count and with multiple service instances. I went through and fixed this in kcm recently, converting the 'enabled' annotation to a 'class' label.

cert-manager should use a 'class' label too I think, to enable the option for multiple instances of cert-manager to run in a cluster and watch only their own Certificate and Ingress resources (e.g. different namespaces/projects/deployments may not want to share the same cert-manager credentials, issuers, DNS providers, especially in the multi-tenant case). If you only have one cert-manager running then use the 'default' class (and default the cert-manager deployment to use that 'default' class). If you don't want a resource processed at all (aka 'enabled'=false), just label it with a 'disabled' class that no cert-manager instance is watching for (or just delete the label).

Anything you only need to use for processing Ingresses and would never want to filter on, then best to use an annotation so the API server doesn't waste time and space indexing it.

Possible example:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: example-ingress
  labels:
    certmanager.kubernetes.io/class: "default"
  annotations:
    certmanager.kubernetes.io/issuer: "letsencrypt-prod"
    certmanager.kubernetes.io/acme-challenge: "dns-01"
    certmanager.kubernetes.io/dns-provider: "route53"
    certmanager.kubernetes.io/dns-provider-secret: "route53-account-1"
...

Since I just when to the trouble of deprecating the 'enabled' annotation in 'kcm', please don't bring it back :-)

from cert-manager.

Thanks for the clarification on filtering -

So I'm currently undecided on whether we want to introduce the concept of a 'class' into cert-manager. I see why it's existed in the past, because neither cert-manager nor kube-lego supported multiple configurations for a single provider without additional processes running (hence, one process per 'class'). But with the move to the Issuer resource, all credentials and access information for providers is stored within the API server, and consequently we can consider the use of RBAC for authorization/access control. The idea of a 'Class' in Ingress exists solely because different ingress controllers support different implementations, but cert-manager consolidates all of that logic into the one binary (instead of having a binary per issuer)

I'm not sold that performance will become an issue if we're watching all resources in a namespace (of the types we're interested in) - as kube-controller-manager, which is designed in a very similar way, manages to watch all resources across all namespaces without issue, and using a single leader elected replica (again, like cert-manager). I'd rather keep the API surface as small as possible if it's not a required feature. I do agree that being able to set cert-manager to only watch particular namespace(s) is important though, and that could provide the isolation that you are looking for, without needing 'classes' maybe?

I'm also not 100% sure I want to expose so much config through annotations on ingress - if a user wants to use the ACME DNS01 challenge support, I think it's best they explicitly create a Certificate resource instead of annotating an Ingress.

I kind of see this method of configuring cert-manager to grab a certificate as solely for the ACME HTTP01 challenge provider - configuration becomes problematic otherwise. wdyt?

from cert-manager.

(1) So long as we are using a label for 'enabled' or 'class' we have the implementation option to watch everything or to filter. If we use an annotation now, and eventually do hit a performance barrier, we have a pretty large breaking change to deal with to add filtering.

(2) I was hoping to retain the ability to run two cert-manager deployments watching the same namespace, or two cert-manager deployments in a cluster watching all namespaces. Consider your own use case, developing the next version of cert-manager in a cluster that already has an operational cert-manager deployed watching all namespaces. You can deploy your test version as class 'test', and create some 'test' resources for it to process.
Whether you decide on 'enabled' or 'class' you have just one annotation/label on the resource. I'm saying don't unnecessarily limit it to two values "true"/"false", how about "disabled"/"enabled"/"my-crazy-use-case" :-)

(3) I agree that it would be preferable not to have heaps of configuration in Ingress annotations. The only reason it was all there at all for kcm was because when processing Ingress resources, was because there is no Certificate/Issuer resources in that case. TLS Secrets were created based on the Ingress resource alone. If cert-manager is going to require Certificates/Issuer resources to go with the Ingress then definitely the configuration belongs there. Would you have different Issuer resources for different challenge methods then?

certmanager.kubernetes.io/issuer: "letsencrypt-prod-http01"
vs.
certmanager.kubernetes.io/issuer: "letsencrypt-prod-dns01-aws"

(4) That brings up an unsupported use case I have run into recently. When you have an Ingress with multiple domain names, but where the certificates for those domain names have to be obtained from different issuers. E.g. because they are hosted by different DNS providers, or because one third-party name requires http-01. Could the new Certificate/Issuer approach solve that? E.g. if the Certificate name matched the TLS Secret name configured in the Ingress resource, and the Certificate resource named the appropriate Issuer resource for that certificate?

(5) We probably have different views about http-01, for us http-01 is what we used before dns-01 was available and have deprecated now. About 98% of our Ingress certs are dns-01-issued, and a only a few edge cases of third party names still use http-01. Having to publicly expose HTTP/1 end-points in an HTTPS-only world seems quite retrograde, and the evolving ACME HTTPS challenge methods (using TLS hand-shake injections) are pretty messy and are going to require complicit Ingress controllers. So my view is, if you have the choice, why use anything but dns-01? :-)

from cert-manager.

(1) agreed - for reference, this issue is specifically relating to the label to go on Ingress resources that will trigger a simple control loop that creates a corresponding Certificate resource that targets the Ingresses tls secret using the Issuer as specified on the Ingress resource. The way that process should work precisely however was the original topic of discussion for the issue 😄 I think we're both in agreement that a label of some sort on the Ingress resource is preferable.

(2) my question here is why you want to do this? Given cert-manager is configured purely via Issuer resources, and so itself has very little configuration surface, I don't see what the advantage of having some Certificate/Issuer resources processed by one instance of cert-manager, and others processed by another (aside from the Namespace case, which I can definitely agree with due to things like RBAC).

(3) Again, that's a topic for discussion in this issue. I see the ability to create Certificates via annotations on an Ingress as an added 'convenience' function to trigger cert-manager to create you a basic Certificate resource that works in the most basic use case (user creating a standard http ingress either gce/haproxy/aws/nginx) that allows http traffic as well as https and so can be validated via http01. I don't see it as the recommended way to configure your certificate, but just a quick short hand.

(4) I thought about this use case in the design, but realised that it's not actually possible to do. The CSR contains a list of domains to get signed - we can't send two different CSRs to two different issuers and then merge them. We can however validate our ownership of a domain via multiple different challenge mechanisms and then create a single CSR for all of those domains, which I think is what you are describing. cert-manager actually already supports that today! I've just updated the example yaml manifests (which are the closest thing to documentation right now) which should make how you can achieve this clearer: https://github.com/jetstack-experimental/cert-manager/blob/master/docs/acme-issuer.yaml and https://github.com/jetstack-experimental/cert-manager/blob/master/docs/acme-cert.yaml

As you can see, I've configured two different DNS providers for a single letsencrypt staging issuer. I've then requested a certificate the validates 2 domains via http01 with the nginx ingress class (this will create new ingress resources for each challenge, and then delete them afterward), one that is validated by modifying an existing ingress (this is a gce load balancer, which we now support fully and just as well as others 😄) and additional we validated one domain with cloudflare and another domain with google clouddns.

(5) I do agree with you - dns is nice and reliable. I do not by any means intend to force users to use http01. On the contrary, I'm investigating how we can implement the tls-sni-02 challenge mechanism into the acme issuer too 😄

from cert-manager.

might also be worth noting that once a user creates adds a label to the Ingress resource and cert-manager creates a corresponding Certificate resource for it, there is nothing to stop a user editing the Certificate resource if they want to change the challenge mechanism or even issuer in future. I'm really envisioning it being a convenience mechanism to trigger the creation of a default Certificate resource, which given we know that HTTP01 will probably work provided the user has a functioning ingress controller, it seems to make sense to default to acme http01 😄 (also legacy reasons as that was the previous behaviour).

I'm also open to this not being true/false still though. We could potentially just use a label of certmanager.k8s.io/issuer: letsencrypt-prod to cause cert-manager to create a Certificate resource configured with acme http01 for the letsencrypt-prod issuer?

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: example-ingress
  labels:
    certmanager.kubernetes.io/issuer: "letsencrypt-prod"
...

from cert-manager.

(1) 👍

(2/3) Nothing to add. See what others say, maybe I'm the only one who cares 😉

(4) I checked out your Certificate / Issuer examples and I love them. I see how the multiple challenge mechanisms per CSR would work, and that will be fabulous ❤️ to have that option available. My only concern was with the auth secrets being stored in a plain resource rather than a ref to a Secret.

(5) I looked into tls-sni-02 for kcm and backed away fast, it seemed to me it couldn't be implemented on the back side of an Ingress, since you need the raw TCP stream to implement the TLS handshake shenanigans tls-sni-02 calls for?

from cert-manager.

(2/3) Let's break the discussion about a 'class' or some kind of label filter that can be set on cert-manager itself into a standalone issue. We're currently discussing the same thing across two different issues it seems, and you do raise a great point regarding the "one instance watching all namespaces and the other instance watching just one" case.

(4) glad you like it! just so you know, those resources are actually functional today so please do test it out and let me know how it works for you :)

regarding storing secret values in a plain resource - which do you mean specifically? clouddns & cloudflare tokens are ready from a Secret resource (although the struct name doesn't make this particularly clear))

clouddns:
  serviceAccount:
    name: service-account-secret-name
    key: service-account.json

should perhaps be

clouddns:
  serviceAccountSecretRef:
    name: service-account-secret-name
    key: service-account.json

(5) that was my understanding too, hence I've not spent more time on it!! I think it will be particularly difficult to implement if it is possible.

from cert-manager.

(4) I missed that the DNS credentials configuration are Secret refs, we've discussed elsewhere changing 'name' to 'secretName'.

(5) It's a hack/subversion of TLS handshakes, and so complicated, when they could e.g. just connect with unvalidated TLS (e.g. self-signed cert) and use http-01 style content challenges. I think wait for tls-sni-03 :-)

from cert-manager.

So picking this back up...

1. Let's allow the tls-acme annotation on Ingress resources
2. It should be a special annotation that is tightly coupled to the ACME HTTP01 provider only
3. If users want to request a certificate for an ingress using anything other than ACME HTTP01, they should create a Certificate resource for it manually.
4. If the user uses the tls-acme annotation, a corresponding Certificate resource will be automatically created.
   a. How do we choose which Issuer to use for this Certificate?
   b. If we introduce the concept of a default Issuer (ie. with an annotation on an Issuer resource):
   1. What happens if the default changes? Do we update each Certificate resource with the new default, or leave it as is?
   2. What if the default Issuer is not an ACME issuer?
   3. If we allow users to override the default with another annotation on the Ingress resource, again, what happens if the user changes this?
      c. Following from @whereisaaron's feedback, perhaps we could make the annotation's value be the name of an Issuer, eg. kubernetes.io/tls-acme: "letsencrypt-prod". This saves extra fields, but will break backwards compatibility with kube-lego (although this is not a stated goal of cert-manager, so we are okay to do this)

@wallrj @simonswine @mattbates @whereisaaron If you've got any views on 4.a-c, I'd be very appreciative. Once I can get these resolved I can get to work. I'm keen to push ahead with this as it provides an easy-to-understand upgrade path for users of older projects.

from cert-manager.

I'm pushing this out of the first 0.0.1 milestone to give us some more time to work on supporting features (e.g. default issuer config etc).

This is still a high priority item, and will hopefully follow in a 0.0.2 or 0.0.3 at the latest.

from cert-manager.

It should be a special annotation that is tightly coupled to the ACME HTTP01 provider only.

If possible - it would be great if this wasn't an explicit requirement.

We have been using kube-lego for quite a while now, but only with public ELBs because of the HTTP challenge. It has been incredible - the seamless integration with ingress - we've been able to secure almost 100% of our Ingress traffic because of it.

Going forward, we have many clusters that can't be directly connected to by the internet and were planning to try and use the DNS challenge for these internal ELBs.

If we could use the DNS challenge mechanism for these Ingress resources that would be great.

Thanks for all your work.

from cert-manager.

I've better testing with the shim for HTTP-01 this week @munnerz and it works great! I greatly enjoy it not being dependent on a particular ingress controller and supporting namespaced ingress controllers too. Tremendous work @munnerz 👏

I don't know if you can do this already, but it would be helpful to be able to add annotations to the shim Ingress resource, to ensure the (right) ingress controller picks it up.

Supporting the old - should have been deprecated already already - kubernetes.io/tls-acme annotation doesn't excite me 😄 but it does make it easy for users to switch, which is worth it. And that annotation is embedded in lots of Helm charts and examples. So definitely a good way to go.

Sorry I missed you picking up the discussion in August. Regards 4(c) I favor not overloading the one annotation to both, enable/scope the Ingress, and to specify the (cert-manager specific) issuer. I don't think two annotations places an unreasonable burden on the Ingress manifest versus one :-)

kubernetes.io/tls-acme: "true"
certmanager.kubernetes.io/issuer: "letsencrypt-prod"

How does the shim affect/enable Ingresses that specify an issuer with another challenge method like @darron talked about? A shim isn't needed for Ingress if the default or annotated issuer is a DNS or CA or vault issuer is it?

from cert-manager.

Hey @whereisaaron - we've been using the ingress-shim with the DNS challenge method for a little bit now and it's been working great. We are running the shim with:

--default-issuer-name=letsencrypt-prod
--default-issuer-kind=ClusterIssuer
--default-acme-issuer-challenge-type=dns01
--default-acme-issuer-dns01-provider-name=route53

We have a ClusterIssuer that looks like this:

apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v01.api.letsencrypt.org/directory
    email: [email protected]
    privateKeySecretRef:
      name: letsencrypt-prod
    dns01:
      providers:
      - name: route53
        route53:
          region: aws-region-goes-here

We're using kube2iam to give the cert-manager containers IAM access needed for Route53.

Thanks for your work @munnerz - really appreciated.

from cert-manager.

Good combo @darron. You use ingress-shim to create the Certificate, but then dns01 is used to create the secret. That's great. Does the ingress controller update ok? The patching of the ingress for http01 tends to be useful to trigger the ingress controller to pick up and use the new Secret.

from cert-manager.

It seems to take a little longer to create and activate than with kube-lego - but the ingress controller updates OK and it's the cert ends up being active without any user intervention.

It's pretty magical actually.

from cert-manager.

Yeah @darron; kube-lego and cert-manager+http01+ingressClass always update the Ingress after the certificate is issued to 'unpatch' the Ingress That beneficially triggers a reload on the ingress controller so it starts using the newly available certificate immediately. That's why I wondered if there was a delay.

from cert-manager.

@darron / anyone - how are you getting kube2iam to give the cert-manager containers proper route53 IAM credentials? My cert-manager pods are running with the proper iam.amazonaws.com/role: <arn> annotation and everything else seems to be in place, but i'm getting errors like the following:

E0408 17:18:43.892926       1 controller.go:196] certificates controller: Re-queuing item 
"default/nginx-test-certs" due to error processing:
error presenting acme authorization for domain "<domain>": 
error getting route53 secret access key: secret "" not found

I have verified that my kube2iam annotation works for other pods with the same role... it seems cert-manager is not picking up the credentials from the metadata service?

EDIT: i'm running a ClusterIssuer with a similar setup to your comment

from cert-manager.