May need to be more explicit about service account secret creation about router HOT 5 CLOSED

@krancour - I think this could be caused by the discussion in kubernetes/kubernetes#12186. I don't have certificates on the api or the controller-manager. Can you confirm that you have this in your setup?

from router.

I do, but I'm not the original reporter of the problem. See deis/workflow#314. This is not to say that other issues may still be lurking, but in this case, I know for certain that this is a result of a missing ServiceAccount admission control. The simple resolution is to manually, explicitly, generate the secrets that will be used by the service account and then it won't matter if that admission control happens to not be in play.

from router.

I do, but I'm not the original reporter of the problem.

@lachie83, I owe you an apology. Somehow I failed to recognize that you were the original reporter of the problem in the deis/workflow#314 issue that I referenced... If I had made that connection, I might have realized that my initial advice re: admission controls hadn't resolved deis/workflow#314 and that there was more to this issue than meets the eye. So again... sorry.

I realized all of this while attempting to reproduce your original issue this morning. After about a dozen failed attempts to do so, I re-read the kubernetes issue you referenced and found it enlightening. So, thank you. That thread is a bit protracted, so I'll summarize here for the benefit of any other Deis user who runs into this...

To start, no option on the apiserver has any bearing on reproducing this issue. That is to say, no matter what SSL related config I strip out of my apiserver, I cannot get service accounts to be created without an associated token-bearing secret. This is not to say that these apiserver configs do not matter. They certainly matter very much, but getting them wrong won't reproduce this specific issue.

What will reproduce the issue is dropping SSL related config (specifically --root-ca-file and --service-account-private-key-file) from the controller-manager. These are used to generate and sign tokens.

Now, all that being said... go back to the apiserver and if that's not configured correctly, then the tokens generated by the controller manager won't be of any use because the apiserver won't recognize them.

On a related note, this got me curious about what happens when apiserver exposes only the insecure endpoint. In such cases, service accounts and their associated tokens should be irrelevant. As it turns out, however, most Deis components use the NewInCluster() function of the k8s.io/kubernetes/pkg/client/unversioned package. This function seems to assume the use of a secure endpoint and expects a token to be available. If it's not found it fails. I'll open a ticket on that against kubernetes/kubernetes, because it seems that could be improved upon.

Bottom line, however, is that on multiple fronts, all SSL-related options must be configured correctly in a k8s cluster or else Deis won't work. Most k8s distributions I am aware of should adequately address this out-of-the-box, so it's likely just "custom" clusters that might run into such problems.

I'm going to close this issue because there isn't much that can be done on the Deis end to insulate the platform from failures that result from an improperly configured cluster.

from router.

fwiw...

most Deis components use the NewInCluster() function of the k8s.io/kubernetes/pkg/client/unversioned package. This function seems to assume the use of a secure endpoint and expects a token to be available. If it's not found it fails. I'll open a ticket on that against kubernetes/kubernetes, because it seems that could be improved upon.

Researching the matter more, the secure endpoint and the requirement for client authentication on that endpoint can never be fully disabled. If not explicitly configured, the secure endpoint will bind to port 6443 on 0.0.0.0, secured with a self-generated certificate, and will use client certificate authentication.

So my imaginative scenario where an SA and its secret wouldn't be relevant was invalid.

This reinforces the bottom line that SSL, auth, etc. must be configured properly in a k8s cluster as a prerequisite for Deis.

from router.

@krancour - Thank you very much for the follow up. Understood. We were an early adopter and didn't have SSL ironed out. I'll roll these changes and try again. In the meantime I tried in GKE without any issues. Thanks again

from router.