OKD 3.11 - Maistra 0.12 : Automatic sidecard injection doesn't work about istio HOT 31 CLOSED

Hello Brian,
This is what I would like to highlight here. The installation is a "all-in-one" installation with one master and one node on the same virtual machine according to the documentation https://docs.okd.io/latest/install/index.html#single-master-single-box . For this configuration, I extract the archive tar.gz in a folder and I executed the command "cluster up -- based-dir= …". This helps me to have multiple flavors of the installation seeing that the core stays the same (okd, registry, …). In this approach, you have three folders inside the folder defined in the base-dir:
./kube-apiserver/master-config.yaml
./openshift-apiserver/master-config.yaml
./openshift-controller-manager/master-config.yaml
So, three master-config.yaml which is not located in the /etc/ directory (However, in the container itself the configuration file master-config.yaml is in). This explains my doubt … :-) : Which master-config.yaml file to change… I made a few tests and it's enough to modify the kube-apiserver only because this is the foundation… what I understood when I read the blog deep dive. So, if you change this master-config.yaml file, the automatic sidecar injection works.
Olivier

from istio.

Hey @opotmans,
you didn't include the manifest you used to deploy sleep, so I have to ask: did you add the automatic injection annotation? I understand it might be a bit misleading that we didn't update the sleep.yaml in this repository.

Best,
Daniel

PS: we use JIRA for issues, if this turns out to be a bug, it should go there.

from istio.

Yes i do it i'm on road now. then i have a pause i attach the file

from istio.

@opotmans In the textfile you attached, I see:
image: "docker.io/maistra/proxy-init-ubi8:0.12.0"

This proxy-init image is based on nftables and does not work on RHEL7/OCP3.11, which is based on iptables. The iptables one is: maistra/proxy-init-centos7

Can you add:

proxy_init: image: proxy-init-centos7

to your SMCP? Without it, the pod won't be configured to go through the sidecar proxy.

from istio.

Just rereading your comments. I might be getting a bit ahead of myself (although the wrong proxy-init image will still stop it from working). Is the problem that the automatic injection isn't working or that the sidecar gets injected but doesn't have any logs/doesn't show up on Kiali/no Istio configs work or that the pod itself isn't being injected (i.e. oc get pods -n sleepNS shows 1/1 instead of 1/2)? If it's the former, then my suggestion above will likely fix your issue.

from istio.

Hello,
thanks for your info
here is the file as requested.
sleep-inject.txt
with the inject in the annotation
Effectively you are right, the proxy init must be centos (about iptables)
I have adapted the config map of the istio-sidecar-injector

and here is the result, only one container in the pod. So 1/1

Based on the debug logs of the istio-sidecar-injector, i don't see the activation of the installation of the proxy init...

thanks for your help!

Olivier

from istio.

additional remark: I adapted the smcp according the documentation https://maistra.io/docs/troubleshooting/troubleshooting_sidecar_init/

[...]
global:
proxy:
proxy_init:
image: proxy-init-centos7
resources:
limits:
cpu: 500m
memory: 128Mi
requests:
cpu: 100m
memory: 128Mi
However, when I applied the file, the config_map of the container sidecar-injector is not updated,
I manually updated the image by putting:
policy: disabled
template: |-
rewriteAppHTTPProbe: false
initContainers:
[[ if ne (annotation .ObjectMeta sidecar.istio.io/interceptionMode .ProxyConfig.InterceptionMode) "NONE" ]]

• name: istio-init
  image: "docker.io/maistra/proxy-init-centos7:0.12.0"
  args:
  according to I found on docker hub

from istio.

I suspect the issue here is that 0.12 is configured to install using multitenant mode by default, which means you need to add your namespace to the ServiceMeshMemberRoll.spec.members field in order for it to become part of the mesh. You can add ServiceMeshControlPlane.spec.istio.global.multitenant=false to your CR, but you should now that multitenant will be the only install type in 1.0.

Also, for injection to work properly, you will need to add the namespace to the SMMR resource prior to creating your deployments/pods.

FWIW, I'd recommend against using default namespace for user applications.

from istio.

Whoops, should have read the initial description more closely.

from istio.

Verify that maistra.io/member-of label exists on default namespace and that its value matches the namespace within which the istio control plane is installed. The mutating webhook used by the sidecar injector uses that label as its selector.

Also, if you're installing the control plane using one of the example CRs, please note that for 0.12 they include an SMMR resource.

from istio.

Hello,
thanks all for your advices.
@rcernich
I modified the smmr resource to add istio-system in the member.
oc get smmr
NAME MEMBERS
default [istio-system bookinfo default]
when I deployed sleep (annotations: sidecar.istio.io/inject: "true") this doesn't work
sleep-86f44c5b-jvtm4 1/1 Running 0 7s

@brian-avery
I tried to modify the minimal-install by adding this in the global parameters
spec:
istio:
global:
proxy:
proxy_init:
image: proxy-init-centos
but the config_map istio-sidecar-injector mapped to /etc/istio/inject doesn't reflect this modification and keep the image "docker.io/maistra/proxy-init-ubi8:0.12.0"
so I edited the config map to propose the image : "docker.io/maistra/proxy-init-centos7:0.12.0" and restart the container. If I go in the container I see in the /etc/istio/inject the correct information now
However, the test "sleep" doesn't work no automatic injection
For information the sa account of the default namespace has scc privileged and anyuid defined

I would like to propose this analysis...
when I activate the debug level add parameter --log_output_level=default:debug (found on https://bani.com.br/2018/09/istio-sidecar-injection-enabling-automatic-injection-adding-exceptions-and-debugging/) of the sidecar, I don't see any action on the sidecar pod when I deploy sleep, so this should mean that the policy of mutatingwebhook is not "activated"..
If I analyse istio-sidecar-injector-istio-system configuration:
oc get mutatingwebhookconfiguration istio-sidecar-injector-istio-system -o yaml
I see:
I also a question about namespaceSelector
namespaceSelector:
matchExpressions:
- key: maistra.io/member-of
operator: In
values:
- istio-system
- key: maistra.io/ignore-namespace
operator: DoesNotExist
- key: istio.openshift.com/ignore-namespace
operator: DoesNotExist

I don't see any relation with the annotation that we need to put in the deployment template sleep
sidecar.istio.io/inject. This is only related to if istio-system is member-of... am i wrong in my analysis ?

I'm observed the same behaviour in maistra 0.11 and I think that in maistra 0.6, the sidecar.istio.io was in the configuration of the mutatingwebhook...

furthermore in the config_map of the sidecar-injection pod called istio-sidecar-injector which is mapped to /etc/istio/inject/config I see policy disabled means that this needs to be overcome..
policy: disabled
template: |-
rewriteAppHTTPProbe: false
initContainers:
[[ if ne (annotation .ObjectMeta sidecar.istio.io/interceptionMode .ProxyConfig.InterceptionMode) "NONE" ]]

• name: istio-init
  image: "docker.io/maistra/proxy-init-centos7:0.12.0"

Thanks for your help

Kind Regards

Olivier

Additional point: I observed in the kiali that only the namespace istio-system is visible. The other namespaces are not visible in kiali... In the maistra 0.11 all the namespaces were visible

from istio.

Can you post your operator logs? I'm wondering if you're running into this: https://issues.jboss.org/browse/MAISTRA-621?_sscc=t

from istio.

In reference to the proxy-init content, it should be:

istio: global: multitenant: false proxy_init: image: proxy-init-rhel7

In your comment above, it's under istio->global->proxy->proxy_init.

In regards to the sidecar.istio.io/inject annotation, we check that internally for all pods with the memberof label here: https://github.com/Maistra/istio-operator/blob/maistra-1.0/pkg/controller/servicemesh/memberroll/controller.go#L884

from istio.

I'd verify that the istio-sidecar mutating webhook is installed and that webhooks are enabled for your okd install, in addition to looking for errors in the operator log.

from istio.

I'm facing the same problem with bookinfo app. Here's the gist of what I have gathered so far https://gist.github.com/bartoszmajsak/42922bdd05553004be96afde6f23d41f. If you need more please let me know.

from istio.

@opotmans, the namespace selector ensures that the webhook only applies to namespaces with those labels, so the namespace has to have maistra.io/member-of=istio-system and neither maistra.io/ignore-namespace or istio.openshift.com/ignore-namespace (i.e. if it has either of the last two, injection will be skipped). Verify that the namespace you need injection to work on has maistra.io/member-of=istio-system (assuming your control plane is installed in istio-system). The smmr controller will add this label for all namespaces in members.

from istio.

@brian-avery
operator_logs.txt
Tell me if you need more.
@rcernich
What do you expect to support you ? In the file attached troubleshooting.txt, I put a lot of information to troubleshoot the problem. You have all the aspects about the istio mutating webhook. Do you have specific commands that you need that I execute?
thanks for your help
Olivier

from istio.

The errors in the operator log would indicate that the member namespaces are not being configured. This is most likely the problem, as the labels won't be added to those namespaces. Also, the role bindings won't be created, which will cause other features to not work properly (citadel, galley, injection).

from istio.

ok thanks for the information. So, what I need to do to correct it ... Which label do I need to add ?
thanks!
olivier

from istio.

@opotmans, adding the label might work (maistra.io/member-of: istio-system), but without the role bindings, galley won't be able to read istio resources in the namespace and citiadel won't be able to create certs/secrets.

I'll have to do some investigating to see why there is no networking configuration in OKD. I've been doing all my testing with OCP. (The SMMR controller uses this resource to determine which ovs plugin is being used by the cluster so it can configure networking for the projects, e.g. oc adm pod-network join-projects or adding NetworkPolicy resources to ensure member namespaces can communicate with each other and with the mesh.)

from istio.

@opotmans I managed to get that working and I believe there are two things which you should change in your scripts.

• master-config.yaml should be patched in openshift.local.clusterup/kube-apiserver only
• your installation CR should have following bit in place:

apiVersion: maistra.io/v1
kind: ServiceMeshControlPlane
metadata:
  name: basic-install
spec:

  istio:
    global:

      multitenant: false 

      proxy_init:
        image: proxy-init-centos7

  ...

Here's the shell script I used https://git.io/fjyma

from istio.

@opotmans I updated the script so you don't have to restart the cluster in order to have admission hooks enabled. You can give it a spin.

from istio.

Dear all,

Firstly again thanks to support me in this troubleshooting

@rcernich, I check for the command oc adm pod-network join-projects. However, this command is only possible if ovs is activated on the environment. The okd is installed "all-in-one" on a virtual machine (this is a demo on a laptop).

here is the result of the command.

oc adm pod-network make-projects-global istio-system

error: managing pod network is only supported for openshift multitenant network plugin

@bartoszmajsak

Thanks for your script. the multitenant: false helps me to progress... I saw 0 error at the level of the istio-operator.

oc logs istio-operator-6f7488cccc-7g4rc | grep error give not result :-)

However, the sidecar injection doesn't work with bookinfo

just for your information, i changed a little bit the script because I use base-dir and routing-suffix as argument. the base dir enables me to have different flavours of okd to start... Hope that this doesn't have any impact the behaviour of okd/istio

I observe for the activation of the admission and mutating webhook option that the modification for the master-config.yaml for the openshit api and controllers is not implemented if I change the kube-apiserver/master-config.yaml in the openshift.local.clusterup folder
So when the cluster up is installed from scratch, I patched the master-config yaml in the directory openshift-apiserver and openshift-controller-manager and restart the services.
These are the two services that you need to restart according to the document maistra
https://maistra.io/docs/getting_started/install/#_updating_the_master

It seems that this is activated

oc api-resources | grep webhook

mutatingwebhookconfigurations admissionregistration.k8s.io false MutatingWebhookConfiguration

validatingwebhookconfigurations admissionregistration.k8s.io false ValidatingWebhookConfiguration

Unfortunately, the automatic sidecar injection for the application bookinfo doesn't work...

So I started a troubleshooting to understand where the problem is (divide and conquer approach)

If I'm not wrong, (please comment this), here is the process for the automatic sidecar injector:

When a pod is created, an api request is sent to the api server . This api request passes through the api request lifecycle (Api HTTP Handler, Authentication/Authorization, Mutating admission controller). At the mutating admission controllers, there is a mutating admission webhook, this webhook is configured by the mutatingwebhookconfiguration policy (coming from the script proposed by @…)

This policy proposes to send a request to the service istio-sidecar-injector (ip address: 172.30.159.197 in my case; linked to the ip address of the pod 172.17.0.18) on the folder /inject. /inject is mapped to the config map istio-sidecar_injector which is (file istio-sidecar-injector) if and only if the following conditions are agreed :

namespaceSelector: 

  matchExpressions: 

  - key: maistra.io/ignore-namespace 

    operator: DoesNotExist 

  - key: istio.openshift.com/ignore-namespace 

    operator: DoesNotExist 

rules: 

- apiGroups: 

  - "" 

  apiVersions: 

  - v1 

  operations: 

  - CREATE 

  resources: 

  - pods 

The /inject correspond to the sidecar to inject in the container.

To validate if this process works, I enabled at the level of the istio-sidecar-injector pod the debug and the stack trace mode by adding the following arguments: - '--log_output_level=default:debug' and - '--log_stacktrace_level=default:debug' in the yaml deployment file of this pod.

Here is the result file when I made an oc create –f sleep_autoinject.yaml in the default namespace.
1st try
trace_stack_sidecar_injector.txt

Full trace including a second try...
logs_istio_sidecar_stacktrace_debug.txt

With this information, we can follow the different codes calling... I can't go deeper, I will need time to reverse analyse the code.

I also activate a network sniffing on all the interfaces (docker included) .
1st try
Here is the result file:
trace_network_injector.txt

2nd try (capture network based on port 443.)
sniffing.txt

Observations:
we see that the api server and istio-sidecar-injector pod communicates but it seems that this is only ACK packet for TCP keep alive the second try we have https communication initiatiated by api..;

oc get svc --all-namespaces | grep 172.30.0.1

default kubernetes ClusterIP 172.30.0.1 443/TCP 3d

192.168.122.138 is the ip of the VM and is assigned to openshift-apiserver-pod

May I conclude that (there is no interaction or doesn't work) between the apiserver and the istio injector? If yes may I conclude that the mutatingwebhook is not activated because the condition is not true or this is not well configured on the apiserver?

However, oc get mutatingwebhookconfiguration istio-sidecar-injector-istio-system -o yaml works and the two api ressources mutating and validating are activated.

How to check that and validate that the webhook is really activated?

When you read the stack trace, the first line says :
2019-07-28T09:20:16.981210Z debug istio.io/istio/pilot/pkg/kube/inject/config.go:65: Watch close - *v1beta1.MutatingWebhookConfiguration total 0 items received

Is it normal? do I conclude that the request to the /inject of the Istio-side-car service is not executed?

Other observation: maybe useless but difference which could have importance seeing that the connection with the istio-sidecar-injector service is in https. If I compare the configmap and the configuration of the webhook, the caBundle is missing in the configmap. Is it normal ?

oc get mutatingwebhookconfiguration istio-sidecar-injector-istio-system -o yaml | grep caBundle

caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMzakNDQWNhZ0F3SUJBZ0lSQU14RnorblJvSHc5NWF5S1V0QUFkcmd3RFFZSktvWklodmNOQVFFTEJRQXcKR0RFV01CUUdBMVVFQ2hNTlkyeDFjM1JsY2k1c2IyTmhiREFlRncweE9UQTNNamN4TWpVME16TmFGdzB5T1RBMwpNalF4TWpVME16TmFNQmd4RmpBVUJnTlZCQW9URFdOc2RYTjBaWEl1Ykc5allXd3dnZ0VpTUEwR0NTcUdTSWIzCkRRRUJBUVVBQTRJQkR3QXdnZ0VLQW9JQkFRRExPUGQyaHhJNDNYMUxWWXJsZTh5YUI3ajVEZEVPYVo4QkU5aTMKRmUrTGYwcjhpRlFBRG9OOGY0aWNlSGlsRWRqblhpVytuRzNYOVFNU3lYdDNTY1dxOFIvWDltNFVvczh2aG5keQpRTktSNUJ2RjhlajFwVW9YdE1SejFyd1dsTDY0eTB2bXRIYmJVeEphNjlMYzk0MSt1bFZtTVpNKzVUdVVqT3dwCk0ra1h3MEIwWWsyWHBjdGFaOUh3T3NINDZvdGRITnk4dmxrelV2Ylhpb3BTbEFYSHlLY0dURE9oMi9MVWptZHAKL0ljY2pNSmwxUURSQ0dCVWhWOU1JYmxUaTNFekttSGVvU2NCYVdYTGgxTWRVRkl0R0F0dlZJdXBsUzAvMDJ1NApWbVlySEpCbUZkb3R2NHV2bWI0ajdnR1o5eGZieUFDTHd2Z3FoWVBSTjhTeFpRdGRBZ01CQUFHakl6QWhNQTRHCkExVWREd0VCL3dRRUF3SUNCREFQQmdOVkhSTUJBZjhFQlRBREFRSC9NQTBHQ1NxR1NJYjNEUUVCQ3dVQUE0SUIKQVFCd0pXdlFpSXRqRGE0OWUyaDI4WEY1RlNvc2RRMG5qOVJySzgwMEZ6QjRLaDZuNmdNSDFGNFZwRHZxNlFRYgpOcENDRWhCWVBKVVM3RWx0SFU1S3hPZ1hXYU5zRkNyYm8xaE5LQjJieE1tNXQ1Y29rRitGaHQ2NDJsUFBERm1sCjVac0ROc21sYlhUWXExRG9zZjNnSCsrZVNNME1IUHhDbnVSWWVWSDVzalN1SmFoZlNqaXlvSjVKMWNIZHRPemkKeE9yWTJlaGZlVXNvbE8wVEVra1BBSmVBQ2ZqUXZxdlp0MDN2L0tNUmZMVkE1RUcyekxsNnI3aUVXYng5SUY1RwpJLy9yRVdHdlU1blU5RmVvRDZVdW5DQmZpa3l4NkJZYU1UOU1NaFA3VWE2UElncWRLUkk4MXNCY0p5TWRDUkdICjh0bytsYjVPdnJDc2hRVjBPTXRtbEo2LwotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg== 

oc get configmap istio-sidecar-webhook-configuration -o yaml | grep caBundle

      caBundle: "" 

  {"apiVersion":"v1","data":{"mutatingwebhookconfiguration.yaml":"apiVersion: admissionregistration.k8s.io/v1beta1\nkind: MutatingWebhookConfiguration\nmetadata:\n  name: istio-sidecar-injector-istio-system\n  labels:\n    app: sidecarInjectorWebhook\n    chart: sidecarInjectorWebhook\n    heritage: Tiller\n    maistra-version: 0.12.0\n    release: istio\nwebhooks:\n  - name: sidecar-injector.istio.io\n    clientConfig:\n      service:\n        name: istio-sidecar-injector\n        namespace: istio-system\n        path: \"/inject\"\n      caBundle: \"\"\n    rules:\n      - operations: [ \"CREATE\" ]\n        apiGroups: [\"\"]\n        apiVersions: [\"v1\"]\n        resources: [\"pods\"]\n    failurePolicy: Fail\n    namespaceSelector:\n      matchExpressions:\n      - key: maistra.io/ignore-namespace\n        operator: DoesNotExist\n      - key: istio.openshift.com/ignore-namespace\n        operator: DoesNotExist"},"kind":"ConfigMap","metadata":{"annotations":{"maistra.io/mesh-generation":"1"},"labels":{"app":"sidecarInjectorWebhook","chart":"sidecarInjectorWebhook","heritage":"Tiller","istio":"sidecar-injector","maistra-version":"0.12.0","maistra.io/owner":"istio-system","release":"istio"},"name":"istio-sidecar-webhook-configuration","namespace":"istio-system","ownerReferences":[{"apiVersion":"maistra.io/v1","blockOwnerDeletion":true,"controller":true,"kind":"ServiceMeshControlPlane","name":"basic-install","uid":"a53b69c6-b06d-11e9-bff9-525400052835"}]}} 

This configmap is mapped to the directory /etc/istio/webhook

I also check the directory /etc/istio/certs which contains three keys. But nothing compliant with the CA bundle.

Hope that this is not stupid questions, I'm not a specialist just learn on the fly :-) , i tried to detect in the different trace if the behavior matches the theory... If you have any question or need more data/test please tell me...

Olivier

from istio.

@rcernich, I check for the command oc adm pod-network join-projects. However, this command is only possible if ovs is activated on the environment. The okd is installed "all-in-one" on a virtual machine (this is a demo on a laptop).

Sorry, I didn't mean to confuse things. There was an error related to getting the clusternetwork resource in your operator log. This error would prevent the namespaces in the ServiceMeshMemberRoll from being configured correctly. The pod-network bit was given as an example of one of the things the SMMR reconciler does when configuring a member project. It only does that if the network is configured to use ovs-multitenant, which it determines from the clusternetwork resource (which it's failing to get). Hope that helps to clarify things.

from istio.

@opotmans I think only kube-apiserver needs patching, the same way as it was needed for enabling knative prior to the creation of the operator [1]. Where did you find information about patching other yamls?

I updated the gist with the latest code. In the comments section you can find a screencast with the demo of starting up oc cluster with maistra from scratch using that script.

Could you share your scripts so I can understand where do we differ?

[1] https://knative.dev/v0.4-docs/install/knative-with-openshift/

from istio.

Hello all,
Thanks for your answers... really appreciated
@rcernich no problem, my comment was just to precise my environment. It's clear that my future test will be multi environments :-)
@bartoszmajsak
I would like to inform you that your analysis is right....
To explain the story, I was badly influenced by the maistra documentation...
https://maistra.io/docs/getting_started/install/#_preparing_the_installation.
In the preparation, the document said that we need to modify the master configuration on each master to include support for webhooks and signing of Certificate Signing Requests (CSRs). However, in the configuration, you have three master-config.yaml one for master
find . -name "master-config.yaml" in the base-dir folder.
./kube-apiserver/master-config.yaml
./openshift-apiserver/master-config.yaml
./openshift-controller-manager/master-config.yaml
and after the documentation asks to restart the master (api and controller)
master-restart api
master-restart controllers
So I concluded that this was for openshift-apiserver and openshift-controller seeing that the pod for kube-api is called master-api-localhost
So, to close the understanding gap, I investigate, this morning, the difference between openshift-api and kube-api https://blog.openshift.com/openshift-v3-deep-dive-docker-kubernetes/
and this is more relevant to only patch the master-api-localhost which is related to the folder ./kube-apiserver, the openshift-api seems to be the above layer (used by oc).
So, I made a reinstallation with base-dir approach by patching only the master.yaml in the folder kube-api server and the sidecar injection works.... This was compliant with my observation Sunday that no rule was applied...
Attention point: as said by rcernich, you need to use the init proxy "centos minded" otherwise, doesn't work (the traffic doesn't pass to the envoy proxy; validated) seeing that the new version of Redhat doesn't use anymore the IPtables but nftables...
So thanks you all very much for your support... we can close the incident... I will write a blog post with the complete info ... Keep you in touch.
I also recommend to adapt the documentation of the site of maistra... if you need help tell me. Olivier

from istio.

Thanks for the feedback @opotmans. Looking at the OKD documentation, (https://docs.openshift.com/container-platform/3.11/install_config/master_node_configuration.html) there's no mention of the 3 files you mentioned above, only the /etc/origin/master/master-config.yaml file discussed in the Maistra docs.

The three files you mentioned seem to be specific to oc cluster up, which we no longer work with Maistra as of TP12.

from istio.

Hello people
What documentation can I follow to install Maistra correctly?
I have okd 3.11, CentOS Linux 7
https://maistra.io/docs/getting_started/install/#_preparing_the_installation it does not work

from istio.

Hello
If this is the link which doesn't work, please use that one:
https://maistra.io/docs/installation/
If this is your installation which doesn't work, feel free to ask the question
Olivier

from istio.

Thanks @opotmans , works when I change kube-apiserver.

But in RedHat documentation don't inform what master-config.yaml is to change
(In my minishift installation has 3 files with this name)

from istio.

Closing this as it seems to have been resolved

from istio.