NSC, Error from monitorConnection, RPC error code: PermissionDenied, Desc: no sufficient privileges about deployments-k8s HOT 9 OPEN

@denis-tingaikin

Unfortunately, setting the LIVENESS to false did not seem to help:
nsc.log
nse.log

Did ping work for your scenario?

At the NSC, cannot get to a point to try anything because of the error.

Do you have a diagram/scheme/proposal what do you finally get?

This is the test scenario:

And this would be an example of building random topologies using these universal nodes:

from deployments-k8s.

In the provided test results, (unintentionally) both the nsc and the nse nodes provide the "icmp-responder" service (that was originating from the default Config). We corrected this after the logs were captured. Same result. Still get the error.
Also in the actual topology configuration there are no redundant service offers. Although each node provide and consume services, each service name is unique, based on the node id.

from deployments-k8s.

/cc @glazychev-art , @anastasia-malysheva May this be related to monitor OPA staff?

from deployments-k8s.

Thanks for the detailed information!

Most likely yes, it is related to OPA for monitoring. But it is not actually an error if you are not using init-container (cmd-nsc-init).
We probably need to rewrite the error message in order not to mislead anyone.

from deployments-k8s.

@glazychev-art Any idea what the root cause might be? I'm not entirely sure why we would be seeing this, do you have a more specific idea?

from deployments-k8s.

@edwarnicke
cmd-nsc does Monitor connections before the Request. This is necessary to take the connection if there was cmd-nsc-init container before.
But as you know, you've implemented an open-policy for monitoring, and it is based on the spiffieID from the Request. But if we did not have an init container, then there were no Requests either. And an authorization error is returned.

But here the problem is different - as I see from the logs, there are many healing errors:

Aug 15 22:00:44.223�[33m [WARN] [id:nsc-858c5dc57-2bf6l-0] [heal:eventLoop] [type:networkService] �[0m(7.1)         Data plane is down
Aug 15 22:00:44.223�[37m [DEBU] [id:nsc-858c5dc57-2bf6l-0] [heal:eventLoop] [type:networkService] �[0m(7.2)         Reconnect with reselect

need to figure out why this is happening

from deployments-k8s.

That's an interesting scenario where we're trying to run nsc/nse together in the same container.

@bilgehan-erman

1. Do you have a diagram/scheme/proposal what do you finally get?
2. Did ping work for your scenario?
3. I think that the problem with authz is related to dp healing. Could you re-test the setup with disabled dp heal https://github.com/networkservicemesh/cmd-nsc/blob/main/internal/config/config.go#L49? (means set env NSM_ LIVENESS_CHECK_ENABLED=false)

from deployments-k8s.

@bilgehan-erman
Sorry to keep you waiting

I looked at your setup and logs, and I think I understand what's going on.
The main reason is that the NSC is trying to connect to itself (to its endpoint). By the way, I'm not sure that this is possible due to routing...
But you need a different scenario. I think selectors can help you with this.
I prepared an example according to your last picture:

1. nsc1 ---> nsc2
2. nsc1 ---> nsc3
3. nsc2 ---> nsc3

When I say nsc I mean nsc+nse on the same pod (like your "node").
I did not make new image, I just added the client and endpoint as different containers in one pod. Most likely yamls can be simplified, I just want to show the idea:

1. We can separately declare a network service and specify selectors there (netsvc.yaml).
2. We have 3 pods where we label NSEs (NSM_LABELS: "dst_endpoint:node*")
3. And also specify labels on the NSC, saying who we want to connect to (for example kernel://icmp-responder/nsm-1-2?dst_endpoint=node2). Due to the selectors and labels, we will be able to select the desired endploint.

So, to try you need:

1. Deploy spire
2. Deploy basic NSM
3. kubectl create ns ns-topology
4. Apply kustomize file from nsc_nse_setup.zip

nsc_nse_setup.zip
(I used the main branch, but I think it will work on 1.5.0 too)

I really hope this helps!

from deployments-k8s.

@glazychev-art thank you very much for looking into this.
I'll try it out your suggestions and see how it goes.

from deployments-k8s.