NSM interface deleting periodically about deployments-k8s HOT 12 CLOSED

Tested on kind. The issue is reproducible.

from deployments-k8s.

Logs from my local running:

nsc-init.txt
nsc.txt

from deployments-k8s.

@edwarnicke Can we consider this issue ASAP?

from deployments-k8s.

@denis-tingaikin Yes

from deployments-k8s.

Root cause:

1. Spire gives a certificate for 1h
2. NSM schedules refresh after 1h * 1/3
3. On refreshing spire updates certificates for all applications
4. At refresh request moment nsmgr has a new certificate from spire, but authInfo from gRPC keeps the old certificate from step1.
5. nsmgr updates token with new certificate
6. client can not validate the token from nsmgr because authInfo from gRPC keeps the old certificate from step1. (failure here)

Currently, I'm not found a good solution for this issue, started to look into gRPC source code.

from deployments-k8s.

Tested today two workarounds:

1. https://golang.org/pkg/crypto/tls/#RenegotiationSupport -- it is not helped
2. remove connection caching in connect -- it is working

Still looking for other solutions.

from deployments-k8s.

@edwarnicke

I've asked spire guys about the issue and got the next answer:

Andrew Harding 14 hours ago
This is expected. gRPC will reuse the existing connection when you issue RPCs unless you redial. Since no new TLS handshake takes place, the new client credential is never communicated.
white_check_mark
eyes
raised_hands

Andrew Harding 14 hours ago
The x509source returns a channel from Updated() that callers can use to know when the SVID has been updated so they can re-establish a connection with the new credential.

Question: Can we modify connect chain elements to wait for update SVID to make re-dial?

Note: we can just pass option to wait to channel to not depend on spire functions

from deployments-k8s.

Currently we have the next options to fix the issue:

1. Do redial as suggested spire guys on svid updating #1929 (comment)
2. Remove policy last token signed.
3. Keep and use first certificates for client and server on token generating.
4. Your option.

For me option 1 looks good.

@edwarnicke Please share your thoughts on these options.

from deployments-k8s.

I'm curious... are they saying that GRPC won't close existing connections that have a TLS certificate that has expired after the connection was established?

from deployments-k8s.

Yes, as I got it, a handshake is doing once per dial.

from deployments-k8s.

@denis-tingaikin It looks like we need to do something that involves option 1 above... but lets try to keep it simple and natural :)

from deployments-k8s.

The root cause is fixed in networkservicemesh/sdk#1005

But I found that the issue can be reproduced via unstable healing. This reproducing periodically.
Tested a fix networkservicemesh/sdk#1005 without heal and it working fine in 100% cases.

from deployments-k8s.