Sometimes heal continue working indefinitely about deployments-k8s HOT 5 CLOSED

Here are logs from another test run: nsmgr-2021-07-02T16.36.45+07.00.zip (20 MB)
This time the issue occurred after first run of a test with 50 requests.

from deployments-k8s.

I think I have found the root cause of the issue.

Let's look at our heal:
When heal server is requested to do heal, it does it on a heal context.
This context doesn't have any timeout, only cancel.
So heal either succeeds, or is stopped manually, or works indefinitely.

Usually, client sends Close, heal receives it, stops heal and doesn't accept new heal requests for this connection.
However, in my tests sometimes the following was happening:

1. Heal starts and blocks serialize element for this connection
2. nsmgr receives close, but it is blocked by serialize
3. Heal attempt ends, close starts propagating, but heal server doesn't receive it, so it doesn't know it has to stop.

The culprit here is discover server: it tries to find alive endpoints. Usually right after we delete the endpoint, the registry still holds its record, and everything works.
In my tests I was sometimes getting long enough delay for the endpoint record in registry to expire, discover doesn't find NSE in registry, and then it just doesn't propagate close.

Here are logs of this happening: logs.zip
There is a bunch of connections in these logs. You can can use grep 2197eb15-19ce-49e8-ad98-d43db51896c2 to get the part of the logs that allows you to see the issue.

from deployments-k8s.

This only happened to me when I created ~50 connections (or more), and nsmgr was CPU limited. Also when I modified the logs in some way, it suddenly stopped happening (I wasn't able to figure out which exactly of my modifications affected it).

from deployments-k8s.

Looks like this issue is reproducible only in lack of resources. Even then, if client fails to close the connection until it cancels, heal will be closed by local NSMgr timeout.

There are 2 cases how can we met this issue:

1. Close doesn't reach NSMgr until the NSC Close context timeout happens - #2085.
2. Close reaches NSMgr, but it fails to reach all hops in the chain until the NSC Close context timeout happens.

In [1] case there is nothing to do - NSMgr doesn't know anything about this Close, so timeout here is the only way.
In [2] case we can create additional chain element to capture the Close and restart it with the NSMgr context.

@denis-tingaikin
What do you think about the [2]? Do we need it or should we think about both [1, 2] cases in the same way?

from deployments-k8s.

Made an investigation of NSM work on high load (networkservicemesh/sdk#1031), so here are additional 2 issues for the [2] case:

1. networkservicemesh/sdk#1028
2. networkservicemesh/sdk#1026

from deployments-k8s.