Connection to the gRPC server gets stuck about anycable-go HOT 8 CLOSED

I'll try to reproduce it locally and come back when I find something.

from anycable-go.

I've tried to reproduce it at that simple chat application, but unfortunately (or fortunately) couldn't experience the problem.

from anycable-go.

@palkan Sorry, I can't reproduce it after upgrading from Ubuntu 16.04 to 18.04. It might have been something related to that specific version of Nginx for Ubuntu, or I might have misconfigured something else in Nginx that I have not noticed.

from anycable-go.

Hey @avlazarov! Have you tried to play with grpc_read_timeout configuration? What happens when the default timeout, 60s, passes?

from anycable-go.

I assume that AnycableGo is not aware of you RPC servers went done because nginx still keeps the connection because of the grpc_read_timeout and grpc_send_timeout directives.

from anycable-go.

I assume that AnycableGo is not aware of you RPC servers went done

As I understood, other clients (new connections) work fine, i.e., gRPC connectivity is restored.

The problem is that the first one, the one that "caught" the broken connection, is getting stuck:

Bottom line is that performing actions on a subscription after getting error 502 blocks all new actions from being performed by the anycable-go server for a particular client/subscription.

@avlazarov Right?

And that's strange: if other clients could successfully perform an action, the first one should do this as well on the next attempt, since they uses the same grpc pool.

from anycable-go.

@palkan Yes, the odd part is that even when the next client makes a series of successful actions, the first one remains stuck. If instead of error 502 I totally shutdown nginx (causing refused connection), anycable-go will perform the operations, print errors but once nginx is back again, the gRPC servers will correctly receive the actions and the client will no longer be stuck.

from anycable-go.

@avlazarov Please, take a look the @bibendi 's PR above. We couldn't reproduce the problem. Are we missing something?

from anycable-go.