Linkerd proxy fails to connect to other proxy about linkerd2 HOT 2 OPEN

We enabled DEBUG logs on the client proxy:
https://gist.github.com/andrewdinunzio/e60cd0c6d4cbc9b5c0c1c28a33ad3b6c
(some info redacted)

I captured 1000 lines before any mention of svc-d93b402b4545fe597edf8ba9e49be61ae01973c4 (the problematic server in this case) to 1000 lines after the last mention of it.

These seem suspect:

[  2483.392885s] DEBUG ThreadId(01) outbound:accept{client.addr=10.244.27.147:45346 server.addr=10.0.89.172:80}:proxy{addr=10.0.89.172:80}:http: linkerd_stack::loadshed: Service has become unavailable
[  2483.392892s] DEBUG ThreadId(01) outbound:accept{client.addr=10.244.27.147:45346 server.addr=10.0.89.172:80}:proxy{addr=10.0.89.172:80}:http: linkerd_stack::loadshed: Service shedding load

But it's only one of the client replicas having this issue. Others are sending requests just fine.
Is it just assuming it's due to load if the service is "unavailable"? Or is it the other way around and marking the service unavailable if it's under load. It's strange because I definitely don't expect the server to have many simultaneous requests; I'd expect 5-10 max.

Also, these logs seem like they would be good to have in INFO; it's difficult for us to toggle DEBUG logs on because they log user tokens, etc.

from linkerd2.

I'm afraid I can't give you a lot of guidance here without having a reproducible scenario. We have some docs about failfast and 503 and 504 errors that might help. What's special about the client that is causing these issues? Is it generating too many requests that the resources allocated to the proxy aren't enough? You could also look at the proxy metrics for the problematic pod and see how they compare to the healthy ones.

from linkerd2.