Ongoing connection reset by peer about reactor-netty HOT 6 CLOSED

@AbhiramDwivedi Have you checked https://projectreactor.io/docs/netty/release/reference/index.html#faq.connection-closed, especially the part where a Network Component drops a connection silently.
If you have checked that, please provide the TCP dump.
You might be interested in checking this https://medium.com/tenable-techblog/lessons-from-aws-nlb-timeouts-5028a8f65dda (in case you use AWS NLB) and this https://youtu.be/O4oZS-SAq14?t=526

from reactor-netty.

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

from reactor-netty.

Hi @violetagg : Those links were quite useful in understanding the TCP settings. We tried with pretty aggressive settings and that did not help. This is almost solved with TCP changes on target cluster, with following changes:

• keep-alive reduced from 3000 to 300
• keep-alive-requests increased from 100 to 1000
• upstream-keepalive-timeout increased from 60 to 300
  However, we still run into it some times, and there is no real consistent way of reproducing or solving this.

A hundreds of million project was delayed due to this, and is now live with known intermittent issues. All network and dev teams have exhausted their capacities. Sometimes, its OK to move on than staying stuck to solve.

For a case like this, or other future cases, I would expect project developers to create an option to kill the pool and act as resttemplate. We are probably going to make that code change anyway at our end, and use two different ways of invoking endpoints.

This bug is not about "my" issue, but rather a permanent solution

from reactor-netty.

@AbhiramDwivedi You changed the timeouts on the target but did you add any configuration on your client e.g. maxIdleTime as it is suggested in our FAQ?

from reactor-netty.

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

from reactor-netty.

Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open.

from reactor-netty.