CRT HTTP Client: HTTP Connections are not properly released about aws-sdk-java-v2 HOT 8 CLOSED

Hi @i-volkov thank you for reaching out.

• Was there any change in code or in the environment, besides upgrading the SDK version?
• Which DynamoDB API are you using?
• Do you have the logs with the retry errors that correlates to the increased LeasedConcurrency? They might help in reproducing the issue on our side

from aws-sdk-java-v2.

Hi @i-volkov thank you for reaching out.

• Was there any change in code or in the environment, besides upgrading the SDK version?
• Which DynamoDB API are you using?
• Do you have the logs with the retry errors that correlates to the increased LeasedConcurrency? They might help in reproducing the issue on our side

Thank you for the resposne.
Answering your questions:

• Only SDK version was updated
• GetItem and PutItem
• By errors I mean exceptions caused by the exceeded api attempt timeout. With 50 ms limit for attempt there are more retires than with 150 ms and LeasedConcurrency is growing more slowly. We don't have logs for that, because these exceptions are logged by AWS SDK using DEBUG logging level as I understand

from aws-sdk-java-v2.

@i-volkov I think I 'm able to reproduce, still need to confirm if it's the same problem you are running into.

If I set a low apiCallAtemptTimeout like 20ms, some requests will succeed, but some will fail and will be retried. When using version 2.18.34 (and aws-crt-client 2.18.34-PREVIEW) and calling GetItem in a loop, LeasedConcurrency is 2 at max. Running the same code with version 2.18.35 (and 2.18.35-PREVIEW for aws-crt-client), the LeasedConcurrency jumps to 50.

We still need to investigate the root cause.

from aws-sdk-java-v2.

@debora-ito
Yes, this is pretty much the same behaviour we are observing, the only difference is that in our case it eventually leads to lack of available connections in the pool and all requests start to time out.

With 50 ms limit for attempt there are more retires than with 150 ms and LeasedConcurrency is growing more slowly

The above sentence from my previous post is incorrect, the right way of phrasing is: the more retries we have the faster LeasedConcurrency metric grows. Sorry for the confusion.

from aws-sdk-java-v2.

The issue was occurring because of PR-3603 added in 2.18.35. Prior to this fix in 2.18.34, since we did not return when Future.isCompletedExceptionally complted, the connection used to get closed in successResponseHandler.onStream(publisher), which is right below it.
However, the latest fix PR-4825 introduced a new ResponseHandlerHelper that releases the connection in these cases. I tried with Version 2.23.2 and no longer saw the issue.

The fix is available in version 2.23.2. I would recommend using the latest version.

from aws-sdk-java-v2.

The issue was occurring because of PR-3603 added in 2.18.35. Prior to this fix in 2.18.34, since we did not return when Future.isCompletedExceptionally complted, the connection used to get closed in successResponseHandler.onStream(publisher), which is right below it. However, the latest fix PR-4825 introduced a new ResponseHandlerHelper that releases the connection in these cases. I tried with Version 2.23.2 and no longer saw the issue.

The fix is available in version 2.23.2. I would recommend using the latest version.

Thank you! We did a quick test using version 2.23.2 and the issue is gone

from aws-sdk-java-v2.

@i-volkov thank you for confirming the fix is working. Closing this.

from aws-sdk-java-v2.

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

from aws-sdk-java-v2.