I have vertical querying a dynamodb, i tested it using both AwsCrtAsyncHttpClient and VertxNioAsyncHttpClient.

AwsCrtAsyncHttpClient seems 55% faster than VertxNioAsyncHttpClient.

1. DynamoDbAsyncClient using AwsCrtAsyncHttpClient (inside my Verticle start method)

DynamoDbAsyncClient dbAsyncClient =
            DynamoDbAsyncClient.builder()
                    .httpClient(AwsCrtAsyncHttpClient.builder().build())
                    .build();

Throughput = 12.4K rps

2. DynamoDbAsyncClient using AwsCrtAsyncHttpClient (inside my Verticle start method)

DynamoDbAsyncClient dbAsyncClient = VertxSdkClient.withVertx(
                        DynamoDbAsyncClient.builder()
                        , context)
                .build();

Throughput = 7.7K rps

Not sure if i am doing something wrong, is it expected?
What is primary advantage of using VertxSdkClient vs building DynamoDbAsyncClient using aws sdk directly?

Hi

Response should be paused until the subscriber (ReadStreamPublisher) requests data via (stream.fetch(n) which automatically resumes) else you are having the chance of missing data. This is visible when you are fetching big files from s3

See: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/

Use:

- name: Set output
run: echo "{name}={value}" >> $GITHUB_OUTPUT

instead

I am trying to download 1000s of S3 objects (2-6mb each) from an s3 bucket simultaneously. I've done basically nothing out of the ordinary. I'm constructing the s3asyncclient like this:

s3AsyncClient = VertxSdkClient.withVertx(S3AsyncClient.builder(), vertx.getOrCreateContext()).build();

and calling it like this:

    final Path p = PATH.resolve(UUID.randomUUID().toString());
    final CompletableFuture<GetObjectResponse> cf = s3AsyncClient.getObject(getLargeImageRequest(index), p);

The result, however, is a corrupted image that, from the looks of it is having its data written in the wrong order. This happens even if i only make one request. I did see #35 from last year and was able to get a single file downloading doing like this:

  Future<String> writeFile(Buffer data)
  {
    final Path p = PATH.resolve(UUID.randomUUID().toString() + ".jpg");
    final Promise<Void> promise = Promise.promise();
    vertx.fileSystem().writeFile(p.toString(), data, promise);
    return promise.future().map(p.toString());
  }

  Future<Buffer> getImage(int index, long length)
  {
    final String ebAddress = "s3-forward";
    final MessageProducer<Buffer> producer = vertx.eventBus().sender(ebAddress);
    final VertxAsyncResponseTransformer<GetObjectResponse> transformer = new VertxAsyncResponseTransformer<>(producer);

    final Buffer received = Buffer.buffer();
    final Promise<Buffer> promise = Promise.promise();
    vertx.eventBus().<Buffer>consumer(ebAddress, msg -> {
      received.appendBuffer(msg.body());
      if (received.length() == length)
      {
        promise.complete(received);
      }
    });

    s3AsyncClient.getObject(getLargeImageRequest(index), transformer);

    return promise.future();
  }

Is it expected that the default AWS SDK transformer for a Path object wont work out of the box? I guess i could use an event bus and my own async response transformer to stream data to file handles as it comes in but this seems like a lot of moving parts.

Can anyone advise how I should proceed here? I can upload the entire project very easily but its really not much more than what I pasted.

Consider upgrading to a superior version.

For now, tests are failing on version >= 2.3.0 because Kinesis doesn't seem to work with CBOR compression disabled.
Hence, the tests with localstack are failing.

2.17.115 has resolved several CVEs that are marked on 2.15.45. Is there an easy way i can prove to myself and the maintainers that updating to this does not break aws-sdk? If so I will make a PR against this repo to update the README

Hi
I am trying to create vertx microservice accessing DynamoDb in the backend

Kindly provide some working example for the same

Also is this vertx client production ready?

My particular situation required supporting a proxy server, so I forked and hard coded the required additional ProxyOptions in the VertxNioAsyncHttpClient, where the Vert.x HttpClient is created.

It would perhaps be beneficial to support passing Vert.x HttpClientOptions as an optional parameter in the VertSdkClient.withVertx call?

• Vert.x => 4.4.2
• AWS-SDK => 2.20.74

Currently the VertxExecutor uses a Vertx instance for executing task. The AWS client uses it to dispatch back to the application when the response should be delivered to the application, however this is achieved internally by an async CompletableFuture completion on the ForkJoin pool which forces the executor to pick a new Vert.x Context.

Change the VertxExecutor to use instead of Context instance and use it for task execution.

Update to

• Vertx 3.8.0
• AWS SDK 2.7.8

Is it possible to stream/pump the bytes received from s3 to a vertx stream (like pushing the received bytes to a vertx HttpServerResponse). From looking at the code and tests, I think we would need a streaming implementation of AsyncResponseTransformer.toBytes, which would pump bytes as soon as they are available.
Or did I miss something and this already is possible?

Today, the project depends on: software.amazon.awssdk:aws-sdk-java:2.2.0
Which tears down every aws service implementation as a dependency.

This can really be problematic when building fatJars.
(example, in a Kotlin project, fatJar reduced from 140M down to 30M by using only Kinesis instead of the full aws-sdk).

One possibility, would be to set this dependency in compileOnly scope, and let the user decide which parts of the AWS SDK he needs. But this could lead to version clash.

Letting the user exclude / include sounds like a bad design.

Probably a cleaner solution can be found.

I'm using VertxSdkClient.withVertx to create my non-blocking DynamoDB client and it works. However, I see some occurrences of "Connection reset java.net.SocketException: Connection reset" (the full stack trace is given below) while running my application.

To try and get some more feedback, I added an exceptionHandler to VertxNioAsyncHttpClient like this:
`private HttpClient createVertxHttpClient(Vertx vertx) {
HttpClientOptions options = new HttpClientOptions()
.setSsl(true)
.setKeepAlive(true);

    return vertx.createHttpClient(options).connectionHandler(con -> {
        con.exceptionHandler(err -> {
            logger.error("VertxNioAsyncHttpClient connectionHandler.exceptionHandler: " + err.getMessage(), err);
        });
    });
}`

This exceptionHandler is getting called.
Any ideas what causes the "Connection reset" or what I can do to avoid them?

The full stack trace as being logged:
VertxNioAsyncHttpClient connectionHandler.exceptionHandler: Connection reset java.net.SocketException: Connection reset at java.base/sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:345) at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:376) at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:247) at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1147) at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:347) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514) at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:830)

See: https://github.com/actions/create-release

Pick within

• elgohr/Github-Release-Action
• marvinpinto/action-automatic-releases
• softprops/action-gh-release
• ncipollo/release-action

And re-work the API gateway test, which should now work since: localstack/localstack#1030 (comment) has been fixed.

@wem said:

Are you interested in Kotlin extensions for this library in a separate sub-lib like aws-sdk-lang-kotlin? I think about a collection of AWS SDK extensions with the Await approach, like most of the other Vert.x libraries have. I ask because it would be some work :)

I personally don't have enough knowledge in Kotlin koroutines to manage such a set of extension methods.

I think reactiverse maintainers should probably be consulted and give their opinion. But in the meantime, maybe @wem you could set the foundations of such a project in a personal repository, so that we can get a glimpse at what it looks like.

In its latest release, localstack broke the Python lambda support.

To get tests pass again, I simply @Disabled it, but in a near future, it'd be good to test with another type of lambda function (like a node.js one) instead, maybe this type of lambdas are still supported.

Worth the try so that we can have more tests.

What's the release date for v1.0.0? The last release (0.7.0) was almost one year ago...

Looks like very useful library but no documentation on how to use. How do i add dependencies ? any setup required to use this?

Hi folks! We'd love to use this project, but when we follow the readme to import this project in gradle, we're not able to resolve the artifact.

Would it be possible to start publishing releases for this project to a maven repo?

it would be great to have Travis integration for PR's

I am looking into using this library.
I saw the compatibility matrix, but wanted to ask anyway: Is this library compatible with Vert.x 3.7.1?

Hi there,

are there any plans for a 0.4.0 release, which will make the changes from 390b42a available in an official release?

I was receiving a signature verification exception when trying to make an S3 list request. Making the request with both this Vert.x async client and the built-in client that AWS provides, I noticed that the request using Vert.x was missing the query parameters

Vert.x

GET / HTTP/1.1
amz-sdk-invocation-id: 8cd60990-8554-bb8c-3683-cd004c6c0c00
....

Built-in async client

GET /?prefix=some-prefix%2F HTTP/1.1
amz-sdk-invocation-id: 78f057b3-2e8a-e93b-213b-fb5ee1d31071
....

This fix to the VertxNioAsyncHttpClient worked:

    private static RequestOptions getRequestOptions(SdkHttpRequest request) {
        return new RequestOptions()
                .setHost(request.host())
                .setPort(request.port())
                .setURI(createRelativeUri(request.getUri()))
                .setSsl("https".equals(request.protocol()));
    }

    private static String createRelativeUri(URI uri) {
        return (StringUtils.isEmpty(uri.getPath()) ? "/" : uri.getPath()) +
                // AWS requires query parameters to be encoded as defined by RFC 3986.
                // see https://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html
                // uri.toASCIIString() returns the URI encoded in this manner
                (StringUtils.isEmpty(uri.getQuery()) ? "" : "?" + Strings.split(uri.toASCIIString(), "?")[1]);
    }

Since the context is different on every request? If I don't then wouldn't the context be wrong except for the first request?

Please, add support for software.amazon.awssdk.services.s3.internal.crt.S3CrtAsyncHttpClient

Documentation https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/crt-based-s3-client.html

We have connectivity issues during data transfer, using the standard S3AsyncClient. From the documentation:

The AWS CRT-based S3 client improves transfer reliability in case there is a network failure. Reliability is improved by retrying individual failed parts of a file transfer without restarting the transfer from the beginning.

In my case it concerns the S3 SDK.

L39 of WriteStreamSubscriber.java crashes when calling byteBuffer.array() because the S3 SDK passes a bytebuffer that is backed by an array but is read-only.

From docs:

ReadOnlyBufferException - If this buffer is backed by an array but is read-only

This prevents us from upgrading beyong 2.20.75

Are there plans in motion for adding RxJava(2) support? Also, could you add to the documentation that as the project stands, it is not compatible with vertx-rx?

Hi, I've noticed some strange behavior around S3AsyncClient when wrapping it with the reactiverse aws-sdk wrapper.

I'm currently running into an issue where trying to upload files where the s3 key contains a special character, in my scenario +, results in a signature mismatch expection.
java.util.concurrent.CompletionException: software.amazon.awssdk.services.s3.model.S3Exception: The request signature we calculated does not match the signature you provided. Check your key and signing method.

I dug a bit deeper, and to summarize my findings:

• s3 does in-fact support these special characters
• removing the special character results in a success
• aws cli and aws web console allow this file to be uploaded to the expected key
• The synchronous S3 client works
• The async s3 client without the aws-sdk wrapper works

I was able to narrow down a JUnit 5 test scenario to outline the inconsistent results. The only thing you'll need to do to run it is update the bucket variable at the top with a bucket of the test.

import io.reactiverse.awssdk.VertxSdkClient;
import io.vertx.core.Future;
import io.vertx.core.Vertx;
import io.vertx.core.buffer.Buffer;
import io.vertx.junit5.VertxExtension;
import io.vertx.junit5.VertxTestContext;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
import software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3AsyncClient;
import software.amazon.awssdk.services.s3.model.PutObjectRequest;
import software.amazon.awssdk.services.s3.model.PutObjectResponse;

import java.nio.file.Path;
import java.util.UUID;

@ExtendWith(VertxExtension.class)
public class ReactiverseS3AsyncClientBug
{
    private static final AwsCredentialsProvider provider = DefaultCredentialsProvider.create();
    private static final Region region = Region.US_EAST_2;
    private static final String bucket = "my-bucket";
    private static final Path filePath = Path.of("sample_file.txt");

    @BeforeAll
    static void setup(Vertx v, VertxTestContext ctx)
    {
        final String text = "Some text data to store to disk";
        v.fileSystem().writeFile(filePath.toString(), Buffer.buffer(text)).onComplete(ctx.succeedingThenComplete());
    }

    @Test
    void uploadStuff_withSpecialChar_Async_reactiverseWrapper(final Vertx v, final VertxTestContext ctx)
    {
        final S3AsyncClient s3AsyncClient =
                VertxSdkClient.withVertx(S3AsyncClient.builder().credentialsProvider(provider).region(region),
                                         v.getOrCreateContext()).build();

        saveFileToS3(s3AsyncClient).onComplete(ctx.succeedingThenComplete());
    }

    @Test
    void uploadStuff_withSpecialChar_Async(final Vertx v, final VertxTestContext ctx)
    {

        final S3AsyncClient s3AsyncClient =
                S3AsyncClient.builder().credentialsProvider(provider).region(region).build();

        saveFileToS3(s3AsyncClient).onComplete(ctx.succeedingThenComplete());
    }

    private Future<PutObjectResponse> saveFileToS3(final S3AsyncClient client)
    {
        final String s3Key = "specialCharTest/Test_Sync_With_Special_Char&$@=;:+_" + UUID.randomUUID() + "_.txt";

        final PutObjectRequest reqObj = PutObjectRequest.builder().bucket(bucket).key(s3Key).build();
        return Future.fromCompletionStage(client.putObject(reqObj, filePath));
    }
}

The S3AsyncClient built with the VertxSdkClient.withVertx fails, where as the default one does not.

I ran into a similar case a while back where I abused the executeBlocking function on vertx to run the synchronous client as a work around. That scenario had similar behavior when trying to run s3AsyncClient.listObjectsV2, where it was failing when using the client wrapped by VertxSdkClient however passing without the wrapper.

I have not dove too deep into the investigation. Without special characters the putObject functionality works fine. From what I've read so far, special characters need to be url encoded when doing a http request. Its possible the encoding part is not working as intended causing the signature mismatch.

Let me know if I'm simply doing something wrong here, or if I can provide additional details to reproduce.

Thanks!