Is there an existing issue for this? <li class="

<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/us

bug: S3 bucket doesn't exist after being successfuly created with Java SDK about localstack HOT 5 CLOSED

nicolasduminil commented on July 24, 2024

bug: S3 bucket doesn't exist after being successfuly created with Java SDK

from localstack.

Comments (5)

localstack-bot commented on July 24, 2024

Welcome to LocalStack! Thanks for reporting your first issue and our team will be working towards fixing the issue for you or reach out for more background information. We recommend joining our Slack Community for real-time help and drop a message to LocalStack Pro Support if you are a Pro user! If you are willing to contribute towards fixing this issue, please have a look at our contributing guidelines and our contributing guide.

from localstack.

bentsku commented on July 24, 2024

Hello! The shared logs are not the LocalStack output it seems. Could you share how you are creating your S3 client? I suspect the request might not be recognized, and that could be why it says the bucket does not exist.
About creating the S3 client and issue with virtual hosted requests vs path style: https://docs.localstack.cloud/user-guide/aws/s3/#path-style-and-virtual-hosted-style-requests

Also, could you try using at least LocalStack 3.4? It seems it is using 3.0.1.

You can set more verbose logs for LocalStack by setting LS_LOG=trace (see https://docs.localstack.cloud/references/configuration/)

from localstack.

nicolasduminil commented on July 24, 2024

@bentsku : Thanks for your response. As explained, I'm not using localstack directly but on the behalf of the testcontainers. Hence, the logs above aren't the localstack ones but, as explained as well, the testcontainers ones, upon the localstack startup.

As for the S3 client instantiation, here is it is done:

  private static AmazonS3 s3Client = AmazonS3ClientBuilder.standard().withRegion(Regions.DEFAULT_REGION).build();

As for the localstack version, the 3.0.1 is the one to be used with the the current Quarkus version (3.10.0). While it's possible, of course, to use another version, this isn't recommended and won't help to fix anything, on the opposite. However, I tried 3.4 and here is the log file:

[INFO] Running fr.simplex_software.aws.iac.quarkus.s3.tests.S3FileUploadTest
2024-05-29 12:50:05,753 INFO  [io.qua.ama.lam.run.MockEventServer] (build-44) Mock Lambda Event Server Started
2024-05-29 12:50:06,355 INFO  [org.tes.doc.DockerClientProviderStrategy] (build-7) Loaded org.testcontainers.dockerclient.UnixSocketClientProviderStrategy from ~/.testcontainers.properties, will try it first
2024-05-29 12:50:06,560 INFO  [org.tes.doc.DockerClientProviderStrategy] (build-7) Found Docker environment with local Unix socket (unix:///var/run/docker.sock)
2024-05-29 12:50:06,562 INFO  [org.tes.DockerClientFactory] (build-7) Docker host IP address is localhost
2024-05-29 12:50:06,576 INFO  [org.tes.DockerClientFactory] (build-7) Connected to docker: 
  Server Version: 26.0.1
  API Version: 1.45
  Operating System: Ubuntu 22.04.4 LTS
  Total Memory: 31730 MB
2024-05-29 12:50:06,591 INFO  [org.tes.ima.PullPolicy] (build-7) Image pull policy will be performed by: DefaultPullPolicy()
2024-05-29 12:50:06,592 INFO  [org.tes.uti.ImageNameSubstitutor] (build-7) Image name substitution will be performed by: DefaultImageNameSubstitutor (composite of 'ConfigurationFileImageNameSubstitutor' and 'PrefixingImageNameSubstitutor')
2024-05-29 12:50:06,598 INFO  [org.tes.DockerClientFactory] (build-7) Checking the system...
2024-05-29 12:50:06,603 INFO  [org.tes.DockerClientFactory] (build-7) ✔︎ Docker server version should be at least 1.6.0
2024-05-29 12:50:06,723 INFO  [tc.loc.4] (build-7) LOCALSTACK_HOST environment variable set to localstack-3gjmj (to match last network alias on container with non-default network)
2024-05-29 12:50:06,724 INFO  [tc.loc.4] (build-7) Creating container for image: localstack/localstack:3.4
2024-05-29 12:50:06,728 INFO  [org.tes.uti.RegistryAuthLocator] (build-7) Failure when attempting to lookup auth config. Please ignore if you don't have images in an authenticated registry. Details: (dockerImageName: localstack/localstack:3.4, configFile: /home/nicolas/.docker/config.json, configEnv: DOCKER_AUTH_CONFIG). Falling back to docker-java default behaviour. Exception message: Status 404: No config supplied. Checked in order: /home/nicolas/.docker/config.json (file not found), DOCKER_AUTH_CONFIG (not set)
2024-05-29 12:50:06,733 INFO  [tc.tes.6.0] (build-7) Creating container for image: testcontainers/ryuk:0.6.0
2024-05-29 12:50:06,843 INFO  [tc.tes.6.0] (build-7) Container testcontainers/ryuk:0.6.0 is starting: c3e7a9eeea8f7e1ea1a11d6e58ecb1aeba050745cfa1a71a3125d72ca5e155b5
2024-05-29 12:50:07,177 INFO  [tc.tes.6.0] (build-7) Container testcontainers/ryuk:0.6.0 started in PT0.443363537S
2024-05-29 12:50:07,902 INFO  [tc.loc.4] (build-7) Container localstack/localstack:3.4 is starting: 5e9ef284f6efbaabb040a1270e54416f37dfe29c727ce19d72eb38321a92efce
2024-05-29 12:50:09,400 INFO  [tc.loc.4] (build-7) Container localstack/localstack:3.4 started in PT2.67657709S
2024-05-29 12:50:11,937 INFO  [io.qua.ama.com.dep.DevServicesLocalStackProcessor] (build-7) Amazon Dev Services for localstack (s3) started. Other Quarkus applications in dev mode will find the LocalStack automatically.
2024-05-29 12:50:12,763 INFO  [io.qua.ama.lam.run.AbstractLambdaPollLoop] (Lambda Thread (TEST)) Listening on: http://localhost:8081/_lambda_/2018-06-01/runtime/invocation/next
2024-05-29 12:50:12,791 INFO  [io.quarkus] (main) cdk-quarkus-s3 0.1 on JVM (powered by Quarkus 3.10.0) started in 8.205s. 
2024-05-29 12:50:12,792 INFO  [io.quarkus] (main) Profile test activated. 
2024-05-29 12:50:12,792 INFO  [io.quarkus] (main) Installed features: [amazon-lambda, amazon-s3, cdi, hibernate-validator, rest, rest-jackson, security, smallrye-context-propagation, vertx]
software.amazon.awssdk.services.s3.model.NoSuchBucketException: The specified bucket does not exist (Service: S3,  Status Code: 404, Request ID: ec4dbdb9-8603-4070-8ad2-3a96cea0efbf, Extended Request ID: s9lzHYrFp76ZVxRcpX9+5cjAnEH2ROuNkd2BHfIa6UkFVdtjf5mKR3/eTPFvsiP/XV/VLi31234=)
    at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:156)
    at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleResponse(AwsXmlPredicatedResponseHandler.java:108)
    at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:85)
    at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:43)
    at software.amazon.awssdk.core.internal.handler.BaseClientHandler.lambda$successTransformationResponseHandler$7(BaseClientHandler.java:279)
    at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler.lambda$prepare$0(AsyncResponseHandler.java:92)
    at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150)
    at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
    at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179)
    at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler$BaosSubscriber.onComplete(AsyncResponseHandler.java:135)
    at software.amazon.awssdk.core.internal.metrics.BytesReadTrackingPublisher$BytesReadTracker.onComplete(BytesReadTrackingPublisher.java:74)
    at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler$DataCountingPublisher$1.onComplete(ResponseHandler.java:519)
    at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler.runAndLogError(ResponseHandler.java:254)
    at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler.access$600(ResponseHandler.java:77)
    at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler$PublisherAdapter$1.onComplete(ResponseHandler.java:375)
    at software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerPublisher.publishMessage(HandlerPublisher.java:402)
    at software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerPublisher.flushBuffer(HandlerPublisher.java:338)
    at software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerPublisher.receivedDemand(HandlerPublisher.java:291)
    at software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerPublisher.access$200(HandlerPublisher.java:61)
    at software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerPublisher$ChannelSubscription$1.run(HandlerPublisher.java:495)
    at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:1583)

from localstack.

bentsku commented on July 24, 2024

@nicolasduminil thanks for your response.

First, could you try creating your client the following way and see if that would fix the issue?

private static AmazonS3 s3Client = AmazonS3ClientBuilder
    .standard()
    .withRegion(Regions.DEFAULT_REGION)
    .withPathStyleAccessEnabled(true)
    .build();

Even though you're using LocalStack with TestContainers, I believe LocalStack logs might still be available.
The problem is that is your client's logs, and we cannot understand what is going on LocalStack and why it would fail, and I'm not sure how I can help you.

Looking at the documentation of the LocalStack testcontainers module, it seems you can create your client with more settings: https://java.testcontainers.org/modules/localstack/, but I'm not sure how much Quarkus is doing here.

It seems you can also access your container logs like this: https://java.testcontainers.org/features/container_logs/

The problem here is I don't think this is an issue with LocalStack directly, as our S3 service is one of our most used service, and something like creating a bucket should not be an issue. I believe this might be a configuration issue, or an integration between the different libraries used?
We have some documentation about Quarkus integration here: https://docs.localstack.cloud/user-guide/integrations/quarkus/

I hope the client configuration solves the issue.

from localstack.

nicolasduminil commented on July 24, 2024

@bentsku : Thanks again for your help. Using the path style access, like you suggested, upon the client instantiation, doesn't change anything here.

Thank you also for directing me to these links but, again, since localstack is started via testcontainers which, in turn, is started via the Quarkus unit testing framework, they aren't relevant in my case.

My test case aims at testing an API which, among others, uploads some resources into an S3 bucket. This requires the S3 bucket to exists and this was the reason that I was trying to create it using JUnit 5 @BeforeAll annotation.
I didn't succeed to implement this way my test case, neither to understand why it failed, but I discovered that, using the Quarkus property quarkus.s3.devservices.buckets, the mentioned S3 buckets are automatically created by the framework.

Accordingly, I don't need anymore to explicitly create S3 buckets, as test prerequisites, in the code, since I'm taking advantage of this Quarkus automatic feature . However, the issue still stands and, while I know that I'll probably face it again later, in a different context, I'm closing it here for now.

Many thanks again for your help and support.

from localstack.

bug: S3 bucket doesn't exist after being successfuly created with Java SDK about localstack HOT 5 CLOSED

Comments (5)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent