bug: S3 Presign URL returns incorrect URL (NoSuchBucket). about localstack HOT 9 CLOSED

Welcome to LocalStack! Thanks for reporting your first issue and our team will be working towards fixing the issue for you or reach out for more background information. We recommend joining our Slack Community for real-time help and drop a message to LocalStack Pro Support if you are a Pro user! If you are willing to contribute towards fixing this issue, please have a look at our contributing guidelines and our contributing guide.

from localstack.

Hello and thanks for your report,

Here is the documentation about path style and virtual hosted request: https://docs.localstack.cloud/user-guide/aws/s3/#path-style-and-virtual-hosted-style-requests

Regarding this report, I'm not sure I understand:

So, either fix the bucket API behavior to the same as AWS, which takes a bucket name from URL or modify Presign API so it returns an URL which works.

Currently it is incosistent.

Calling s3Client.presignPutObject is purely a client side operation, your client does not send anything to LocalStack, and LocalStack does not return anything to you. Your client will automatically generate the presigned URL. The client simply takes the endpoint you've configured, and add the query string parameters needed for S3 to understand your request. LocalStack does not return any endpoint to you, which is why it doesn't understand your request, as it's not following the right format.
You need to either configure your s3Client to use an endpoint compatible with the virtual host styled request (prefixed with s3. in the host) or configure path style, like the documentation above specifies.

From the other issue, I've linked this table: https://docs.aws.amazon.com/sdkref/latest/guide/feature-ss-endpoints.html#ss-endpoints-envar

And it seems Kotlin is compatible with the AWS_ENDPOINT_URL, which means you can modify the behavior of your tests without modifying the code, either with environment variable or some JVM system properties.

aws.endpointUrl - JVM system property: Java/Kotlin only

Which means you can have specific configuration for your tests (I believe you must configure the endpoint somehow to localhost currently already?) with aws.endpointUrlS3 set to s3.localhost.localstack.cloud:4566 so that when running hour tests against LocalStack, it would use the right endpoint to create your pre-signed URL.

To answer the question about LocalStack emulating AWS, sometimes we cannot control the behavior of the SDK, so some configuration is needed. We're trying to make it as easy as possible for you, and AWS is doing progress as well with their SDK to transparently configure them, like the above.

from localstack.

Thank you for a promt and detailed response. After consulting the linked docs I managed to work around this with setting

private val s3Client by lazy {
	S3Client {
		region = configuration[ConfigurationSpec.region]
		credentialsProvider = awsCredentialsProvider
		endpointUrl = awsEndpointUrl
		forcePathStyle = configuration[AwsSpec.S3Spec.forcePathStyle] // true
	}
}

The critical part here is setting

forcePathStyle=true

in the SDK client configuration.

For future reference, this is how I configure it in application.yaml:

aws:
  s3:
    forcePathStyle: "$DEVELOPMENT:true"

Which will set to false after deployment because I set DEVELOPMENT to false in the Kubernetes manifest.

from localstack.

Great, I'm glad we could get it solved! I'm sorry about the need to change the configuration of the client, like you said, everything should be as transparent as possible. We're working on that, and are very appreciative of the changes from the AWS SDK team too to enable setting different endpoints.

Thanks also for the explanation! We work to add a documentation page for Kotlin in this page so that it would be easier to find.

Thanks again for the feedback 🙏

from localstack.

We work to add a documentation page for Kotlin so that it would be easier to find.

kotlin uses equivalent SDK as Java, so it's rather a general issue with a JVM client.

Java:
AWS docs: https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/examples-s3.html
API docs: https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/services/s3/S3BaseClientBuilder.html

kotlin:
AWS docs: https://docs.aws.amazon.com/sdk-for-kotlin/latest/developer-guide/config-endpoint.html
API docs: https://sdk.amazonaws.com/kotlin/api/latest/s3/aws.sdk.kotlin.services.s3.endpoints/-s3-endpoint-parameters/index.html

from localstack.

Also apparently javascript also has it

https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html
https://stackoverflow.com/questions/20305877/how-to-configure-aws-s3-sdk-for-node-js-to-be-used-with-localhost

So maybe it's a general option in many (if not all) SDKs.

from localstack.

Nice, thanks a lot for the added context! Yes, pathStyle configuration should be global to almost every SDK, just in different format almost every time. We'll make sure to add a note for the JVM clients.

Can we consider the issue closed?

from localstack.

Yes, it worked. Thank you for your support.

from localstack.

I've noticed that on the .NET guide: https://docs.localstack.cloud/user-guide/integrations/sdks/dotnet/

you already suggest to set ForcePathStyle to true.

from localstack.