non-`io.Seeker` body in streaming-blob operation causes content-length to not be set, resulting in 5xx about aws-sdk-go-v2 HOT 6 CLOSED

Actually no -- just double-checked and the behavior here appears to have changed. We used to tell you "not seekable, can't do it", but now it looks like a pure Reader implementation causes us to not set content-length, which does in fact result in that exact error you've given. That is much worse than what we did before. Reopening.

from aws-sdk-go-v2.

SDK operation returns a 501, api error NotImplemented: A header you provided implies functionality that is not implemented

FYI this is unrelated. The error you get when using a non-seekable body is going to be different (we're going to detect that and return it as an error before even sending the request).

This is a known pain point as you've indicated, I'm going to spawn a dedicated issue for tracking.

EDIT: jk, read on

from aws-sdk-go-v2.

Trivially reproducible:

package main

import (
	"context"

	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/config"
	"github.com/aws/aws-sdk-go-v2/service/s3"
)

type reader []byte

func (r reader) Read(p []byte) (n int, err error) {
	copy(p, r)
	return len(r), nil
}

func (r reader) Seek(offset int64, whence int) (int64, error) {
	return 0, nil
}

func main() {
	cfg, err := config.LoadDefaultConfig(context.Background())
	if err != nil {
		panic(err)
	}

	svc := s3.NewFromConfig(cfg, func(o *s3.Options) {
		o.ClientLogMode = aws.LogRequest
	})
	_, err = svc.PutObject(context.Background(), &s3.PutObjectInput{
		Bucket: aws.String("bucket"),
		Key:    aws.String("key"),
		Body:   reader("foo"),
	})
	if err != nil {
		panic(err)
	}
}

If you make this request as-is, it will succeed. If you remove the Seek implementation from reader, we will not set content-length and thus the request will fail with a 501 (at least that's what S3 does - other services may behave differently).

from aws-sdk-go-v2.

@ivanshiras note that if you're streaming from another S3 response (or you know the content length yourself some other way), you can get around this by just setting ContentLength on the input directly (hardcoded in this example, but you'd pass it in from the response etc.):

package main

import (
	"context"

	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/config"
	"github.com/aws/aws-sdk-go-v2/service/s3"
)

type reader []byte

func (r reader) Read(p []byte) (n int, err error) {
	copy(p, r)
	return len(r), nil
}

// no seek

func main() {
	cfg, err := config.LoadDefaultConfig(context.Background())
	if err != nil {
		panic(err)
	}

	svc := s3.NewFromConfig(cfg, func(o *s3.Options) {
		o.ClientLogMode = aws.LogRequest
	})
	_, err = svc.PutObject(context.Background(), &s3.PutObjectInput{
		Bucket: aws.String("bucket"),
		Key:    aws.String("key"),
		Body:   reader("foo"),
		ContentLength: aws.Int64(3),
	})
	if err != nil {
		panic(err)
	}
}

from aws-sdk-go-v2.

@ivanshiras note that if you're streaming from another S3 response (or you know the content length yourself some other way), you can get around this by just setting ContentLength on the input directly (hardcoded in this example, but you'd pass it in from the response etc.):

Thanks Luc, I switched to the using the S3 manager Uploader which doesn't have the issue and has some performance benefits where I don't need to parse the response into memory and/or disk before uploading.

from aws-sdk-go-v2.

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.

from aws-sdk-go-v2.