rhnvrm / simples3 Goto Github PK

View Code? Open in Web Editor NEW

137.0 3.0 28.0 66 KB

Simple no frills AWS S3 Golang Library using REST with V4 Signing (without AWS Go SDK)

License: Other

Go 100.00%

aws-s3 aws golang aws-signature-v4

simples3's Introduction

simples3 : Simple no frills AWS S3 Library using REST with V4 Signing

Overview

SimpleS3 is a Go library for manipulating objects in S3 buckets using REST API calls or Presigned URLs signed using AWS Signature Version 4.

Install

go get github.com/rhnvrm/simples3

Example

testTxt, _ := os.Open("testdata/test.txt")
defer testTxt.Close()

// Create an instance of the package
// You can either create by manually supplying credentials
// (preferably using Environment vars)
s3 := simples3.New(Region, AWSAccessKey, AWSSecretKey)
// or you can use this on an EC2 instance to 
// obtain credentials from IAM attached to the instance.
s3, _ := simples3.NewUsingIAM(Region)

// You can also set a custom endpoint to a compatible s3 instance. 
s3.SetEndpoint(CustomEndpoint)

// Note: Consider adding a testTxt.Seek(0, 0)
// in case you have read 
// the body, as the pointer is shared by the library.

// File Upload is as simple as providing the following
// details.
resp, err := s3.FileUpload(simples3.UploadInput{
    Bucket:      AWSBucket,
    ObjectKey:   "test.txt",
    ContentType: "text/plain",
    FileName:    "test.txt",
    Body:        testTxt,
})

// Similarly, Files can be deleted.
err := s3.FileDelete(simples3.DeleteInput{
    Bucket:    os.Getenv("AWS_S3_BUCKET"),
    ObjectKey: "test.txt",
})

// You can also download the file.
file, _ := s3.FileDownload(simples3.DownloadInput{
    Bucket:    AWSBucket,
    ObjectKey: "test.txt",
})

data, _ := ioutil.ReadAll(file)
file.Close()

// You can also use this library to generate
// Presigned URLs that can for eg. be used to
// GET/PUT files on S3 through the browser.
var time, _ = time.Parse(time.RFC1123, "Fri, 24 May 2013 00:00:00 GMT")

url := s.GeneratePresignedURL(PresignedInput{
    Bucket:        AWSBucket,
    ObjectKey:     "test.txt",
    Method:        "GET",
    Timestamp:     time,
    ExpirySeconds: 86400,
})

Contributing

You are more than welcome to contribute to this project. Fork and make a Pull Request, or create an Issue if you see any problem or want to propose a feature.

Author

Rohan Verma [email protected]

License

BSD-2-Clause-FreeBSD

simples3's People

Contributors

Stargazers

Watchers

simples3's Issues

B2 support

Discussion forked from knadh/listmonk#617

detectFileSize does not measure size of the entire stream

Hello,

When uploading small objects to my local Minio instance, I got the following error from s3.FileUpload()

400 Bad Request: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>EntityTooSmall</Code><Message>Your proposed upload is smaller than the minimum allowed object size.</Message><BucketName>u9k-dev</BucketName><Resource>/u9k-dev</Resource><RequestId>16354459047EE460</RequestId><HostId>4a57eb7d-dd77-496f-aee1-f604ca81f4f8</HostId></Error>

This was strange because I should also be able to upload small files.
I figured out that it was happening after I added a content type detection function in my code, which does the following:

// adapted from https://golangcode.com/get-the-content-type-of-file/
func getFileContentType(r io.Reader) string {
	// Only the first 512 bytes are used to sniff the content type.
	buffer := make([]byte, 512)

	_, err := r.Read(buffer)
	if err != nil {
		log.Printf("Failed to detect Content-Type: %s\n", err)
		return "application/octet-stream"
	}

	// Use the net/http package's handy DectectContentType function. Always returns a valid
	// content-type by returning "application/octet-stream" if no others seemed to match.
	contentType := http.DetectContentType(buffer)
	return contentType
}

Nothing complicated, just reads the first 512 bytes of the filestream and analyzes it.

But this breaks s3.FileUpload, in particular the detectFileSize routine, because it does not count from the beginning. I believe this is a bug.

	pos, err := body.Seek(0, 1) // this does not do anything, because it is seeking 0 bytes relative to the current position (1 = io.SeekCurrent)
	if err != nil {
		return -1, err
	}
	defer body.Seek(pos, 0)

	n, err := body.Seek(0, 2)
	if err != nil {
		return -1, err
	}
	return n, nil

See https://godoc.org/io#Seeker for reference.

A version of the function that calculates the size of the entire stream would look like this:

	pos, err := body.Seek(0, io.SeekStart)
	if err != nil {
		return -1, err
	}
	defer body.Seek(pos, 0)

	n, err := body.Seek(0, io.SeekEnd)
	if err != nil {
		return -1, err
	}
	return n, nil

Please let me know (and document) if this is intended behavior.

simples3/simples3.go

Line 191 in 59b9f0b

func detectFileSize(body io.Seeker) (int64, error) {

Need support for ACL and Content disposition while uploading files.

Both of these are needed while uploading pdfs(or jpegs) with specific permissions.

Encoding issues for object keys with special characters

Hey,

so I stumbled across something odd. Uploading files with filenames containing special characters works flawlessly, but downloading or deleting them fails. In my particular case, my local Minio instance complains that the request signature does not match, but I think that is just a side effect.

Consider this test example:

package storage

import (
	"bytes"
	"path"
	"testing"

	"github.com/rhnvrm/simples3"
)

func TestFileName(t *testing.T) {
	inData := []byte("HelloWorld\nFooBar\nOneTwoThree\n")
	testKey := "example?file%with$special&chars(1).txt"

	s3 = simples3.New("S3Region", "S3AccessKey", "S3SecretKey")
	s3.SetEndpoint("http://localhost:9000")

	_, err := s3.FileUpload(simples3.UploadInput{
		Bucket:      "simples3",
		ObjectKey:   testKey,
		ContentType: "text/plain",
		FileName:    path.Base(testKey),
		Body:        bytes.NewReader(inData),
	})
	if err != nil {
		t.Fatalf("Failed to store file %s: %s", testKey, err)
		return
	}
	_, err = s3.FileDownload(simples3.DownloadInput{
		Bucket:    "simples3",
		ObjectKey: testKey,
	})
	if err != nil {
		t.Fatalf("Failed to get file %s: %s", testKey, err)
	}

	err = s3.FileDelete(simples3.DeleteInput{
		Bucket:    "simples3",
		ObjectKey: testKey,
	})
	if err != nil {
		t.Fatalf("Failed to delete file %s: %s\n", testKey, err)
	}
}

It fails with the following error (go test -run.test TestFileName simples3_test.go):

--- FAIL: TestFileName (0.11s)
    simples3_test.go:35: Failed to get file example?file%with$special&chars(1).txt: status code: 403 Forbidden

The Server logs the following:

localhost [REQUEST s3.PostPolicyBucket] 12:11:08.211
localhost POST /simples3
localhost Host: localhost:9000
localhost Accept-Encoding: gzip
localhost Content-Length: 1938
localhost Content-Type: multipart/form-data; boundary=8f92e8553bf46e645c1f13ef35cb60bc203127facf66cdbcf215f5b2f690
localhost User-Agent: Go-http-client/1.1
localhost <BODY>
localhost [RESPONSE] [12:11:08.211] [ Duration 8.449ms  ↑ 2.0 KiB  ↓ 691 B ]
localhost 201 Created
localhost X-Amz-Request-Id: 16409DE3997F5B48
localhost X-Xss-Protection: 1; mode=block
localhost Accept-Ranges: bytes
localhost ETag: "ea3b2e64587b58a724e2f5a15a45a1f7-1"
localhost Content-Type: application/xml
localhost Location: http://localhost:9000/simples3/example%3Ffile%25with$special&chars%281%29.txt
localhost Server: MinIO/RELEASE.2020-02-20T22-51-23Z
localhost Vary: Origin
localhost Content-Length: 305
localhost Content-Security-Policy: block-all-mixed-content
localhost <BODY>
localhost 
localhost [REQUEST s3.GetObject] 12:11:08.212
localhost GET /simples3/example?file%with$special&chars(1).txt
localhost Host: localhost:9000
localhost Authorization: REDACTED
localhost Content-Length: 0
localhost Date: 20201023T121108Z
localhost User-Agent: Go-http-client/1.1
localhost X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
localhost Accept-Encoding: gzip
localhost <BODY>
localhost [RESPONSE] [12:11:08.212] [ Duration 467µs  ↑ 87 B  ↓ 658 B ]
localhost 403 Forbidden
localhost X-Xss-Protection: 1; mode=block
localhost Accept-Ranges: bytes
localhost Content-Length: 401
localhost Content-Security-Policy: block-all-mixed-content
localhost Content-Type: application/xml
localhost Server: MinIO/RELEASE.2020-02-20T22-51-23Z
localhost Vary: Origin
localhost X-Amz-Request-Id: 16409DE39A0FE29B
localhost <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><Key>example</Key><BucketName>simples3</BucketName><Resource>/simples3/example</Resource><RequestId>16409DE39A0FE29B</RequestId><HostId>4a57eb7d-dd77-496f-aee1-f604ca81f4f8</HostId></Error>
localhost

On the server side, the file is correctly stored as example?file%with$special&chars(1).txt.

So, my suspicion here is that URL-encoding is not necessary when POSTing the object (because the object key is in the body), but it is necessary for GET and DELETE requests, because the object key is part of the URL.

What do you think about this? Should the Upload and Delete functions maybe by default URL-encode the object key?

NewUsingIAM() blocks forever outside of AWS

Accidentally initialized NewUsingIAM() outside of the AWS/IAM environment and the request blocked forever. Given that the instance metadata is generated from AWS "magic" URLs that are instant, I guess hardcoding a short $N second timeout here should be fine, given that:

NewUsingIAM() is currently unable to take any additional config.
Calling this outside of the AWS environment should return an error.

simples3/simples3.go

Line 151 in ad0419e

resp, err := http.Get(baseURL)

Temporary session tokens don't seem to work

I've the following tokens in my env:

AWS_ACCESS_KEY_ID=REDACTED
AWS_SECRET_ACCESS_KEY=REDACTED
AWS_SESSION_TOKEN=REDACTED

With this simple initializing code:

	var (
		accessKey, _ = os.LookupEnv("AWS_ACCESS_KEY_ID")
		secretKey, _ = os.LookupEnv("AWS_SECRET_ACCESS_KEY")
		region, _    = os.LookupEnv("AWS_DEFAULT_REGION")
		token, _     = os.LookupEnv("AWS_SESSION_TOKEN")
	)

	// Set a default region.
	if region == "" {
		region = "ap-south-1"
	}

	// Lookup for env keys if they are present and initalise S3 based on those keys.
	if accessKey != "" && secretKey != "" {
		s3 := simples3.New(region, accessKey, secretKey)
		s3.SetToken(token)
		return s3
	} else {
		// Else check if IAM is accessible in this env.
		s3, err := simples3.NewUsingIAM(region)
		if err != nil {
			fmt.Println("error initialising s3 client using IAM: %v", err)
			os.Exit(1)
		}
		return s3
	}

I am trying to access a bucket and these credentials have access to them. With simpleS3 I am getting status code: 403 Forbidden on .FileDownload() method.

However, same works when I do:

aws s3 cp s3://my-bucket/my-key/data.json /tmp/data.json
download: s3://my-bucket/my-key/data.json to ../../../../../../tmp/data.json

I tried creating a temporary user with just access key/secret key and with same IAM policies attached and I was able to access it via simpleS3... which makes me wonder if this .Token is not being used properly.

B2 - invalid timestamp using S3 API

Related to: #17 (comment)

I'm using listmonk with an S3-compatible B2 bucket. But I get the following error.

2022/03/23 21:23:08error uploading file: status code: 400 : "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"yes\"?>\n<Error>\n <Code>InvalidRequest</Code>\n <Message>Timestamp '20220323T212308Z' is invalid</Message>\n</Error>\n"

I tested using an AWS S3 bucket and was able to successfully upload an image. I tried to see if maybe there was an issue with the datetime string formatting. But I'm starting to suspect there is something slightly different in Backblaze's API versus AWS S3.

package main

import (
  "fmt"
  "time"
)

func main() {
  amzDateISO8601TimeFormat := "20060102T150405Z"
  time := time.Now().UTC()
  amzdate := time.Format(amzDateISO8601TimeFormat)
  fmt.Println(time)
  fmt.Println(amzdate)
}

The request signature we calculated does not match the signature you provided

Hi, I'm having an error using this library, I found this error when using with the code below

<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>

this is sample code

s3 := simples3.New("us-east-1", "xxxxx", "xxxxx")
	res, err := s3.FileDownload(simples3.DownloadInput{
		Bucket:    "mybucket",
		ObjectKey: "test.txt",
	})
	if err != nil {
		panic(err)
	}
	body, err := ioutil.ReadAll(res)
	if err != nil {
		panic(err)
	}

	fmt.Println(string(body))

I have tested the credentials and there are no issues