aws / aws-sdk-go-v2 Goto Github PK
View Code? Open in Web Editor NEWAWS SDK for the Go programming language.
Home Page: https://aws.github.io/aws-sdk-go-v2/docs/
License: Apache License 2.0
AWS SDK for the Go programming language.
Home Page: https://aws.github.io/aws-sdk-go-v2/docs/
License: Apache License 2.0
The SDK's protocol marshalers will need to be updated to correctly marshal collection elem value types.
The v1 SDK exposes a single interface that most errors in the SDK try to conform to. awserr.Error
. This error doesn't make sense for a lot of the errors the SDK core and utilities expose. Specifically the code
and message
values only make sense for API response errors.
The v2 SDK should refactor awserr.Error
into an APIError
interface type that basically is the awserr.RequestFailure
type. I think the generic awserr.Error
can be replaced in favor with errors that support the github.com/pkg/errors
package's Wrap
and Cause
pattern.
Refactoring these error types should also satisfy a goal of error checking in user applications should be based more on type assertions instead of string comparison.
The global Config
type has several service specific configuration values. These values need to be moved to the individual service client types.
DynamoDB.UpdateItemRequest is quite painful to work with.
If my struct property might be empty, I have to take dynamodb.UpdateItemInput out and create a condition that somehow updates ExpressionAttributeNames, ExpressionAttributeValues & let's not forget UpdateExpression.
I wish it was as easy as the example at https://godoc.org/github.com/aws/aws-sdk-go-v2/service/dynamodb/dynamodbattribute when I can dynamodbattribute.MarshalMap my complete struct to item and UpdateItemRequest with simply the Item:. https://godoc.org/github.com/aws/aws-sdk-go-v2/service/dynamodb#UpdateItemInput seems to require a very complex structure for an update. :/ Unless I am missing something?
The external configuration loading package needs to be documented. Its currently very bare, and not obvious how to use its extended functionality.
Port aws/aws-sdk-go#1640 to v2.
The SDK's aws.Request
type is too generic and does too much. We want to break this apart. Specifically decouple the http.Request
/Response
from the the SDK's request structure. In v1 SDK having the HTTP request built early on made it difficult to extend and modify how the SDK handled requests. e.g HTTP Body value couldn't be fixed to be set to nil when there wasn't actually a request body.
One way this could be done is to abstract the Transport from the SDK's API request. This potentially could manifest as a Sender
, or Transport
interface that all SDK's service clients have as a member. This type could be responsible for the protocol (un)marshaling, and round tripping the request over the wire. This pattern would more easily allow the SDK to support other transport protocols. Such as TCP and MQTT.
What's the command to determine this?
Else, going to assume 493e3ce
go version
)?go version go1.9.2 linux/amd64
With medialive the way to test an empty token is *next == ""
. on mediapackage it's next == nil
. I'm puzzled why.
Compare
Notice if live.go uses:
if next == nil {
break
}
It just goes in an infinite loop.
V2 is an opportunity to refactor the pain points of the S3 crypto client.
~$ go version
go version go1.9.2 linux/amd64
~$ git --git-dir ~/go/src/github.com/aws/aws-sdk-go-v2/.git describe --all
tags/v2.0.0-preview.2
Hi, is there a better pattern to use to work out where a service (in my example, medialive) is active?
http://play.golang.org/p/O4MlKjZAP5O
Functions with region endpoints.Region and cfg.Region = region.ID()
looks ugly to me.
There are several places where the v1 SDK's documentation is still exists in v2 repo. These need to be cleaned up and removed.
Related to the confusion caused in #83
The SDK's protocol (un)marshaling needs to be updated for typed enums. Specifically handling typed enums being string value type aliases not pointer.
zero enum value instructs marshaler the field is unset.
Could you please provide instructions to update https://amazonlinux.com/ to a >=1.9 release?
sudo yum install go
attempts to install 1.8.3-1.amzn2 currently.
Thanks!
Customers have requested that the SDK's generated iface packages generate an interface per API operation, instead of grouping all operations together in a single interface.
This feedback is both from usability study, and v1 SDK users that found the size, and breaking changes of the current iface
package to be difficult to use.
Please fill out the sections below to help us address your issue.
go version
)?go version go1.9.2 linux/amd64
main.go:13:2: cannot find package "github.com/aws/aws-sdk-go-v2/aws/session" in any of:
/usr/lib/go/src/github.com/aws/aws-sdk-go-v2/aws/session (from $GOROOT)
/home/hendry/go/src/github.com/aws/aws-sdk-go-v2/aws/session (from $GOPATH)
Copy the example from https://godoc.org/github.com/aws/aws-sdk-go-v2#hdr-Complete_SDK_Example
Paste it into a main.go somewhere inside $GOPATH/src
Run it
~$ go version
go version go1.9.2 linux/amd64
~$ git --git-dir ~/go/src/github.com/aws/aws-sdk-go-v2/.git describe --all
tags/v2.0.0-preview.2
What happened to CredentialsChainVerboseErrors
in v1?
Please fill out the sections below to help us address your issue.
v2
go version
)?1.9
Hi. So I have a test like this:
func (fc *fakeCreateCFClient) DescribeStacksRequest(input *cloudformation.DescribeStacksInput) cloudformation.DescribeStacksRequest {
if fc.stackname == "NotEmptyStack" || fc.stackname == "DescribeStackFailed" {
return cloudformation.DescribeStacksRequest{
Request: &aws.Request{
Data: &NotEmptyStack,
Error: fc.err,
},
}
}
return cloudformation.DescribeStacksRequest{
Request: &aws.Request{
Data: &cloudformation.DescribeStacksOutput{},
},
}
}
I was expecting it to fail with a given error in the fake client which was working perfectly fine with v1 since that particular method was returning an error. Now, I don't have a perfect way to mock Send
so I guessed, that maybe a request error like the above is the way to make errors propagate through to Send
. Alas, that's not the case.
Thus, my question is, how do I make error scenarios happen?
Thanks,
Gergely.
To prevent s3manager
's input shapes from getting out of date, the SDK should use code generation. One solution is to use reflection on service/s3
input shape and generate from that.
In regards to #88
The V2 SDK's models are a little out of date and need updated to the most recent released models.
The SDK's codegen should be updated to render all API enums as typed values. With the enum being a type alias of string value type not pointer.
~$ go version ; git -C ~/go/src/github.com/aws/aws-sdk-go-v2 describe --all
go version go1.9.2 linux/amd64
tags/v2.0.0-preview.2
I find it cumbersome to maintain a mapping between my struct and dynamodb with expression.Set
. Is there a way to map/marshal the whole struct to dynamodb? I can't see how that's done with https://godoc.org/github.com/aws/aws-sdk-go-v2/service/dynamodb/expression
Retries for throttled request do not cap at 5 minutes due to throttled boolean not being checked first.
Please fill out the sections below to help us address your issue.
Version of AWS SDK for Go?
LATEST
Version of Go (go version)?
1.9.2
Support for Cognito just the way it is supported in JavaScript SDK and Android SDK to calculate the Auth Challenge to send in RespondToAuthChallenge
Refer: aws/aws-sdk-go#1245
The service client New
methods should return an error in addition to the client value. This error is to capture validation of configuration, and resolving endpoints.
Currently endpoint resolving is performed per request. This is excessive as the endpoint will always be the same across a client value.
SDK's Usage of EndpointResolver
should be updated so that the DisableSSL
and DualStack
aws.Config values can be removed. Ideally the EndpointResolver is constructed with these options at setup instead of at runtime.
NewSharedConfig
should return errors if it is unable to load the profile or the files. The caller of the function should be responsible for determining what to do with this information. Currently errors are silently dropped.
Currently the code generation is very complex and requires knowledge of previous states to make the correct decision. A better solution would be a visitor pattern and use different walkers to generate a particular piece of code.
Need to sync v1 API models into v2.
SDK API operations such as s3.GetObject
use and io.Reader in the response for the customer to read API response data from. This prevents the SDK from retrying failed get operations as the SDK's Send has already returned.
An alternative approach would be for the SDK to take an io.Writer
builder/getter as input and write to that stream. This would cause the Send
to block until the response is written to the io.Writer
...
Example customer code:
var f *os.File
req := svc.GetObjectRequest(params)
resp, err := req.Send(ctx, writeTempFileBuilder(&f))
if err != nil {
panic(err)
}
if err := f. Sync(); err != nil {
panic(err)
}
info, err := f.Stat()
if err != nil {
panic(err)
}
fmt.Println("File Info:", Info.Name(), info.Size())
f.Close()
Add the ability to paginate a request from the API's typed request.
One way this can be refactored is to breakout the pagination from a callback to a pattern similar to bufio's scanner.
This is an example using MediaLive's ListInput's API.
V1
err = svc.ListInputsPages(&medialive.ListInputsInput{},
func(page *medialive.ListInputsOutput, lastPage bool) bool {
log.Println("Received", len(page.Inputs), "inputs in page")
Inputs = append(Inputs, page.Inputs...)
return true
})
Proposed V2
req := svc.ListInputsRequest(&medialive.ListInputsInput{})
pager := req.Paginate()
for pager.Next() {
page := pager.Page()
Inputs = append(Inputs, page.Inputs...)
}
if err := pager.Err(); err != nil {
panic("paging failed, " + err.Error())
}
It would be helpful to create usage examples for an application using v1 explaining how to migrate to specific changes in v2.
Looks like there is a memory issue sometimes when building/linking the op_crawler testing utility. A unit test failed with a Go link failure.
go build github.com/aws/aws-sdk-go-v2/internal/awstesting/cmd/op_crawler: /usr/local/go/pkg/tool/linux_amd64/link: signal: killed
Makefile:61: recipe for target 'unit' failed
make: *** [unit] Error 1
Current master as of 2018-01-10: 14d1593
go version
)?% go version
go version devel +9044f018ec Tue Jan 9 01:53:38 2018 +0000 darwin/amd64
s3manager.Uploader panics when uploading an object with custom metadata.
Code (make sure to set the bucket name): https://play.golang.org/p/l1nWAdGf_qT
Output:
% go run upload.go
panic: reflect.Set: value of type map[string]*string is not assignable to type map[string]string
goroutine 1 [running]:
reflect.Value.assignTo(0x139e400, 0xc4206c4030, 0x15, 0x14342cc, 0xb, 0x139ebe0, 0x0, 0x139e400, 0xc4206c4030, 0x15)
/Users/fsouza/.gimme/versions/go/src/reflect/value.go:2235 +0x41d
reflect.Value.Set(0x139ebe0, 0xc4206c6090, 0x195, 0x139e400, 0xc4206c4030, 0x15)
/Users/fsouza/.gimme/versions/go/src/reflect/value.go:1373 +0xa4
github.com/aws/aws-sdk-go-v2/internal/awsutil.rcopy(0x139ebe0, 0xc4206c6090, 0x195, 0x139e400, 0xc42012c230, 0x195, 0xc4206c6000)
/Users/fsouza/src/github.com/aws/aws-sdk-go-v2/internal/awsutil/copy.go:97 +0xc3f
github.com/aws/aws-sdk-go-v2/internal/awsutil.rcopy(0x142a820, 0xc4206c6000, 0x199, 0x14287a0, 0xc42012c1c0, 0x199, 0x1)
/Users/fsouza/src/github.com/aws/aws-sdk-go-v2/internal/awsutil/copy.go:78 +0x261
github.com/aws/aws-sdk-go-v2/internal/awsutil.rcopy(0x14194a0, 0xc4206c6000, 0x16, 0x1373820, 0xc42012c1c0, 0x16, 0x101)
/Users/fsouza/src/github.com/aws/aws-sdk-go-v2/internal/awsutil/copy.go:68 +0x8e7
github.com/aws/aws-sdk-go-v2/internal/awsutil.Copy(0x14194a0, 0xc4206c6000, 0x1373820, 0xc42012c1c0)
/Users/fsouza/src/github.com/aws/aws-sdk-go-v2/internal/awsutil/copy.go:21 +0xb9
github.com/aws/aws-sdk-go-v2/service/s3/s3manager.(*uploader).singlePart(0xc420177ea0, 0x14a50c0, 0xc4206c4000, 0x1, 0x14a39e0, 0xc420010050)
/Users/fsouza/src/github.com/aws/aws-sdk-go-v2/service/s3/s3manager/upload.go:558 +0x65
github.com/aws/aws-sdk-go-v2/service/s3/s3manager.(*uploader).upload(0xc420177ea0, 0x0, 0x0, 0x0)
/Users/fsouza/src/github.com/aws/aws-sdk-go-v2/service/s3/s3manager/upload.go:458 +0x537
github.com/aws/aws-sdk-go-v2/service/s3/s3manager.Uploader.UploadWithContext(0x500000, 0x5, 0x0, 0x2710, 0x14aca40, 0xc420011260, 0x0, 0x0, 0x0, 0x1c08210, ...)
/Users/fsouza/src/github.com/aws/aws-sdk-go-v2/service/s3/s3manager/upload.go:376 +0x184
github.com/aws/aws-sdk-go-v2/service/s3/s3manager.Uploader.Upload(0x500000, 0x5, 0x0, 0x2710, 0x14aca40, 0xc420011260, 0x0, 0x0, 0x0, 0xc42012c1c0, ...)
/Users/fsouza/src/github.com/aws/aws-sdk-go-v2/service/s3/s3manager/upload.go:347 +0xc4
main.main()
/Users/fsouza/upload.go:17 +0x41b
exit status 2
Remove the multiple ways an API can be called and simplify to a single request method that returns a request type specific to the API.
EndpointsResolver
type is duplicated between aws
and enpdoints
package. This needs to be normalized.
Refactor CrdentialsLoader
and CredentialProvider
into a single entity instead of requiring the user to interface with both. This moves the critical section and caching from loader to the provider if the provider needs it.
This replaces the config and client's CredentialsLoader
field with a Credentials CredentialsProvider
. the CredentialsProvider would stay and interface, but it would be the responsibility of the provider to manage the critical section if it needs one.
The SDK needs to be able to code generate collection (slice/map) elem types as value types not pointers.
Requiring the user to retrieve the response metadata from the request type is very awkward. The output type should provide a way to access the response metadata.
The v2 SDK is moving its execution flow way from the need of creating functional option and towards a more explicit configuration pattern. The S3 manager Upload and Download methods need to be refactored to not take functional options and provide a more explicit configuration story.
A major change in the v2 API is the requirement to use Request methods (#33). We currently use the iface
packages to test our code using the v1 API with the strategy described in
mockgen
.
Could you please ensure that this mocking remains possible? Perhaps the Request API can return a Request object that doesn't go out to the web and instead returns a mock Response? Other alternative mock strategies would be equally welcome.
To tack on, the suggestion in #50 to split up the iface
packages would likely make mocking for us more painful as we would have to generate more interfaces.
The aws.Config
pointers be converted to value` types.
The embedded Request
within the API typed request is mutable and incorrectly will persist data.
With V2, we are curious if people are using the setters in the service APIs. As this isn't too idiomatic to Go, we want to ensure that there is a benefit that people are getting other than just convinience. Should we keep setters in V2?
{
"checksumSHA1": "VsyHSneU5fMUAaQDQ1BAqZcvrG0=",
"path": "github.com/aws/aws-sdk-go-v2",
"revision": "01b01bc24609090862fbae454a86704b75a0f3b3",
"revisionTime": "2018-01-11T17:45:17Z"
},
go version
)?1.9.2
I was just tried to upgrade from v1 to v2. I'm using cloudformation API. I could successfully created a stack with CreateStackRequest
and got the stack id. But then when I used WaitUntilStackCreateComplete
, it never returned.
// create stack
req := client.CreateStackRequest(&input)
output, err := req.Send()
if err != nil {
return
}
fmt.Println(output)
// wait the stack to be created
describeInput := cf.DescribeStacksInput{
StackName: output.StackId,
}
fmt.Printf("Waiting until the stack is created...")
if err = client.WaitUntilStackCreateComplete(&describeInput); err != nil {
fmt.Println(err)
deleteStack(stackId)
return
}
fmt.Println("Done.")
This is a snippet of my code. I never saw the Done
printing out.
If you have have an runnable example, please include it.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.