It'd be very convenient to be able to pass a flag into go build to enable all the debug logging.

would be nice to have

so those of us using Windows instances can tell them apart from real computers more easily. I have a dodgy version I'm using locally that works (added Platform string xml:"platform" to RunInstanceResp and Instance structs in goamz/ec2/ec2.go) but haven't added tests (and have no experience in golang).

I'm going to try to write (my first public) pull request and see how I go.

thought I'd open this to see if anyone else also wanted to contribute. I've created a branch to add a client for AWS ElasticBeanstalk - Code is available at my github fork

Never mind, it's a fallback. I misread the code. Apologies for the false issue, this has been closed.

https://github.com/mitchellh/goamz/blob/master/aws/aws.go#L382-L386

This seems to be causing a signature error in a route53 request that someone was running into, so I thought that it was worth noting here.

Is it possible to stream directly from STDIN to an S3 bucket? I've tried the following:

package main

import (
    "github.com/mitchellh/goamz/aws"
    "github.com/mitchellh/goamz/s3"
    "log"
    "os"
)

func main() {
    auth := aws.Auth{
        AccessKey: "...",
        SecretKey: "...",
    }
    region := aws.USEast
    conn := s3.New(auth, region)
    bucket := conn.Bucket("...")
    err := bucket.PutBucket(s3.Private)
    if err != nil {
        log.Fatal(err)
    }

    multi, err := bucket.Multi("/data.zip", "application/zip", s3.Private)
    if err != nil {
        log.Fatal(err)
    }

    parts, err = multi.PutAll(os.Stdin, 1024*1024*5)
    if err != nil {
        log.Fatal(err)
    }

    err = multi.Complete(parts)
    if err != nil {
        log.Fatal(err)
    }
}

But receive the following error at multi.PutAll:

seek /dev/stdin: illegal seek

When I swap out os.Stdin for a file that I've opened with os.Open() it doesn't return an error.

Any help is much appreciated.

Hello,

I use goamz in a few of my services, like https://github.com/ProTip/cw-engine . Turns out that there are no less than three goamz forks being actively contributed to! The three in mind are mitchellh/goamz, crowdmob/goamz, and goamz/goamz. I'm starting to work on something that involves creating RDS snapshots and that's when I discovered this project is further built out for RDS than crowdmob(so I forked this and added snapshot support). However, I noticed this project lacks some crowdmob features, notably cloudwatch support which I used in cw-engine. Is there any way we might be able to consolidate these efforts into building out a go-to goamz package?

I have also raised this question in the goamz/goamz project : goamz/goamz#11 .

Cheers,
-Greg

Noticed that there's a large number of issues and pull requests banked up, which shows there's a active community wanting to add and improve the goamz project. I think it would be beneficial to open up a discussion on how to tackle this.

1. How should people contribute to make it easy to verify and merge in changes?
2. How can we add people to help with the process? I'm happy to volunteer my time

https://www.github.com/hashicorp/terraform is dependent on goamz, and it would be really nice to be able to use IAM roles in launch configurations, rather than needing to pass around AWS keys.

Hi Hashicorp team. Thanks for all your projects and products.

The hardwired default of S3 attempts is a bit problematic for me.

In:

// The S3 type encapsulates operations with an S3 region.
type S3 struct {
    aws.Auth
    aws.Region
    private byte // Reserve the right of using private data.
}

add

Attempts *aws.AttemptStrategy

Then change new to point to the current default attempts that is private and hardwired

func New(auth aws.Auth, region aws.Region) *S3 {
    return &S3{auth, region, 0, &attempts}
}

The caller could then point to another aws.AttemptStrategy objects as needed.

We could also change the New api as well but this seemed like the dumbest solution that unblocks everything.

If ok, then I'll proceed with a pull request.

thanks again!

nickg

In order to implement Private DNS on a VPC, a DHCP Option Set has to be created, documentation is at: http://docs.aws.amazon.com/AWSEC2/latest/CommandLineReference/ApiReference-cmd-CreateDhcpOptions.html

I keep getting this message:

2014/12/02 18:11:04 Request failed, got status code: 400. Response: <?xml version="1.0"?>
<ErrorResponse xmlns="https://route53.amazonaws.com/doc/2013-04-01/"><Error><Type>Sender</Type><Code>InvalidInput</Code><Message>Invalid XML ; cvc-complex-type.2.4.a: Invalid content was found starting with element 'Name'. One of '{&quot;https://route53.amazonaws.com/doc/2013-04-01/&quot;:SetIdentifier, &quot;https://route53.amazonaws.com/doc/2013-04-01/&quot;:Weight, &quot;https://route53.amazonaws.com/doc/2013-04-01/&quot;:GeoLocation, &quot;https://route53.amazonaws.com/doc/2013-04-01/&quot;:Region, &quot;https://route53.amazonaws.com/doc/2013-04-01/&quot;:Failover, &quot;https://route53.amazonaws.com/doc/2013-04-01/&quot;:AliasTarget, &quot;https://route53.amazonaws.com/doc/2013-04-01/&quot;:HealthCheckId}' is expected.</Message></Error><RequestId>~~~~~~~~</RequestId></ErrorResponse>

From the following function that uses goamz:

Can you help me figure out? (http://docs.aws.amazon.com/Route53/latest/APIReference/API_ChangeResourceRecordSets_Requests.html)

func changeRecordSets(
    client *route53.Route53,
    hostedzoneID string,
    comment string,
    changes []route53.Change,
) (*route53.ChangeResourceRecordSetsResponse, error) {
    req := route53.ChangeResourceRecordSetsRequest{
        Comment: comment,
        Changes: changes,
    }
    return client.ChangeResourceRecordSets(hostedzoneID, &req)
}

func ForceDelete(
    client *route53.Route53,
    id string,
    domain string,
) (changeID string, deleteID string, ERR error) {
    listOpt := route53.ListOpts{
        Name: domain,
    }
    resp0, err := client.ListResourceRecordSets(id, &listOpt)
    if err != nil {
        changeID = ""
        deleteID = ""
        ERR = err
        return
    }
    changes := []route53.Change{}
    for _, record := range resp0.Records {
        change := route53.Change{
            Action: "DELETE",
            Record: route53.ResourceRecordSet{
                Name:       strings.Replace(record.Name, ".com.", ".com", -1),
                Type:       record.Type,
                TTL:        record.TTL,
                Records:    record.Records,
                RecordsXML: strings.Replace(record.RecordsXML, ".</Name>", "</Name>", -1),
            },
        }
        changes = append(changes, change)
    }
    fmt.Printf("%+v\n", changes)
    resp1, err := changeRecordSets(client, id, "Delete", changes)
    if err != nil {
        changeID = ""
        deleteID = ""
        ERR = err
        return
    }
    resp2, err := client.DeleteHostedZone(id)
    changeID = route53.CleanChangeID(resp1.ChangeInfo.ID)
    deleteID = route53.CleanChangeID(resp2.ChangeInfo.ID)
    ERR = err
    return
}

Can we please change this to 30 seconds?

Was curious if the Sts (secure token service) will be added as an authentication strategy. http://docs.aws.amazon.com/STS/latest/APIReference/Welcome.html

Hi,

i have issue that my app is a commercial application how can i link this library in go? Is there already a pre compiled version some where?

AWS recently opened EU (Frankfurt) region, need to add it to the available regions

Hi,

Should func GetAuth try credentials in the same order as botocore?

Currently aws.go GetAuth tries:

PackerConfig->Shared->Env->Instance

As seen in

In contrast, botocore, which the aws-cli tools use tries:

Env->Shared->Config->OriginalEc2->boto->Instance

As seen in: https://github.com/boto/botocore/blob/f93423146eb28e7e0d46aca8929ad16c3d5cf931/botocore/credentials.py#L57

Having Packer check in a different order sometimes causes confusion when users expect the evaluation order to be the same as the official CLI tools.

Cheers,

Lauchlin

Ability to add BlockDeviceMapping is missing in launchconfiguration implementation.

users will be operating in disparate environments, so one-size-fits-all configuration for resilience is not appropriate.

the client should be able to change eg the number of retries without recompiling the library.

This could be similar to EnvAuth(..), i.e. EnvRegion(..).

We need more visibility down into the request layer to tackle bugs like hashicorp/packer#668. We can do this by logging. Logging unconditionally on the root logger sucks, so I recommend we expose a log.Logger to the EC2 struct that everyone will use.

By default, we can make this a Logger instance that goes to ioutil.Discard.

For API, I recommend adding a new function NewWithLogger:

func NewWithLogger(auth aws.Auth, region aws.Region, logger *log.Logger) *EC2 {
}

We can't support something like a SetLogger because modifying the logger is a race if any requests are happening.

I'm unable to Get an s3 object. My guess is that it's because I have spaces in my key - keys without spaces download fine.

I haven't debugged this fully - are you url encoding the path?

In Nov 2014, Amazon announced support for Private DNS within a VPC.

For this, there are several steps that need to happen in order to implement it on a VPC, first, you need to create a private hosted zone. This is documented at: http://docs.aws.amazon.com/Route53/latest/APIReference/API-create-hosted-zone-private.html

It is basically the same endpoint to create public zones but with different request parameters.

Hi, nice library.
Support for the Product Advertising API would be great. Do you have any plans on adding this?
Cheers Bijan

At this moment, aws.go supports explicit Key, Secret pair access only.

If you use IAM role you can get the temporary credentials without explicit configuration. This credential contain extra "token" field.

IAM_ROLE=$(curl -s http://169.254.169.254/latest/meta-data/iam/security-credentials/)
# will return (key, secret, token) triple
curl -s http://169.254.169.254/latest/meta-data/iam/security-credentials/${IAM_ROLE}

By the way, it is exactly the same way how aws s3 cli, s3cmd tool works: it will use your IAM account by default.

Does it makes sense to add IAM roles authentication feature?

# to aws.go:
func IamAuth()
``

As of 9e419ba, a request like Action=RunInstances&BlockDeviceMapping.1.DeviceName=%2Fdev%2Fsdb&BlockDeviceMapping.1.Ebs.DeleteOnTermination=false&BlockDeviceMapping.1.VirtualName=ephemeral0 will be generated for the following BlockDeviceMapping array:

block_devices := []ec2.BlockDeviceMapping{
    ec2.BlockDeviceMapping{
        DeviceName: "/dev/sdb",
        VirtualName: "ephemeral0",
    },
}

(There should be no BlockDeviceMapping.1.Ebs.DeleteOnTermination.)

However, this causes EC2 to interpret the BlockDeviceMapping as referring to an EBS volume (even though it could be referring to an instance-store volume).

S3 purported supports Range requests, but there's no way to specify these in Bucket.Get and friends. Perhaps a GetHeader method or similar is required?

The following build error seems to be caused by 937d133

$ make dev
==> Getting dependencies...
# github.com/mitchellh/goamz/ec2
../../mitchellh/goamz/ec2/ec2.go:2360: cannot use options.MapPublicIpOnLaunch (type bool) as type string in assignment
../../mitchellh/goamz/ec2/ec2.go:2362: cannot use SimpleResp literal (type *SimpleResp) as type *ModifySubnetAttributeResp in assignment
Makefile:9: recipe for target 'dev' failed
make: *** [dev] Error 2

reverting to 9e419ba solves this issue. Please check this PR again and fix bugs.

We've been using NoDevice: true on our BlockDeviceMapping in order to turn off a few devices of an AMI that we're using.

Today we tried to launch Spot Instances using some the same code, but we got this error:

strconv.ParseBool: parsing "": invalid syntax

We tracked it down to this where the response XML is Unmarshalled.

It seems like the empty noDevice tag inside

                    <item>
                        <deviceName>/dev/sdb</deviceName>
                        <noDevice/>
                    </item>

Cannot be mapped back to a boolean.

When we changed NoDevice to be a string, it worked:

This is obviously not the correct fix for this problem, but I'm not 100% sure what would be.

There is no way to associate a parameter group (DBParameterGroupName here: http://docs.aws.amazon.com/AmazonRDS/latest/APIReference/API_CreateDBInstance.html) with the RDS instance being created. This will be very useful to then plumb through to terraform.

A DHCP Option Set needs to be associated to a VPC in order to support private DNS

AWS Documentation on how to associate a DHCP Option Set to a VPC: http://docs.aws.amazon.com/AWSEC2/latest/CommandLineReference/ApiReference-cmd-AssociateDhcpOptions.html

Hello.

I am using this code to download file from S3 bucket

// key, secret and bucketNames are valid vars
auth, err := aws.GetAuth(key, secret)
if err != nil {
panic(err.Error())
}
conn := s3.New(auth, aws.USEast)
bucket := conn.Bucket(bucketName)
data, err := bucket.Get("aws-programmatic-access-test-object")
if err != nil {
return err
}

The .Get() function return error : "Get : 301 response missing Location header"
I can't find what the reason or any solution.
Does anybody knows what the problem is?

Thank you in advance.

Hola,

for S3, one can have custom HTTP headers on a PUT request but not on a GET request.

I would like parity here, but doing the following:

func (b *Bucket) GetResponse(path string) (*http.Response, error) {
    return b.GetResponseHeader(path, make(http.Header))
}
func (b *Bucket) GetResponseHeader(path string, customHeaders http.) (*http.Response, error) {
    req := &request{
        bucket: b.Name,
        path:   path,
                headers: customerHeaders,
    }
        etc
}

Likewise add a GetHeader, and GetReaderHeader

Kinda gross, but not sure of a better way that preserves API compatibility and/or rewrite.

thoughts? I'm happy to do a pull request.

nickg

Signing is not using SHA1 anymore. It is already solved for EC2 (#120) but needs an update for all AWS services.

The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.

It's present in the AWS docs now, and leaving it out locks you into magnetic storage.

I think that some error messages would benefit from including more information, so debugging is easier. Example aws/aws.go:342:

err = errors.New("No valid AWS authentication found")

vs

err = errors.New("No valid AWS authentication found: " + err.Error())

The retrying transport calls the RoundTrip() method on the transport here: https://github.com/mitchellh/goamz/blob/master/aws/client.go#L79

Per the net/http docs, the RoundTripper interface will always read and close the body, even on errors: http://golang.org/pkg/net/http/#RoundTripper. In addition, since a request body is an io.ReadCloser it is impossible to seek within even an unclosed body for retries.

The only workaround that I can think of that will allow a request to be correctly be retried at transport level is to copy the request body and into a new buffer to be passed to the RoundTripper on each attempt. This is obviously ugly and potentially expensive for large bodies, but something like this could work:

func (t *ResilientTransport) tries(req *http.Request) (res *http.Response, err error) {
    var b []byte
    // save body for retries
    if req.Body != nil {
        b, err = ioutil.ReadAll(req.Body)
        if err != nil {
            return nil, err
        }
    }

    for try := 0; try < t.MaxTries; try++ {
        // reset req.body
        r := bytes.NewReader(b)
        req.Body = ioutil.NopCloser(r)
        res, err = t.transport.RoundTrip(req)
        if !t.ShouldRetry(req, res, err) {
            break
        }
        if res != nil {
            res.Body.Close()
        }
        if t.Wait != nil {
            t.Wait(try)
        }
    }
    return
}

If there's a better way to do this I'd love to know ( I ran into this same issue trying to implement a retrying transport for s3gof3r and ended up retrying outside the client where I could seek in request bodies). Otherwise, I can open a pull request with the above code.

Amazon recently decided to stick to this name and use it as a convention in all AWS-related tools.

I think that supporting both variable names, the same way that botocore does is the way to go:
https://github.com/boto/botocore/blob/c76553d3158b083d818f88c898d8f6d7918478fd/botocore/credentials.py#L260-262

If you agree, I can send a PR.

I'm trying to write some code that will store files on S3, but not overwrite them if they already exist. To do this I'm calling HEAD first to see if the file exists.

I tried to do something like

details, err := s.Bucket.Head(path)
    if details.StatusCode == 404 {
    ...
    }

but found out that details is nil if the status is 404.

I ended up having to do this:

_, err := s.Bucket.Head(path)
 if err != nil && err.Error() != "404 Not Found" {
    return errFileExists
}

which does work, so I understand if you don't want to change this, but this just seems like an unexpected result This also forces me to rely on string matching an error string which just feels less stable and less idiomatic than being able to check the status code directly. It would also lead to much uglier code if I were trying to handle multiple different status codes.

I'm happy to write some tutorials and test code.

When creating a Network ACL entry with ICMP port an IcmpCode must be provided.

It's very useful to be able to use the sign function directly, for example to sign request on the server for the client.

I wanted to expose the CreationDate field from the Get Service call for S3[1]. However the way this API is setup this means that when you create a Bucket the field will be blank, even after you call PutBucket as there isn't a easy call that exposes this (for example Get Bucket doesn't return this). I have a implementation[2] but it suffers from what I describe above so I was looking for feedback if this is ok or a pointer as to what the right solution is.

[1] - http://docs.aws.amazon.com/AmazonS3/latest/API/RESTServiceGET.html
[2] - dyarnell@2542bab

@mwhooker - Since merging the retryable stuff, the EC2 tests fail. Can you take a look at that?

Would it be possible to add an option to ignore SSL certificate errors to allow using self-signed certificate (when using the Amazon S3 API with an Object Storage platform in a Private Cloud environment) ?

Sometimes (in 1 of 5 cases, can be reproduced easily) when I start Docker container and use IAM role, after several seconds I receive the following error:

No valid AWS authentication found using IAM roles

To fix this, I suggest:

1. Add detailed error output (particularly related to this issue), it is hard to figure out what is actually broken without modification of source code.
2. Fix getInstanceCredentials() or GetMetaData() function. At least one of them returns fragile result.

Does it makes sense?

While using packer, I noticed that my block devices that were marked with "delete_on_termination: false" were still being deleted on termination on EC2. I believe the problem is here in goamz, where in "addBlockDeviceParams" in "ec2.go" the request is built up with the following:

if k.DeleteOnTermination {
    params[prefix+"Ebs.DeleteOnTermination"] = "true"
}

This makes sense if the default for DeleteOnTermination was false, but according to http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/terminating-instances.html#preserving-volumes-on-termination

By default, we do the following:
...
Delete the volumes that you attach to your instance at launch, including the root device volume, when you terminate the instance
...

So, unless I'm mistaken, it seems there is no way to set DeleteOnTermination to false.

Because you can't use http.DefaultClient, you can't use http.Get()

I started working on some changes to allow you to create an http.Client and pass it to the SNS to use but now i'm getting 400 errors from AWS so I'm trying to figure out what's wrong.

I am trying to use packer with cn-north-1. I am stuck the following error:

2014/10/13 17:33:39 packer-builder-amazon-ebs: 2014/10/13 17:33:39 get { https://ec2.cn-north-1.amazonaws.com.cn/?AWSAccessKeyId=AKIAOTSXDIKK75XAFENQ&Action=DescribeImages&ImageId.1=ami-56bd2f6f&Signature=aO1ARGp7Za1dEYDwV7s29I97U0wiEqu6f6ByTJJy0vY%3D&SignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2014-10-14T00%3A33%3A39Z&Version=2014-06-15 } -> {
2014/10/13 17:33:40 packer-builder-amazon-ebs: 2014/10/13 17:33:40 response:
2014/10/13 17:33:40 packer-builder-amazon-ebs: 2014/10/13 17:33:40 HTTP/1.1 401 Unauthorized
2014/10/13 17:33:40 packer-builder-amazon-ebs: Transfer-Encoding: chunked
2014/10/13 17:33:40 packer-builder-amazon-ebs: Date: Tue, 14 Oct 2014 00:33:40 GMT
2014/10/13 17:33:40 packer-builder-amazon-ebs: Server: AmazonEC2
2014/10/13 17:33:40 packer-builder-amazon-ebs:
2014/10/13 17:33:40 packer-builder-amazon-ebs: fe
2014/10/13 17:33:40 packer-builder-amazon-ebs: <?xml version="1.0" encoding="UTF-8"?>
2014/10/13 17:33:40 packer-builder-amazon-ebs: <Response><Errors><Error><Code>AuthFailure</Code><Message>AWS was not able to validate the provided access credentials</Message></Error></Errors><RequestID>5625fd3a-c5ce-40aa-88eb-e29e7421d135</RequestID></Response>
2014/10/13 17:33:40 packer-builder-amazon-ebs: 0
2014/10/13 17:33:40 packer-builder-amazon-ebs:
2014/10/13 17:33:40 packer-builder-amazon-ebs:
2014/10/13 17:33:40 packer-builder-amazon-ebs: }
==> amazon-ebs: Error querying AMI: AWS was not able to validate the provided access credentials (AuthFailure)

I have verified that the same credentials works with the python boto client against cn-north-1. Could this be a signature issue? I wonder if V4 is needed.

Thank you for your awesome work with goamz and packer by the way!

The XML structure defined to parse out these values are wrong so they're not getting read in during a DescribeDBInstances call.