Will there be !GetAtt support, or is this impossible?

In intrinsics/fngetatt.go it just returns nil.

Hello,

Thanks you for the library !!

I was wondering if there was documentation with examples that were planned / released ?

If not is there a process in place to submit documentation?

• format
• structure of exampels .
• PR's
• etc

Thanks

Hello,
Any idea on when the Fn::GetAtt intrinsic function shall be supported?
Cheers
Seth

e.g. when creating an auto scaling group. This is because an UpdatePolicy is defined inline with the Type and Properties fields, which goformation doesn't easily allow you to compose.

Given the following template:

Resources:
  Bucket:
    Type: AWS::S3::Bucket
Outputs:
  BucketName:
    Value: !Ref Bucket

and the following code:

package main

import (
        "fmt"

        "github.com/awslabs/goformation"
)

func main() {
        t, _ := goformation.Open("broken.yaml")
        y, _ := t.YAML()
        fmt.Println(string(y))
}

You get the following output:

Outputs:
  BucketName:
    Value: null
Resources:
  Bucket:
    Type: AWS::S3::Bucket

The Value of the Output has disappeared :(

This is just a dump of some thoughts I had while looking at this library. Feel free to ignore or take onboard.

deoxxa [11:30 AM]
hrmm, i do see some problems with it
e.g. comparing https://godoc.org/github.com/awslabs/goformation/cloudformation#AWSRDSDBInstance and https://godoc.org/github.com/crewjam/go-cloudformation#RDSDBInstance
the crewjam library uses pointers for just about everything, which means the default value is null, rather than the zero value for the type
the aws library doesn't use pointers, so if you leave out a parameter, you'll get the zero value, which sometimes has a distinctly different meaning to null
e.g. PubliclyAccessible there - if you leave it as null or don't put it in the template, it'll inherit its value from the vpc subnet group
if you explicitly set it to false, it'll never be public
likewise with true, it'll always be public
but the way amazon have defined those struct types, if you set it to false it won't go into the template
so if your subnet group has its default as true, and you set it to false in the struct, it'll end up being public

See for (reduced, simulated) example: https://play.golang.org/p/MJoSKm4ph2j

I'll preface this entire question by stating that I'm REALLY new to Go and might be doing something stupid, apologies in advance if that is the case!

In the main README of this repo, I attempted to follow the example for unmarshaling a CFN template (YAML based) into Go structs and ran into issues. Regardless of trying to inspect all values or just looking for certain types of resources, I end up with errors:

Looking for all resources: resource.Type undefined (type interface {} is interface with no methods)

Looking for certain resource type: function.Type undefined (type cloudformation.AWSEC2RouteTable has no field or method Type)

Tracking through the code, focusing up on just looping through all values in a template, I see that the Template struct are defined as map[string]interface{}, which should be arbitrarily populated by the JSON data (if I understand things correctly). This doesn't seem to be working as expected.

The only way I can get this to work as expected in the example is to modify my code as follows:

for name, resource := range template.Resources {
		resType := resource.(map[string]interface{})

		// E.g. Found a resource with name MyLambdaFunction and type AWS::Lambda::Function
		log.Printf("Found a resource with name %s and type %s", name, resType["Type"])

	}

I have to add that type assertion to get to the data, which isn't covered in the example. Is something not behaving as expected here or has something changed that isn't covered in the example anymore?

Is the DependsOn attribute on the road map or is it already supported some how?

New to go but I did not see any example of using Parameters

Issue

When I try
PS> go get github.com/awslabs/goformation

I get the error:
go build github.com/awslabs/goformation/cloudformation: C:\Go\pkg\tool\windows_amd64\compile.exe: fork/exec C:\Go\pkg\tool\windows_amd64\compile.exe: The filename or extension is too long.

Background

I currently have my $env:GOPATH as:

PS> $env:GOPATH
C:\godir 

The file path to cloudformation:
C:\godir\src\github.com\awslabs\goformation\cloudformation

Environment

PS> go version
go version go1.10.3 windows/amd64

Windows 10

Other

I notice this issue appears similar to #37 (which had been resolved). As per #37, I realize that the Go compiler can't accept long file paths on Windows.

Does anyone else get this issue (on Windows)?

I am using this library to read a SAM template, update the CodeUri for each function, and then render it back out.

Below is an example of what I am trying to do...

package main

import (
	"io/ioutil"
	"log"

	"github.com/awslabs/goformation"
	"github.com/awslabs/goformation/cloudformation"
)

const templateStr = `
AWSTemplateFormatVersion : '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: main.handler
      Runtime: python3.6
      CodeUri: .
      Events:
        MyEvent:
          Type: SNS
          Properties:
            Topic: MyTopic
`

func main() {
	template, _ := goformation.ParseYAML([]byte(templateStr))
	functions := template.GetAllAWSServerlessFunctionResources()

	for name := range functions {
		function := functions[name]
		newPath := "somewhere"
		codeURI := cloudformation.AWSServerlessFunction_StringOrS3Location{String: &newPath}
		function.CodeUri = &codeURI

		template.Resources[name] = &function
	}

	y, err := template.YAML()
	if err != nil {
		log.Fatalf("Failed to generate YAML: %s\n", err)
	}

	newTemplate := "new-template.yaml"
	if err = ioutil.WriteFile(newTemplate, y, 0644); err != nil {
		log.Fatalf("Failed to create new-template.yaml: %s\n", err)
	}
}

When I run this, the SNS event properties value in new-template.yaml is {}...

AWSTemplateFormatVersion: 2010-09-09
Resources:
  HelloWorldFunction:
    Properties:
      CodeUri: somewhere
      Events:
        MyEvent:
          Properties: {}
          Type: SNS
      Handler: main.handler
      Runtime: python3.6
    Type: AWS::Serverless::Function

In "CreationPolicy" type from cloudformation/policies.go there is a property named "ResourcesSignal" when it should be "ResourceSignal", otherwise the generated template is not correct.

--- a/cloudformation/policies.go
+++ b/cloudformation/policies.go
@@ -6,8 +6,8 @@ type CreationPolicy struct {
        // AutoScalingCreationPolicy specifies how many instances must signal success for the update to succeed.
        AutoScalingCreationPolicy *AutoScalingCreationPolicy `json:"AutoScalingCreationPolicy,omitempty"`
 
-       // ResourcesSignal configures the number of required success signals and the length of time that AWS CloudFormation waits for those signals.
-       ResourcesSignal *ResourcesSignal `json:"ResourcesSignal,omitempty"`
+       // ResourceSignal configures the number of required success signals and the length of time that AWS CloudFormation waits for those signals.
+       ResourceSignal *ResourceSignal `json:"ResourceSignal,omitempty"`
 }
 
 // AutoScalingCreationPolicy specifies how many instances must signal success for the update to succeed.
@@ -17,8 +17,8 @@ type AutoScalingCreationPolicy struct {
        MinSuccessfulInstancesPercent float64 `json:"MinSuccessfulInstancesPercent,omitempty"`
 }
 
-// ResourcesSignal configures the number of required success signals and the length of time that AWS CloudFormation waits for those signals.
-type ResourcesSignal struct {
+// ResourceSignal configures the number of required success signals and the length of time that AWS CloudFormation waits for those signals.
+type ResourceSignal struct {
 
        // Count is the number of success signals AWS CloudFormation must receive before it sets the resource status as CREATE_COMPLETE. If the resource receives a failure signal or doesn't receive the specified number of signals before the timeout period expires, the resource creation fails and AWS CloudFormation rolls the stack back.
        Count float64 `json:"Count,omitempty"`

Hi there,

First of all, as a Go (and cloudformation) enthusiast I really enjoy this project 😄.

But I have a little problem 😢.

AWSCognitoUserPool does not contain the related resource AWSCognitoUserPool_InviteMessageTemplate.

This doesn't work on Cloudformation neither. So it seems that InviteMessagleTemplate is in the wild with no real support. Any help/information about that ?

See PR #148

The following breaks sam validate with error invalid YAML template: json: unsupported type: map[interface {}]interface {}

Fails

Resources:
  Function:
    Type: AWS::Lambda::Function
    Properties:
      Name: !Join ['-', [{Ref: ApplicationId}, 'TenantPools', {Ref: DeploymentStage}]]

But if you replace !Join with regular object, it works fine.

Working fine

Resources:
  Function:
    Type: AWS::Lambda::Function
    Properties:
      Name: 
        Join: ['-', [{Ref: ApplicationId}, 'TenantPools', {Ref: DeploymentStage}]]

In CFN, IAM Policy conditions are represented in a PolicyDocument object with a Condition key.

Unfortunately, GoFormation interprets this as a Condition intrinsic and attempts to process it, fails and represents the output as null.

We are using a workaround based on string replacement, but would be good to solve properly.

As we've patched the go-yaml package, we need to:

1. Push new patched version to github.com/sanathkr/go-yaml
2. govendor remove gopkg.in/yaml.v2 && rm -rf vendor/gopkg.in/yaml.v2
3. govendor fetch github.com/sanathkr/go-yaml
4. Fork github.com/ghodss/yaml to github.com/sanathkr/yaml
5. Modify the fork to use github.com/sanathkr/go-yaml instead of gopkg.in/yaml.v2
6. govendor remove github.com/ghodss/yaml && rm -rf vendor/github.com/ghodss/yaml
7. govendor fetch github.com/sanathkr/yaml
8. Update imports in intrinsics/* to use github.com/sanathkr/yaml

I think that's it - let me know if you can think of a better way.

I am trying to obtain the latest release (v1.2.2). Using Go 1.11.4. Fetching this version results in failure.

> go get github.com/awslabs/[email protected]
go: finding github.com/awslabs/goformation v1.2.2
go get github.com/awslabs/[email protected]: unknown revision v1.2.2

Running the following command only fetches 1.2.1 version.

go get github.com/awslabs/goformation

Any ideas why could this be happening?

There are 2 discrepancies between the latest generated CloudFormationResourceSpecification and the CF docs.

Please see my 2 comment on #104

The following template is successfully parsed by goformation. It must fail:

Resources:
 ExampleFunction:
  Type: AWS::Serverless::Function
  Properties2**:
    Handler2: index.handler
    Runtime: nodejs6.10
    Events:
      Api:
        Type: Api
        iiProperties:
          Path: /
          Method: any 

Use Case

I would like to deploy resources to a Docker image running the local version of DynamoDB directly from my project's CloudFormation template for testing purposes. I'm using Serverless framework to deploy from my template directly to AWS, but to deploy to the locally running DDB image, it seems like the best approach is to create the tables using the AWS Golang SDK in my tests.

Problem

Currently I'm able to get an AWSDynamoDBTable by using goformation. To be able to create the tables on my local DDB instance in my tests however, I need a CreateTableInput from the SDK. It seems like there should be a way to easily convert from one to the other, but I can't seem to find anything.

Suggestion

A PR that converts the aws resources here into the Golang SDK equivalents.

Hello I am new to Go but not new to AWS. I have forked the repo and created a Parameter structs with a simple example.

need your mail id to clear doubts

AWS SAM added Globals

It seems like AWS SAM Local will probably rely on goformation to support this - aws/aws-sam-cli#226

I’ve really been enjoying using Goformation! I found a bug recently though that I’m not sure how to fix (I'm still learning Golang), but I tracked down the reason, and figured you might immediately know how to fix.

Summary:
The Fn::Join is not joining values together. I've traced this down to find that it is not reading the array inside the array. I believe the "goformation/intrinsics/fnjoin.go" file needs to be updated to loop the inner array. Let me know if you’d like me to put this in Github Issues

# TROUBLESHOOTING WALKTHROUGH

Example CF snippet from the example template in the repo:

Role:
  Fn::ImportValue:
   !Join ['-', [!Ref 'ProjectId', !Ref 'AWS::Region', 'LambdaTrustRole']]

If you try to unmarshall, and output the function.Role, it gives you nothing:

   > 2018/05/12 08:40:09 Found a AWS::Serverless::Function named GetHelloWorld (Role: )

Let's remove the ImportValue function:

   Role: !Join ['-', [!Ref 'ProjectId', !Ref 'AWS::Region', 'LambdaTrustRole']]

If you try to unmarshall this again, and output the function.Role as before, it gives you just "-":

   > 2018/05/12 08:40:34 Found a AWS::Serverless::Function named GetHelloWorld (Role: -)

So then for fun, I removed the inner array, and this worked:

   Role: !Join ['-', !Ref 'AWS::Region']

The output is:

   > 2018/05/12 08:42:03 Found a AWS::Serverless::Function named GetHelloWorld (Role: -us-east-1)

However, this syntax is in invalid.

I'm pretty sure the "goformation/intrinsics/fnjoin.go" file needs to loop the inner array.

Thanks!

New user here, seeing whether this is worth me using over troposphere.

I tried to add an Output to the template, but the godoc is rather opaque, merely stating map[string]interface{}. This is surprising since there are documented fields for an Output, but this doesn't seem to be represented in goformation. The same isn't true for troposphere, which will fail (albeit at runtime) if you pass an invalid key (eg typoing Descripption).

Do you really expect people to be typing:

template.Outputs["MyOutput"] = map[string]interface{}{
    "Description": "my desc",
    ....
}

I'm hoping I'm just missing the type definition somewhere, otherwise this is actually less safe than using troposphere :(

Looks like new event types for lambda's have been released but when go generate no new files have been touched. My guess is that the cloudformation specs have not been released yet. @PaulMaddox who should be nudged to get that out?

The following throws an error...

package main

import (
	"fmt"

	"github.com/awslabs/goformation"
)

const templateStr = `
AWSTemplateFormatVersion : '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: main.handler
      Runtime: python3.6
      CodeUri: .
      Tracing: true
`

func main() {
	template, _ := goformation.ParseYAML([]byte(templateStr))
	functions := template.GetAllAWSServerlessFunctionResources()

	for name := range functions {
		fmt.Println(name)
	}
}

The error:

ERROR: json: cannot unmarshal bool into Go struct field Properties.Tracing of type string

I'm trying to create an AWSApplicationAutoScalingPolicy. The ScalingTargetId has to be a string. I need to include a Ref to a resource defined in the template.

If I do

  ScalingTargetId: cloudformation.Ref("foo")

I get base64 encoded output in the YAML and I can't figure out what to put in to get the output to be

  ScalingTargetId:
    Ref: foo

Is this a bug or am I doing something wrong?

On a related note, I'm creating templates programmatically, but I'm having a lot of trouble testing for the expected output. Any hints about the right way to do that or pointers to a better place to ask?

When working IAM policies the type for the Policy Documents are labeled as JSON in the documentation are interfaces in the code. When trying to generate cloud formation documents how would I add a policy document? I'm guessing it needs to be a special struct with json tags for the items needed (version, statement, actions...)?

Problem Statement

We need to be able to unmarshal AWS CloudFormation resources to typed Go structs, for example:

type AWSLambdaFunction struct {
    Runtime string 
    ...
}

This is complicated, as within a CloudFormation template, a property (such as Runtime) could be a string, or it could be an AWS CloudFormation intrinsic function, such as:

MyLambdaFunction:
    Type: AWS::Serverless::Function
    Properties:
        Runtime: !Join [ "nodejs", "6.10" ]

The above example would fail to unmarshal into the AWSLambdaFunction struct above with json.Unmarshal() or yaml.Unmarshal().

Considerations

• Customers can nest intrinsic functions.
• We don't want to necessarily resolve intrinsic functions, but we should ensure that any intrinsic function is replaced with a primitive object (string).
• Intrinsic functions can be specified in YAML as a tag (e.g. !Sub), which isn't supported in the Go YAML parser today (they get replaced with blank strings).

Implementation

I propose we:

1. Submit a patch to the Go YAML parser to support tags, and provide a callback where the parser user choose what to replace for each tag.

2. We unmarhal all YAML templates into interface{}, then into JSON for handling within Goformation. This allows us to just focus on supporting a single template language in Goformation code.

I propose we implement a decoupled intrinsics handling package with a very simple interface that takes a JSON AWS CloudFormation template (as a []byte), processes all intrinsics, and returns the resulting JSON template (as a []byte).

This package will have a very simple API. Here is a usage example:

package main

import (
	"fmt"

	"github.com/paulmaddox/intrinsic-proposal/intrinsics"
)

const template = `{
	"Resources": {
		"ExampleResource": {
			"Type": "AWS::Example::Resource",
			"Properties": {
				"SimpleProperty": "Simple string example",
				"IntrinsicProperty": { "Fn::Join": [ "some", "name" ] },				
				"NestedIntrinsicProperty": { "Fn::Join": [ "some", { "Fn::Join": [ "joined", "value" ] } ] }
			}
		}
	}
}`

func main() {

	processed, err := intrinsics.Process([]byte(template))
	if err != nil {
		fmt.Printf("ERROR: Failed to process AWS CloudFormation intrinsics: %s\n", err)
	}

	fmt.Print(string(processed))

	// Go on to perform normal template unmarshalling here
	// ...

}

The resulting output of the above example would be:

{
  "Resources": {
    "ExampleResource": {
      "Type": "AWS::Example::Resource",
      "Properties": {
        "SimpleProperty": "Simple string example",
        "IntrinsicProperty": "-- refused to resolve Fn::Join intrinsic function --",
        "NestedIntrinsicProperty": "-- refused to resolve Fn::Join intrinsic function --"
      }
    }
  }
}

Below is an implementation of the intrinsic handling package I am proposing.
It's pretty simple, and recurses through a JSON structure (interface{}), looking for intrinsic functions. If it finds any, it calls out to a hook function that does whatever it needs to in order to "resolve" the intrinsic function back from an object, into a simple primitive.

This hook approach would allow us the possibility of resolving some intrinsic functions in the future if we wanted to, however in the initial implementation we would simply refuse to resolve any of the intrinsic functions, and just return a string value such as "unsupported intrinsic function: Fn::Join".

package intrinsics

import (
	"encoding/json"
	"fmt"
)

// IntrinsicHandler is a function that applies an intrinsic function  and returns the
// response that should be placed in the object in it's place. An intrinsic handler
// function is passed the name of the intrinsic function (e.g. Fn::Join), and the object
// to apply it to (as an interface{}), and should return the resolved object (as an interface{}).
type intrinsicHandler func(string, interface{}) interface{}

// IntrinsicFunctionHandlers is a map of all the possible AWS CloudFormation intrinsic
// functions, and a handler function that is invoked to resolve.
var intrinsicFunctionHandlers = map[string]intrinsicHandler{
	"Fn::Base64":      nonResolvingHandler,
	"Fn::And":         nonResolvingHandler,
	"Fn::Equals":      nonResolvingHandler,
	"Fn::If":          nonResolvingHandler,
	"Fn::Not":         nonResolvingHandler,
	"Fn::Or":          nonResolvingHandler,
	"Fn::FindInMap":   nonResolvingHandler,
	"Fn::GetAtt":      nonResolvingHandler,
	"Fn::GetAZs":      nonResolvingHandler,
	"Fn::ImportValue": nonResolvingHandler,
	"Fn::Join":        nonResolvingHandler,
	"Fn::Select":      nonResolvingHandler,
	"Fn::Split":       nonResolvingHandler,
	"Fn::Sub":         nonResolvingHandler,
	"Ref":             nonResolvingHandler,
}

// nonResolvingHandler is a simple example of an intrinsic function handler function
// that refuses to resolve any intrinsic functions, and just returns a basic string.
func nonResolvingHandler(name string, input interface{}) interface{} {
	result := fmt.Sprintf("-- refused to resolve %s intrinsic function --", name)
	return result
}

// Process recursively searches through a byte array for all AWS CloudFormation
//  intrinsic functions,
// resolves them, and then returns the resulting interface{} object.
func Process(input []byte) ([]byte, error) {

	// First, unmarshal the JSON to a generic interface{} type
	var unmarshalled interface{}
	if err := json.Unmarshal(input, &unmarshalled); err != nil {
		return nil, fmt.Errorf("invalid JSON: %s", err)
	}

	// Process all of the intrinsic functions
	processed := search(unmarshalled)

	// And return the result back as a []byte of JSON
	result, err := json.MarshalIndent(processed, "", "\t")
	if err != nil {
		return nil, fmt.Errorf("invalid JSON: %s", err)
	}

	return result, nil

}

// Search is a recursive function, that will search through an interface{} looking for
// an intrinsic function. If it finds one, it calls the provided handler function, passing
// it the type of intrinsic function (e.g. 'Fn::Join'), and the contents. The intrinsic
// handler is expected to return the value that is supposed to be there.
func search(input interface{}) interface{} {

	switch value := input.(type) {

	case map[string]interface{}:

		// We've found an object in the JSON, it might be an intrinsic, it might not.
		// To check, we need to see if it contains a specific key that matches the name
		// of an intrinsic function. As golang maps do not guarentee ordering, we need
		// to check every key, not just the first.
		processed := map[string]interface{}{}
		for key, val := range value {

			// See if we have an intrinsic handler function for this object key
			if handler, ok := intrinsicFunctionHandlers[key]; ok {
				// This is an intrinsic function, so replace the intrinsic function object
				// with the result of calling the intrinsic function handler for this type
				return handler(key, search(val))
			}

			// This is not an intrinsic function, recurse through it normally
			processed[key] = search(val)

		}
		return processed

	case []interface{}:

		// We found an array in the JSON - recurse through it's elements looking for intrinsic functions
		processed := []interface{}{}
		for _, val := range value {
			processed = append(processed, search(val))
		}
		return processed

	case nil:
		return value
	case bool:
		return value
	case float64:
		return value
	case string:
		return value
	default:
		return nil

	}

}

Reference Information

Possible intrinsic function formats (JSON):

List of intrinsic functions:

• Fn::Base64
• Fn::And
• Fn::Equals
• Fn::If
• Fn::Not
• Fn::Or
• Fn::FindInMap
• Fn::GetAtt
• Fn::GetAZs
• Fn::ImportValue
• Fn::Join
• Fn::Select
• Fn::Split
• Fn::Sub
• Ref

This makes goformation generate invalid serverless templates....

Fixed in #66

Hello,

Thanks for creating goformation, it has a lot of potential!

Currently parsing a template requires evaluation of intrinsic functions and those will only resolve if there are local parameters present otherwise to nil (eg Fn::Sub).

That behavior seems fine for some uses, but when attempting to use goformation as a parser/generator this isn't as helpful, since many parameters will not be available in the template and I want to preserve the intrinsic.

I'd like to be able to completely disable intrinsic evaluation and just leave the functions in place in the template..

Example parsing this bit:

Location:
  Fn::Sub: "s3://${BucketName}/codebuild/builds/${Revision}/${Service}/api.yaml"

And then calling template.YAML() should leave the instrinsic in place and output:

Location:
  Fn::Sub: "s3://${BucketName}/codebuild/builds/${Revision}/${Service}/api.yaml"

Looking at the code, a handler like:

func reallyNonResolvingHandler(name string, input interface{}, template interface{}) interface{} {
	return template
}

Does the trick when providing an override... I propose this is a better default nonResolvingHandler. Thoughts?

I now realize template has the entire tree.. I pushed up a proper fix which skips the intrinsic search altogether.

I have a patch that supports custom tags at sanathkr/yaml@fa65635. It works fairly well in the unit tests I wrote. But I want to test it against the crazy world of CloudFormation templates.

What is the best way for me to swap the original YAML parser with mine?

Summary:

It's currently not possible to build this project on Windows because of the length of some of the file names.

Steps to Reproduce

1. Configure a very short $env:GOPATH:
   PS>mkdir "C:\godir"
PS>$env:GOPATH = "C:\godir"
2. Attempt to "go get" the project:
   PS> go get github.com/awslabs/goformation

Expected Behavior

The package should be installed successfully

Actual Behavior

An error is returned:

go build github.com/awslabs/goformation/cloudformation: C:\Go\pkg\tool\windows_amd64\compile.exe: fork/exec C:\Go\pkg\tool\windows_amd64\compile.exe: The filename or extension is too long.

Environment

Windows

PS>[Environment]::OSVersion | fl

Platform      : Win32NT
ServicePack   :
Version       : 10.0.14393.0
VersionString : Microsoft Windows NT 10.0.14393.0

Go

PS>go version
go version go1.7.4 windows/amd64

Other Notes

I understand that this is technically an issue with the Go compiler as noted in golang/go#18468, but some of the file names in /cloudformation do seem to be excessively long. E.g.:

PS> gci .\cloudformation | Where-Object { $_.Name.Length -ge 80 } | select Name

Name
----
aws-applicationautoscaling-scalingpolicy_targettrackingscalingpolicyconfigur...  
awsserverlessfunction_s3eventorsnseventorkinesiseventordynamodbeventorapieve...  
awsserverlessfunction_stringoriampolicydocumentorlistofstringorlistofiampoli...  

I'm not familiar with go generate yet, but are there, perhaps, any flags which might allow for combining these types into a smaller set of files with shorter names?

The S3Key field on the AWSServerlessFunction_S3NotificationFilter type is a string, but in actual SAM templates it's an object with a Rules key containing an array of objects with Name/Value:

            Filter:
              S3Key:
                Rules:
                  - Name: suffix
                    Value: '.jpg'

The JSON schema files in this repo seem to reflect this correctly, so the problem may be with codegen?

I think this is a great library and here's what I love:

• it's all typed
• it's using Go, so it's fast
• it's readable
• it gets updated automatically as CloudFormation specs are published

But I wanted to use this for one of my projects and it just fails to hit the mark....
I think it's really time you try your own product, see the pain of just doing the following:

Template with:

• Parameters
• Mapping to get an AMI ID
• EC2 Instance
• 1 EBS Volume
• 1 Security Group

And it's currently impossible, but highly necessary to:

• Attach your EBS volume to your EC2 instance (uses Ref on Resources)
• Attach the security group to the EC2 instance (uses Ref on Resources)
• Supply CloudFormation Metadata Init to my EC2 instance in a type safe way (not available as a type)

I'm sure there are also tons of other gaps, but really try to show off an "example" in your repo that just does that - pretty basic stuff. Then you'll have hit something rock solid and I can start recommending this library

I usually try to do PRs but I won't have time for this, hope the feedback helps you in some way, and I really hope to achieve the above in the future

@PaulMaddox - the new intrinsic authoring support is fantastic! I'd like to propose a small enhancement:

The current work supports a subset of intrinsics but missing are condition intrinsics AND user defined intrinsic functions.

In the pre-process we take the default + the overrides to process intrinsics... I propose that we check the keys of these two maps to determine if intrinsic processing is supported in

Why is this useful?

In my use case, I use a program to process a stack.yaml and "decorate" with additional features.... I also take advantage of overrides to invent new intrinsics, which make goformation do interesting things.

for example:

// Looks up the location of a bazel target... opens the file and pushes json into the template
bazelIncludeHandler := func(name string, input interface{}, template interface{}) interface{} {
		if filename, ok := deps[input.(string)]; ok {
			// Load up the file
			file, err := ioutil.ReadFile(filename)
			exitIf(err)

			// Unmarshal as json
			var theJson interface{}
			err = json.Unmarshal(file, &theJson)
			exitIf(err)
			return theJson
		} else {
			exitIf(fmt.Errorf("Couldn't resolve file %s", input.(string)))
		}

		return nil
	}
	overrides["Fn::BazelIncludeJson"] = bazelIncludeHandler

Another powerful case is a true "no-op" intrinsic processor:

func noOpHandler(name string, input interface{}, template interface{}) interface{} {
	intrinsicFunc := map[string]interface{}{
		name: input,
	}

	b, err := json.Marshal(intrinsicFunc)
	if err != nil {
		panic(err)
	}

	return base64.StdEncoding.EncodeToString(b)
}

I can follow up with a PR... but since the map we need is in a different namespace I wanted your take first.

Maybe I am missing something, but in your example of marshaling JSON/YAML, the output does not contain the Type field under each resource. Is this expected? Thanks!

It would be helpful for the repo to create a new tagged release. Using the library with vgo results in an error because the v1.0.0 (Aug 2017) release's cloudformation/awsserverlessfunction_s3eventorsnseventorkinesiseventordynamodbeventorapieventorscheduleeventorcloudwatcheventeventoriotruleeventoralexaskillevent.go file results in too long of a file name. This issue looks to of been resolved at HEAD of the repo. Creating a tagged release for the more recent change that fixes the filename length changes would more easily allow the library to work with vgo.

Repo:

package main

import (
	"github.com/awslabs/goformation/cloudformation"
)

func main() {
	cloudformation.NewTemplate()
}

$ vgo build ./...
vgo: resolving import "github.com/awslabs/goformation/cloudformation"
vgo: finding github.com/awslabs/goformation/cloudformation latest
vgo: finding github.com/awslabs/goformation v1.0.0
vgo: finding github.com/ghodss/yaml 0ca9ea5df5451ffdf184b4428c902747c2c11cd7
vgo: finding gopkg.in/yaml.v2 eb3733d160e74a9c7e442f435eb3bea458e1d19f
vgo: finding github.com/onsi/ginkgo 8382b23d18dbaaff8e5f7e83784c53ebb8ec2f47
vgo: finding github.com/mitchellh/mapstructure d0303fe809921458f417bcf828397a65db30a7e4
vgo: finding github.com/onsi/gomega c893efa28eb45626cdaa76c9f653b62488858837
vgo: finding golang.org/x/net 1c05540f6879653db88113bc4a2b70aec4bd491f
vgo: finding golang.org/x/text e56139fd9c5bc7244c76116c68e500765bb6db6b
vgo: finding golang.org/x/sys c84c1ab9fd18cdd4c23dd021c10f5f46dea95e46
vgo: finding github.com/awslabs/goformation (latest)
vgo: adding github.com/awslabs/goformation v1.0.0
vgo: downloading github.com/awslabs/goformation v1.0.0
-> unzip /home/jasdel/Workspace/golang/src/mod/cache/github.com/awslabs/goformation/@v/v1.0.0.zip: open /home/jasdel/Workspace/golang/src/mod/github.com/awslabs/[email protected]/cloudformation/awsserverlessfunction_s3eventorsnseventorkinesiseventordynamodbeventorapieventorscheduleeventorcloudwatcheventeventoriotruleeventoralexaskillevent.go: file name too long
vgo: import "github.com/jasdel/canary/cmd/buildTemplate" ->
	import "github.com/awslabs/goformation/cloudformation": unzip /home/jasdel/Workspace/golang/src/mod/cache/github.com/awslabs/goformation/@v/v1.0.0.zip: open /home/jasdel/Workspace/golang/src/mod/github.com/awslabs/[email protected]/cloudformation/awsserverlessfunction_s3eventorsnseventorkinesiseventordynamodbeventorapieventorscheduleeventorcloudwatcheventeventoriotruleeventoralexaskillevent.go: file name too long

Docs

https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#s3-location-object

Model

https://github.com/awslabs/goformation/blob/master/generate/sam-2016-10-31.json#L197

Currently the Fn::Sub handler in intrinsics/fnsub.go only deals with:

{ "Fn::Sub": [ "input ${replacement}", [ "replacement": "some string" ] ] }

It also needs to deal with the case that !Sub is used for reference lookups.

{ "Fn::Sub": "input ${MyParameter}" }

It would be great to have an example or two around building resources that make use of the intrinsic functions inside templates. The cloudformation package seems to have no concept of them. Im unclear how to achieve this, but perhaps Im missing something.

Hi,
Are there any thoughts/plans about adding support for the Constraint Rules section?

Use Case:
I'm working on a tool for Quick Starts to automatically create some template constraint rules (https://docs.aws.amazon.com/servicecatalog/latest/adminguide/reference-template_constraint_rules.html) to prevent a template from launching in specific regions when it has resources that don't exist in those regions.

Amazon added the feature of SQS events for lambda functions a few weeks ago (https://aws.amazon.com/blogs/aws/aws-lambda-adds-amazon-simple-queue-service-to-supported-event-sources/) but with current goformation code this feature cannot be used.

Sam specification JSON should be updated to include this feature as it is defined here https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#sqs

Problem Statement

The AWS SAM specification has some fields that are polymorphic (aka, they can accept multiple types). A good example of this is the AWS::Serverless::Function.CodeUri property, which can accept either a string, or an S3 Object Location.

There are two challenges with this:

1. Defining polymorphism in a machine readable schema

Currently, v0.1.0 reads all resources from the AWS CloudFormation Resource Specification, and generates the Go files with the required structs for unmarshalling.

As AWS CloudFormation doesn't have any resources with polymorphic properties, the specification format doesn't include the ability to define them.

The specification currently uses the following fields for defining a property's type:

"PropertyName": {

    // If the property type is a single item of a primitive type, this will be used
    PrimitiveType: "",
    
    // If the property type is a map or list of a primitive type, this will be used
    PrimitiveItemType: ""

    // If the property type is a single item of a custom type, this will be used
    Type: "SomeCustomType",

    // If the property type is a map or list of a custom type, this will be used
    ItemType: "SomeCustomType"

}

2. Unmarshalling polymorphic properties into Go structs

Currently, for each AWS CloudFormation resource, Goformation auto-generates a struct with all of it's properties, like so:

// AWSLambdaFunction CloudFormation Resource (AWS::Lambda::Function)
type AWSLambdaFunction {
    Runtime string
    CodeUri string
    VpcConfig AWSServerlessFunction_VpcConfig
}

// AWSServerlessFunction_VpcConfig custom type
type AWSServerlessFunction_VpcConfig {
    SecurityGroupIds []string
    SubnetIds []string
}

This approach will not work for polymorphic properties such as CodeUri, which can be specified as a string, or S3 Bucket. This would cause the unmarshalling to fail for CodeUri above if an S3 Location was provided.

Implementation

In order to describe the AWS SAM language specification in a machine readable format, I propose we extend the AWS CloudFormation Resource Specification Format to include the following new fields:

{
    PrimitiveTypes: [ "String", "Number" ],
    PrimitiveItemTypes: [ "String", "Number"]
    Types: [ "SomeCustomType" ],
    ItemTypes: [ "SomeCustomType" ],
} 

These 4 new fields would allow us to describe polymorphic properties that allow multiple variations of primitives, lists/maps of primitives, custom types and lists/maps of custom types.

For example, to describe the AWS::Serverless::Function.CodeUri property, we could use:

{
    "ResourceSpecificationVersion": "2016-10-31",
    "ResourceTypes": {
        "AWS::Serverless::Function": {
            "Documentation": "https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction",
            "Properties": {
                "CodeUri": {
                    "Documentation": "https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction",
                    "Required": true,
                    "PrimitiveTypes": [ "String" ],
                    "Types": [ "S3Location" ],
                    "UpdateType": "Immutable"
                },
                ...
            }
        }
    },
    "PropertyTypes": {
        "AWS::Serverless::Function.S3Location": {
            "Documentation": "https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#s3-location-object",
            "Properties": {
                "Bucket": {
                    "Documentation": "https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#s3-location-object",
                    "Required": true,
                    "PrimitiveType": "String",
                    "UpdateType": "Immutable"
                },
                "Key": {
                    "Documentation": "https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#s3-location-object",
                    "Required": true,
                    "PrimitiveType": "String",
                    "UpdateType": "Immutable"
                },
                "Version": {
                    "Documentation": "https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#s3-location-object",
                    "Required": false,
                    "PrimitiveType": "Integer",
                    "UpdateType": "Immutable"
                }
            }
        }
    }
}

This helps us with challenge 2: unmarshalling of polymorphic types into Go structs. It allows us to auto-generate a struct, and helper struct like so:

// AWSServerlessFunction AWS CloudFormation Resource (AWS::Serverless::Function)
// See: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
type AWSServerlessFunction struct {

	// CodeUri AWS CloudFormation Property
	// Required: true
	// See: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
	CodeUri AWSServerlessFunction_StringOrS3Location `json:"CodeUri"`

    ...

}

// AWSServerlessFunction_StringOrS3Location is a helper struct that can hold either String or S3Location values
type AWSServerlessFunction_StringOrS3Location struct {
	String string
	S3Location AWSServerlessFunction_S3Location
}

// AWSServerlessFunction_S3Location AWS CloudFormation Resource (AWS::Serverless::Function.S3Location)
// See: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#s3-location-object
type AWSServerlessFunction_S3Location struct {

	// Bucket AWS CloudFormation Property
	// Required: true
	// See: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
	Bucket string `json:"Bucket"`

	// Key AWS CloudFormation Property
	// Required: true
	// See: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
	Key string `json:"Key"`

	// Version AWS CloudFormation Property
	// Required: true
	// See: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
	Version int `json:"Version"`
}

Proposed Actions

1. The above has been implemented (with tests), I will submit a pull request shortly. I've only implemented AWS::Serverless::Function, and only a few primitive and one polymorphic property in the specification.
2. Update SAM deployment pipeline to auto-generate and publish a schema to S3/CloudFront
3. Speak with CloudFormation team to see if they feel updating the main schema to support polymorphism would be appropriate, or whether we should continue to publish the AWS SAM specification separately.

Reference Information

AWS Serverless Application Model Specification

AWS CloudFormation Resource Specification

AWS CloudFormation Resource Specification Format

It looks like the specs have not been updated since 6/6/2018. In order to avoid having to open an issue every couple months to regenerate this, could this be automated on release by the CloudFormation team? Similar to how the AWS SDK's are generated simultaneously with a new release, why can't the same be done for this repo for CloudFormation releases?

Would be nice to add https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/parameters-section-structure.html#aws-ssm-parameter-types :-)

While working on #103, having go generate pull down latest JSON specs makes it hard, as you get new types and files, while you are trying to focus on subset of things that you are working on. Sometime rebasing on master solves the issue, but master maybe behind.

Hi,

How can I create a reference to an object, for example:

    "Subnet" : {
      "Type" : "AWS::EC2::Subnet",
      "Properties" : {
        "VpcId" : { "Ref" : "VPC" },
        "CidrBlock" : "10.0.0.0/24",
        "AvailabilityZone": "us-east-1a",
        "MapPublicIpOnLaunch": true,
        "Tags" : [ {"Key" : "Name", "Value" : "Subnet-By-CF" } ]
      }
    }

The cloudformation.AWSEC2Subnet VpcId property only accepts string.

Thanks.