kablamooss / kombustion Goto Github PK

When passing in a template of the following format:

Resources:
    myResource:
        DeletionPolicy: Retain
        Type: AWS::Blah::Blah

DeletionPolicy does not appear in the generated output templates.

Using something like this https://github.com/rhysd/go-github-selfupdate

To enable a command like this:

kombustion upgrade [-y]

To pull the latest release from GitHub

Is your feature request related to a problem? Please describe.
To make it easier to install on macOS we should try to add kombustion to homebrew.

Describe the bug
When generating a template values that should be omitted are not, and are showing with null.

To Reproduce
Steps to reproduce the behavior:

1. kombustion generate stack.yaml

They should be omitted, and not shown if the values are null.
4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Smartphone (please complete the following information):

• Device: [e.g. iPhone6]
• OS: [e.g. iOS8.1]
• Browser [e.g. stock browser, safari]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

Needs to have the barebones framework to start writing a plugin.
Needs to be filled with documentation, in place.
Needs to have a working build script.
Needs to have example unit tests.

Describe the bug
In all cases default outputs are added. They should only be added when GenerateDefaultOutputs: true or --generate-default-output, -b is used.

To Reproduce
Run generate or upsert on a stack.

Expected behavior
Default outputs should not be added to a template when GenerateDefaultOutputs: true or --generate-default-output, -b is set to false. Ommision of this flag should also be treated as false.

Want an optional flag to upsert that lets you use a standard cloudformation params file. This flag should overwrite any other method of getting params, except for params added via a clie flag --param.

Describe the bug
Using a Tag with the official aws cloudformation stops the generation of the resource

To Reproduce
Steps to reproduce the behavior:

1. create a basic vpc.yaml

AWSTemplateFormatVersion: 2010-09-09
Description: The base VPC
Resources:
  Vpc:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: 10.0.0.0/16
      Tags:
        - Key: foo
          Value: bar

2. Run generate or upsert

> kombustion upsert stacks/vpc.yaml

Expected behavior
A basic cloudformtion should be created

current behaviour:
incorrect cloudformation with no resources

AWSTemplateFormatVersion: "2010-09-09"
Description: The base VPC for the ECS cluster

Resources: {}

Desktop:

• OS: mac OS High Sierra
• 0.3.7

This bug with cmp keeps coming up, we don't really need it. Should be safe to switch it.

cd /home/travis/gopath/src/github.com/KablamoOSS/kombustion; git submodule update --init --recursive
fatal: No url found for submodule path 'vendor/github.com/google/go-cmp' in .gitmodules
package github.com/KablamoOSS/kombustion/internal/parsers/resources: exit status 128
github.com/google/go-cmp (download)

Describe the solution you'd like
I would like the ability to generate a stack and a params file, which can be upserted with the aws cli. That is, a way to eject a stack from kombustion.

Additional context
This is a feature, that both prevents lock in, and would enable wider adoption by allowing user to use kombustion to generate a template file, while still using their existing tooling.

We already know the params at generate time, so this is possible.

We need tests for everything in ./internal/tasks.

The purpose of these files are to directly interface with the cli library, extract all necessary parameters and then invoke the functions that actually to the task.

Is your feature request related to a problem? Please describe.
I want to be able to decide whatkombustion generate write to. Ideally with a flag like --output ./compiled.

kombustion should make the folder if it doesn't exist.

Describe the bug
Currently basic support for !Ref isnt working. Whilst trying to parse in a single parameter the compiled output will have no Resources

To Reproduce
Steps to reproduce the behavior:

1. create a basic vpc.yaml

AWSTemplateFormatVersion: 2010-09-09
Description: The base VPC
Parameters:
  CidrBlock:
    Description: The range of IPs for the VPC
    Type: String
Resources:
  Vpc:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: !Ref CidrBlock

2. Run Generate

> kombustion upsert  --param CidrBlock=10.0.0.0/16 stacks/vpc.yaml

Current behaviour:

AWSTemplateFormatVersion: "2010-09-09"
Description: The base VPC for the ECS cluster
Parameters:
  CidrBlock:
    Description: The range of IPs for the VPC
    Type: String
Resources: {}

Expected behavior
A basic cloudformation with the reference value populated should be created.

Screenshots
NA

Desktop (please complete the following information):

• macOS High Sierra
• Version 0.3.6

Is your feature request related to a problem? Please describe.
CloudFormation Macros are mostly unsuported.

Describe the solution you'd like
The ability to upsert a template compatible with Macros.

Additional context
https://aws.amazon.com/about-aws/whats-new/2018/09/introducing-aws-cloudformation-macros/
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-macros.html

Describe the solution you'd like
Plugins can add to Resources, Mappings and Outputs, but not Parameters or Conditions. For many use cases the latter 2 are required.

The plugin manager system needs more extensive unit tests to cover the majority of the functionality.

To:

Double dash kebab case, or single dash single letter.

--param-name and -p

Plugins should not be able to load if their prefix is AWS:: or Custom

Github's unauthenticated API is rate limited to 60 req/minute (https://developer.github.com/v3/#rate-limiting).

Even with the aggressive plugin caching, its likely some users will hit this. So we need to ensure it's checked and handled.

The error should explain to the user why they hit the rate limit, and why we are currently using github and not our own registry for kombustion modules. (We may in the future, but there's a significant amount of work to build and maintain a proper secure registry, so right now we're using github.

If --stack-name is not passed, derive it from {{.manifest.name}}-{{.fileName}}-{{.enivronment}}

Describe the bug
Running the -v or --version command against kombustion built from the source returns "kombustion version" without a version number, as this version number is supplied as part of the Travis build and so only visible in releases.

To Reproduce
Steps to reproduce the behavior:

1. Build kombustion from source (sh init.sh)
2. Run ./kombustion -v in the project directory
3. You will receive "kombustion version" without a version

Expected behavior
A version number should be supplied, eg "kombustion version 0.6.1"

Additional context
Given that most people are expected to use the supplied releases and that versioning is handled in part of the build this is unlikely to impact most people, but I think it's worth having a default and consistently valid value returned from this command (as anyone who builds from source is quite likely to use this to test if the build was successful). I'm thinking something like "kombustion version BUILT_FROM_SOURCE", thoughts?

We want to add support to mirror the aws cli --capabilities argument. Currently the only capability is CAPABILITY_IAM, when there are more this will become a priority.

For now --capabilities CAPABILITY_IAM is available with the flag --allowIAMUpsert and -i.

Proposing we add the --capabilities flag with support for CAPABILITY_IAM.

And make --iam a shorthand flag that is functionally identical to --capabilities CAPABILITY_IAM.

Given how much CAPABILITY_IAM is needed when using the tool directly (like setting up accounts etc), this is an improvement to DX.

Describe the bug
The kombustion.yaml file will not work as expected when YAML comments are present

To Reproduce
Steps to reproduce the behavior:

1. Create as per the following

Environments:
  dev:
    AccountIDs:
    - "123456789012"
    Parameters:
      key: value

      # some comment
      key2: value2

2. Attempt to use key2

Expected behavior
key2 is resolved

Screenshots
N/A

Desktop (please complete the following information):

• OS: Mac
• Browser: N/A
• Version: 0.3.6

Additional context
N/A

Following on from #48

Basically, pull from the manifest, and remove all refs to environment.yaml

Describe the bug
I recently update to the last version 0.4.5

after trying to run my CF upsert I got:

✖ Error: No parser found

  | ☞ Resolution:
  | ├─ Name: PrivateLinkEndpoint
  | ├─ Type: AWS::EC2::VPCEndpointService
  | └─ Resolution: You may need to install a plugin to parse the resource.

If you run the stack yaml in version 0.4.1 it works

To Reproduce
Steps to reproduce the behavior:

1. wget -N -q https://github.com/KablamoOSS/kombustion/releases/download/v0.4.5/kombustion-linux-amd64.tgz && tar -zxvf kombustion-linux-amd64.tgz && ./kombustion --manifest-file params.yaml upsert cf/service.yaml --stack-name service
2. our service.yaml contains the following CF

   PrivateLinkEndpoint:
        Type: "AWS::EC2::VPCEndpointService"
        Properties:
            NetworkLoadBalancerArns: 
                - !Ref LoadBalancer
            AcceptanceRequired: "false"
    
    PrivateLinkPermissions:
        Type: "AWS::EC2::VPCEndpointServicePermissions"
        Properties:
            AllowedPrincipals: 
                - !Sub "arn:aws:iam::${AccountId}:root" 
            ServiceId: !Ref PrivateLinkEndpoint

3. See error

Expected behavior
The stack is created

We need more tests specifically to cover the templating engine, including it's interaction with plugins.

This overlaps #34 a bit. However it may make sense to extract the templating to it's own package at ./internal/templating

Is your feature request related to a problem? Please describe.
Tags can be used to track billing between parts of companies or services, as well as track a resource back to its source (via it's commit hash for example).

Describe the solution you'd like
I want to be able to define a set of tags per environment (distinct from parameters), that are added to every stack and resource.

Describe alternatives you've considered
The only real alternatives are adding tags to each resource in the cfn template. This is an area kombustion could really help, in reducing the amount of boilerplate code needed.

There isn't a way to define tags for stacks at the moment with kombustion.

The use case here is for example, setting up a logging service in all your accounts to push logs back to a single bucket.

You could have one environment in a manifest with all the accountID's, and then have a runtime cli flag that pushes to all accountID's in an environment.

Possibly have 2 stacks, one for the bucket and another for the logging service. Run the second upsert with the flag --deploy-to-all-accounts.

Problems:

• What role to assume in each account?
• We may need to pass the accountId's from the env object in the manifest as parameters to the stack

We need a full integration test with a plugin that can test plugin download, and the plugin api with a template.

To succeed the test should send a processed stack to a mock CFN endpoint.

Is your feature request related to a problem? Please describe.
When the accountID's are set for an environment in the manifest, actions should be restricted to those accounts, as a whitelist.

Describe the solution you'd like
When running a cloudformation actions upsert, delete, events, a check should be done to see if the account being used is whitelisted in the manifest.

If it is whitelisted, allow the action.

If it is not whitelisted, and there are accounts listed, deny the actions and explain to the user why they were denied.

If no accountID's are listed for the environment, any accountId is allowed to be used.
(optional), I think it would be good to throw a warning here, to encourage the practice of whitelisting. The rationale being that if you whitelist and account no warning is confirmation that all is well. However if you whitelisted the dev environment but not the prod environment, you could still upsert into the wrong environment without knowing.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Describe the bug
When installing more than a single plugin Kombustion will not find the first plugins but only the last one in the list in the Kombustion.yaml

To Reproduce

1. install the network plugin
2. install the serverless plugin
3. try and generate a stack

Expected behavior
A stack generation should run with no hassles using the stacks that were installed

Screenshots

➜  infrastructure kombustion add github.com/KablamoOSS/kombustion-plugin-network
➜  infrastructure kombustion add github.com/KablamoOSS/kombustion-plugin-serverless
➜  Add plugins
   └─ Found release v0.0.7 for KablamoOSS/kombustion-plugin-network
      └─ Adding plugin: github.com/KablamoOSS/kombustion-plugin-network
➜  Installing plugins
      └─ Installed github.com/KablamoOSS/kombustion-plugin-network                                  
      └─ Already installed github.com/KablamoOSS/[email protected][linux/amd64]
      └─ Already installed github.com/KablamoOSS/[email protected][linux/arm64]
      └─ Already installed github.com/KablamoOSS/[email protected][darwin/amd64]
      └─ Already installed github.com/KablamoOSS/[email protected][linux/386]
      └─ Installed github.com/KablamoOSS/kombustion-plugin-serverless
✔  Installed plugins
➜  infrastructure kombustion install                                            
➜  Install plugins
➜  Installing plugins
      └─ Already installed github.com/KablamoOSS/[email protected][darwin/amd64]
      └─ Already installed github.com/KablamoOSS/[email protected][linux/386]
      └─ Already installed github.com/KablamoOSS/[email protected][linux/amd64]
      └─ Already installed github.com/KablamoOSS/[email protected][linux/arm64]
      └─ Installed github.com/KablamoOSS/kombustion-plugin-network
      └─ Already installed github.com/KablamoOSS/[email protected][linux/arm64]
      └─ Already installed github.com/KablamoOSS/[email protected][darwin/amd64]
      └─ Already installed github.com/KablamoOSS/[email protected][linux/386]
      └─ Already installed github.com/KablamoOSS/[email protected][linux/amd64]
      └─ Installed github.com/KablamoOSS/kombustion-plugin-serverless
✔  Installed plugins
➜  infrastructure kombustion generate stacks/base.yaml                          
➜  Generate template
✖  Fatal: Plugin `github.com/KablamoOSS/kombustion-plugin-network` is not installed, but is included in kombustion.yaml
☞  Resolution: Run `kombustion install` to fix.

Desktop (please complete the following information):

• MacOS High Sierra 10.13.6
• Version 0.3.6

We want to update all error messages to be as human friendly as possible.

So when an error occurs, it should present the user with information on how to resolve it (if we know what the caused the error), or a link to submit an issue if we don't know what caused it.

This includes trying to explain as much as possible about the error, why it happened, and how to resolve (linking to a docs page if need be).

Ideally with a link to the documentation.

Error occurring when trying to upsert LoadBalancer Cloud Formation template.

Works fine when upserting with --noBaseOutputs flag.

Template example

Resources:  
  LoadBalancer:
    Type: AWS::ElasticLoadBalancingV2::LoadBalancer
    Properties:
      Name: "LoadBalancer"
      Scheme: "internal"
      Type: "application"
      Subnets: !Ref Subnets

Error

FATA[0021] ValidationError: Template format error: The Value field of every Outputs member must evaluate to a String.
	status code: 400, request id: 75ac6446-62fd-11e8-b7ae-cfc5ae02eec4 

Describe the bug
Create a DynamoDB table with no stream, and default outputs. Kombustion will try to create the Stream.Arn output, which will cause CFN to fail.

Expected behavior
Outputs should only be added for outputs that are in use.

Fn::Or:
  - Condition: SomeCondition

is converted to

Fn::Or:
  - SomeCondition

Most likely using this package github.com/blang/semver

We need tests to cover the majority of the functionality in ./internal/cloudformation.

Cloudformation API calls should be mocked (using testify), to ensure the right data is being sent to them.

It would be really good, to have an integration test upsert and delete a stack in a real aws account.

This should be doable on the free tier.

It should have the following tests:

• plain CFN
• with a simple plugin
• with a complex plugin

We may need a lambda to clean up the account periodically in case the test fails.

This test can likely be run out of travis on a cron.

Describe the bug

In case there is an !If clause in the body of the property of any Cloudformation resource, Kombustion transforms the template in such a way that null values are assigned to properties instead. Below is an example of how VersioningConfiguration in the S3 bucket resource AWS::S3::Bucket properties is handled improperly when the !If condition is employed in the body of its definition, producing a null as the value of the VersioningConfiguration.

To Reproduce

given this template in cf/alb-access-logs-s3.yaml

AWSTemplateFormatVersion: "2010-09-09"
Description: "Access logs S3 Bucket"
Parameters:
  AccessControl:
    Type: String
    Description: The canned access control list (ACL) that grants predefined
      permissions to the bucket.
    Default: ""
  BucketEncryption:
    Type: String
    Description: Whether server-side bucket encryption is on by default
    Default: "aws:kms"
    AllowedValues:
      - "false"
      - "aws:kms"
      - "AES256"
  BucketName:
    Type: String
    Description: Bucket Name (optional). If not provided CloudFormaiton will
      generate one
    Default: ""
  DenyInSecureQueries:
    Type: String
    Description: Whether to deny insecure requests to S3 bucket
    Default: "true"
    AllowedValues:
      - "true"
      - "false"
  GlacierTransition:
    Type: String
    Description: Glacier transitioning in days, leave blank for no transition
    Default: ""
  StandardIATransition:
    Type: String
    Description: Transitioning to standard IA in days, leave blank for no
      transition
    Default: ""
  VersioningConfiguration:
    Type: String
    Description: Enable versioning to prevent objects from being deleted or
      overwritten by mistake.
    Default: ""
  LoggingBucket:
    Type: String
    Description: Amazon S3 bucket where Amazon S3 store server access log files
    Default: ""
  LoggingPrefix:
    Type: String
    Description: prefix for the all log object keys in the S3 logging bucket
    Default: ""

Conditions:
  AccessControlIsDefined: !Not [ !Equals [ "", !Ref AccessControl ] ]
  BucketEncryptionIsOn: !Not [ !Equals [ "false", !Ref BucketEncryption ] ] 
  BucketNameIsDefined: !Not [ !Equals [ "", !Ref BucketName ] ]
  GlacierTransitionIsDefined: !Not [ !Equals [ "", !Ref GlacierTransition ] ]
  StandardIATransitionIsDefined: !Not [ !Equals [ "", !Ref StandardIATransition ] ]
  VersioningConfigurationIsDefined: !Not [ !Equals [ "", !Ref VersioningConfiguration ] ]
  LifecycleConfigurationIsDefined: !Or
    - !Condition GlacierTransitionIsDefined
    - !Condition StandardIATransitionIsDefined
  LoggingBucketIsDefined:  !Not [ !Equals [ "", !Ref LoggingBucket ] ]
  LoggingPrefixIsDefined:  !Not [ !Equals [ "", !Ref LoggingPrefix ] ]
  SecureTransportOnly: !And
    - !Condition BucketEncryptionIsOn
    - !Not [ !Equals [ "false", !Ref DenyInSecureQueries ] ]

Resources:
  S3Bucket:
    Type: "AWS::S3::Bucket"
    Properties:
      BucketName: !If
        - BucketNameIsDefined
        - !Ref BucketName
        - !Ref AWS::NoValue
      AccessControl: !If
        - AccessControlIsDefined
        - !Ref AccessControl
        - !Ref AWS::NoValue
      LifecycleConfiguration:
        Rules: !If
          - LifecycleConfigurationIsDefined
          -
            - Id: Archive to Glacier/StandardIA
              Prefix: ''
              Status: Enabled
              Transitions:
                - !If
                  - StandardIATransitionIsDefined
                  - StorageClass: STANDARD_IA
                    TransitionInDays: !Ref StandardIATransition
                  - !Ref AWS::NoValue
                - !If
                  - GlacierTransitionIsDefined
                  - StorageClass: GLACIER
                    TransitionInDays: !Ref GlacierTransition
                  - !Ref AWS::NoValue
          - !Ref AWS::NoValue
      VersioningConfiguration: !If
        - VersioningConfigurationIsDefined
        - Status: !Ref VersioningConfiguration
        - !Ref AWS::NoValue
      LoggingConfiguration: !If
        - LoggingBucketIsDefined
        - DestinationBucketName: !Ref LoggingBucket
          LogFilePrefix: !If
            - LoggingPrefixIsDefined
            - !Ref LoggingPrefix
            - !Ref AWS::NoValue
        - !Ref AWS::NoValue
      BucketEncryption:
        ServerSideEncryptionConfiguration: !If
          - BucketEncryptionIsOn
          -
            - ServerSideEncryptionByDefault:
                SSEAlgorithm: !Ref BucketEncryption
          - !Ref AWS::NoValue
      Tags: !If
        - BucketNameIsDefined
        -
          - Key: "Name"
            Value: !Ref BucketName
        - !Ref AWS::NoValue

  S3BucketPolicy:
    DependsOn: S3Bucket
    Type: "AWS::S3::BucketPolicy"
    Properties:
      Bucket: !Ref S3Bucket
      PolicyDocument:
        Statement:
          - !If
            - SecureTransportOnly
            - Action: s3:*
              Condition:
                Bool:
                  aws:SecureTransport: 'false'
              Effect: Deny
              Principal: '*'
              Resource: !Sub "arn:aws:s3:::${S3Bucket}/*"
            - !Ref AWS::NoValue
          - Action: s3:PutObject
            Effect: Allow
            Principal:
              AWS: arn:aws:iam::783225319266:root
            Resource:
              - !Sub "arn:aws:s3:::${S3Bucket}/AWSLogs/*"

Outputs:
  S3Bucket:
    Value: !Ref S3Bucket
    Export:
      Name: !Sub "${AWS::StackName}::S3Bucket"

  S3BucketPolicy:
    Value: !Ref S3BucketPolicy
    Export:
      Name: !Sub "${AWS::StackName}::S3BucketPolicy"

and the parameters in params/nonprod.yaml

Name: ""
Region: "ap-southeast-2"
Environments:
  phoenix:
    AccountIDs:
    - "111111111111"
    Parameters:
      BucketEncryption: "AES256"
      GlacierTransition: "60"
      StandardIATransition: "30"
      DenyInSecureQueries: "true"

The command

kombustion --manifest-file params/nonprod.yaml generate cf/alb-access-logs-s3.yaml --stack-name msp-phoenix-s3-access-logs --environment phoenix --param BucketName=msp-phoenix-s3-access-logs

produces a resulting template with the

        VersioningConfiguration:
        Status: null

the bit that gets transformed into it is

      VersioningConfiguration: !If
        - VersioningConfigurationIsDefined
        - Status: !Ref VersioningConfiguration
        - !Ref AWS::NoValue

Replacing shorthand intrinsics with the Fn::If etc does not resolve the issue. Only if the !If case is completely removed the resulting template is as expected.

Expected behavior

Should not produce null

Screenshots

None

Desktop (please complete the following information):

• OS: OS X Mojave 10.14.1
• Browser N/A
• Version v0.4.4.

Describe the bug
I get the following when deleting a stack:

DELETE_IN_PROGRESS
  | DELETE_IN_PROGRESS
  | DELETE_IN_PROGRESS
  | FATA[0035] Throttling: Rate exceeded
 
To Reproduce
Delete many stacks at the same time

Expected behavior
Should implement exponential backoff and not error out when this occurs

Is your feature request related to a problem? Please describe.
At the moment (v0.4.4) Kombustion does not error whenever it produces a template with any of the properties having a null value. It leads to a very cryptic error messages from CloudFormation during the deployment step (such as "Template format error: At least one Resources member must be defined.")

Describe the solution you'd like
An error with something like "one of the resources in the resulting template is null would be really nice

Any key:value pair that has a value of "Interface" will be stripped and removed from the CFN definition

  VPCEndpoint:
    Type: AWS::EC2::VPCEndpoint
    Properties:
      VpcId:
        Fn::ImportValue: !Sub "${VPCStack}-VPCid"
      ServiceName: !Ref ServiceEndpoint
      VpcEndpointType: Interface
      SubnetIds:
        - Fn::ImportValue: !Sub "${VPCStack}-Subnet-PerimeterInternal1"
        - Fn::ImportValue: !Sub "${VPCStack}-Subnet-PerimeterInternal2"
        - Fn::ImportValue: !Sub "${VPCStack}-Subnet-PerimeterInternal3"
      SecurityGroupIds:
        - Fn::ImportValue: !Sub "${SecurityStack}-SecGroup-${SecurityGroup}"

If you run through generate or upsert the key is removed from properties.

Currently effects 0.3.7 and 0.3.8, it could effect earlier releases.

Manifest should have an option: "generateDefaultOutputs"

Cli should have a flag --generate-default-outputs.

When enabled kombustion will add default outputs to all resources not defined by a plugin.

Describe the bug
Release build process does not push to docker hub.

Expected behavior
It should create a new image on a tag of the same name as the version.

The plugin manager currently only takes latest. We need to add SemVer version constraints, and an algo to parse and respect them.

Require that the core version be BUILT_FROM_SOURCE, to prevent arbitrary code injection with binaries for main line releases.

In addition if we do this, we can then add a hash check to ensure he loaded plugin is the plugin in the lockfile

Currently the cli functions are tied directly to the cli library. This makes it hard to test, and hard to reason about the functions.

We need an adapter layer in between.

So cli -> adapter -> function

The adapter should take the information from the cli call, and translate it into the function calls needed to complete the task.