rbalman / cfn-compose Goto Github PK

View Code? Open in Web Editor NEW

42.0 3.0 11.0 27.02 MB

compose your cloudformation stacks like docker compose

License: MIT License

Go 99.68% Makefile 0.32%

golang aws cloudformation iac orchestration

cfn-compose's Issues

Introduce better test coverage and define Quality Gate metrics

Add support for configurable concurrency mechanism

Summary
The number of concurrent go routines spun at the moment is directly related with number of flows being sent in the configuration. Example below configuration spins two go routines as there are two flows in the configuration. The internal logic of the spinning the go routine is, details here:

//Generate the worker pool as pre the flow counts
for i := 0; i < len(cc.Flows); i++ {
  go executeFlow(ctx, cfnTask, resultsChan, i)
}

Example

....
Flows:
  DemoSQS:
    <<: *default
    Description: Creates Demo SQS Queue
    Stacks:
    - template_file: sqs.yml
      stack_name: demo-{{ .ENV_NAME }}-sqs-queue
    Order: 0
    tags:
      type: "sqs"
  DemoRDSInstance:
    <<: *default
    Description: Demo RDS Instance
    Stacks:
    - template_file: rds.yml
      stack_name: demo-{{ .ENV_NAME }}-rds-instance
    Order: 0
    parameters:
      EngineVersion: '12.4.0'

Acceptance Criteria:
When running cfnc:

the default behavior should spin the go routine as per the count of the flows but
we should be able to override the behavior by passing a flag or an environment variable.
the current concurrency configuration should be some how visible to the end user

Update Licensing Policy

Add the features to pass down the values from one stack to other

Introduce Configurable Retry Mechanism

Summary
If we deploy massive amount of the CloudFormation stacks concurrently then we might encounter rate throttle limits like below:

Acceptance Criteria:
In the scope of this issue we would like to see:

What is the breaking point where rate throttle issue starts to appear
Then see if we can tweak AWS SDK configuration or make changes on our side to introduce retry mechanism
The retry mechanism should be able to configurable through the configuration file

Setup SonarCloud for shifting code quality and security left

SonarCloud enables team to deliver clean code consistently and efficiently with a tool that easily integrates into the cloud DevOps platforms and extend your CI/CD workflow.

Integration PR: #8

Support resuable configuration snippets options in cfn-compose.yml file

Right now the sample configuration looks sth like shown below where we are not sure if we can make use of defaults and override any specific configuration from individual flow. The scope of the issue is to:

find out if it is supported by default with yaml configuration
if not see what can we change in our side to include this and propose if it is worth doing or not.
- if feasible then we should implement the change

Current Status

Description: Demo CloudFormation Compose file
Vars:
  ENV_NAME: demo
  ENV_TYPE: nonproduction
  AWS_PROFILE: demo
Flows:
  DemoSQS:
    Description: Creates Demo SQS Queue
    Stacks:
    - template_file: sqs.yml
      stack_name: demo-{{ .ENV_NAME }}-sqs-queue
      parameters:
        EnvironmentName: '{{ .ENV_NAME }}'
        EnvironmentType: '{{ .ENV_TYPE }}'
      tags:
        EnvironmentName: '{{ .ENV_NAME }}'
        EnvironmentType: '{{ .ENV_TYPE }}'
    Order: 0
  DemoRDSInstance:
    Description: Demo RDS Instance
    Stacks:
    - template_file: rds.yml
      stack_name: demo-{{ .ENV_NAME }}-rds-instance
      parameters:
        EnvironmentName: '{{ .ENV_NAME }}'
        EnvironmentType: '{{ .ENV_TYPE }}'
      tags:
        EnvironmentName: '{{ .ENV_NAME }}'
        EnvironmentType: '{{ .ENV_TYPE }}'
    Order: 0
  DemoEc2Instance:
    Description: Deploy Demo EC2 Instance
    Stacks:
    - template_file: ec2.yml
      stack_name: demo-{{ .ENV_NAME }}-ec2-instance
      parameters:
        EnvironmentName: '{{ .ENV_NAME }}'
        EnvironmentType: '{{ .ENV_TYPE }}'
      tags:
        EnvironmentName: '{{ .ENV_NAME }}'
        EnvironmentType: '{{ .ENV_TYPE }}'
    Order: 1

Expected

Description: Demo CloudFormation Compose file
Vars:
  ENV_NAME: demo
  ENV_TYPE: nonproduction
  AWS_PROFILE: demo

default: &default
  parameters:
    EnvironmentName: '{{ .ENV_NAME }}'
    EnvironmentType: '{{ .ENV_TYPE }}'
  tags:
    EnvironmentName: '{{ .ENV_NAME }}'
    EnvironmentType: '{{ .ENV_TYPE }}'

Flows:
  DemoSQS:
    <<: *default
    Description: Creates Demo SQS Queue
    Stacks:
    - template_file: sqs.yml
      stack_name: demo-{{ .ENV_NAME }}-sqs-queue
    Order: 0
    tags:
      type: "sqs"
  DemoRDSInstance:
    <<: *default
    Description: Demo RDS Instance
    Stacks:
    - template_file: rds.yml
      stack_name: demo-{{ .ENV_NAME }}-rds-instance
    Order: 0
    parameters:
      EngineVersion: '12.4.0'
  DemoEc2Instance:
    <<: *default
    Description: Deploy Demo EC2 Instance
    Stacks:
    - template_file: ec2.yml
      stack_name: demo-{{ .ENV_NAME }}-ec2-instance
    Order: 1

Add Readme and documentation

Manual approval required for workflow run 2713494923: Deploying first version in prod

Workflow is pending manual review.
URL: https://github.com/BalmanRawat/cfn-deploy/actions/runs/2713494923

Required approvers: [balmanrawat]

Respond "approved", "approve", "lgtm", "yes" to continue workflow or "denied", "deny", "no" to cancel.

Add support for running the cfnc in docker containers

Summary
Right now to run the cfnc command either you need to install Golang and compile or install the compiled cfnc binary. Both of them might not be the right choice if you don't want to install and run it as a docker container.

In the scope of this story:

let's dockerize the cfnc as a executable container
publish it in the registry and update the CI
finally update the documentation

eg. outcome: the help command should work as below

docker run cfnc --help

Manage cloudformation stacks at scale. Design and deploy multiple cloudformation stacks either in sequence or in prallel using declarative configuration

Usage:
  cfnc [command]

Available Commands:
  completion  Generate the autocompletion script for the specified shell
  config      Generate, validate and visualize the compose configuration
  deploy      Deploys the stacks based on the sequence specified in the compose configuration
  destroy     Destroys all the stacks in the reverse order of creation
  help        Help about any command

Flags:
  -c, --config string     File path to compose file (default "cfn-compose.yml")
  -d, --dry-run           Run commands in dry run mode
  -h, --help              help for cfnc
  -l, --loglevel string   Specify Log Levels. Valid Levels are: DEBUG, INFO, WARN, ERROR (default "INFO")
  -v, --version           version for cfnc

Use "cfnc [command] --help" for more information about a command.

Add other multiple installation methods for different OS

The installation instructions at Readme should include:

brew for macos
script for linux
script for windows

Manual approval required for workflow run 2713527779: Deploying first version in prod

Workflow is pending manual review.
URL: https://github.com/BalmanRawat/cfn-deploy/actions/runs/2713527779

Required approvers: [balmanrawat]

Respond "approved", "approve", "lgtm", "yes" to continue workflow or "denied", "deny", "no" to cancel.

Manual approval required for workflow run 2713527779: Deploying first version in prod

Workflow is pending manual review.
URL: https://github.com/BalmanRawat/cfn-deploy/actions/runs/2713527779

Required approvers: [balmanrawat]

Respond "approved", "approve", "lgtm", "yes" to continue workflow or "denied", "deny", "no" to cancel.

Introduce Proper Release Pipeline

Introduce subcommands to better play/manage cfn-compose.yml configuration

When things are declarative, things become configuration heavy. It will always be better and productive if we can play with config options by validating, listing the dependencies.

Replace Order Based dependency by dependsOn property

Create more friendly loggging mechanism like serverless compose

Introduce more robust variable ingestion mechanism

Summary
Example Compose file:

Description: Sample CloudFormation Compose file
Vars:
  ENV_NAME: cfn-compose
  ENV_TYPE: nonproduction
  SUBNET_ID: "subnet-033274e18559e7bde"
  VPC_ID: "vpc-0001e3b703212c9cb"
Flows:
  SecurityGroup:
    Order: 0
    Description: Creates Security Group
    Stacks:
    - template_file: sg.yml
      stack_name: sample-{{ .ENV_NAME }}-security-group
      parameters:
        EnvironmentName: '{{ .ENV_NAME }}'
        EnvironmentType: '{{ .ENV_TYPE }}'
        VpcId: '{{ .VPC_ID }}'
      tags:
        EnvironmentName: '{{ .ENV_NAME }}'
        EnvironmentType: '{{ .ENV_TYPE }}'

  EC2Instance:
    Order: 1
    Description: Deploying EC2 Instance
    Stacks:
    - template_file: ec2.yml
      stack_name: sample-{{ .ENV_NAME }}-ec2-instance
      parameters:
        EnvironmentName: '{{ .ENV_NAME }}'
        EnvironmentType: '{{ .ENV_TYPE }}'
        SubnetId: '{{ .SUBNET_ID }}'
      tags:
        EnvironmentName: '{{ .ENV_NAME }}'
        EnvironmentType: '{{ .ENV_TYPE }}'

Right now we use vars section in the cfn-compose.yml file to supply go template variables and replace with them dynamically in the rest of the template configuration. Since the template and variables are declared in the same template we need to parse cfn-compose.yml file twice.

one just to scan the vars as key value pairs and
other to replace the template variable in the compose configuration.

It might be better to separate vars section as a separate file to have better control, extensibility and independence for future use cases.

rbalman / cfn-compose Goto Github PK

cfn-compose's Issues

Recommend Projects

Recommend Topics

Recommend Org