hamlet-io / engine Goto Github PK

• add docs url attribute to apigateway
• add url attribute to cache
• switch to using occurrence exclusively wen generating the standard environment settings
• add optional subcomponent variable for containers

In OSX 10.13.1 the bash version is still quite old (around 3.2) which doesn't support the global declare option (-g) required when generating templates

Side by side installations of bash are recommended instead of replacing the default OSX bash shell.

PBI https://github.com/codeontap/codeontap/issues/23
When an API Gateway is removed the S3 bucket needs to be emptied otherwise the cloudformation script fails and the content of the docs s3 bucket has to be cleared manually.

Since the contents of this bucket are completely created at run time, they can just be deleted.

Currently lambda invoke permissions for an API Gateway deployment are applied as part of the API Gateway solution deployment. This means that if you recreate the Lambda function the permissions are lost and have to be applied manually

Instead the permission should be applied as part of the lambda deployment and the API Gateway attributes are passed to the permissions via a link configured on the Lambda function.

For generating blueprints, or whole of level templates (like iam, lg, dashboard etc), we need to have access to the state of all occurrences simultaneously.

At present, appsettings and credentials are only loaded for the current deployment unit.

• change assembly process for appsettings and credentials to include all units
• associate appsettings and credentials to an occurrence via the deployment unit of the occurrence
• also support changing association to be occurrence based rather than deployment unit based as it is now. Both will be supported for now but use of deployment unit based appsettings will be deprecated
• use getCompositeObject to support hierarchical appsettings/credentials (tenant, account, product, segment, occurrence, du)

id_ecs is using the wrong function to generate the log group id

Shift all netowkr related settings on a tier to a "Network" attribute, and add an enabled/disabled flag. Subnets will only be generated for a tier if it is enabled.

Epic https://github.com/codeontap/codeontap/issues/14

In order to simplify the management of generation framework versions, and to make it easier for multiple versions to be used across products and environments, template generation will now be done inside a docker container. As with other docker images in codeontap, it will accept a workspace via an "indir" directory, and any output will be generated in an outdir (typically mapped to a temporary directory).

For now the docker images should use a java base image to support freemarker. We will include any basic tools the gen3 scripts use, such as dos2unix. Later we can add things like Terraform.

There will be a dockerhub entry for gen3, with builds triggered by new tags on the gen3 repo. Minor releases will happen on a regular cycle (initially every three months). These correspond to the "train" idea that Spotify discuss on their videos.

New version releases will be offset to calendar quarters to avoid December/June periods (Christmas, EOFY) 1-February, 1-May, 1-August, 1-November.

We will also have weekly "candidate" releases. They will reflect the next minor release.

So assuming 5.6.0 is released on 1-May, the weekly releases will then be 5.7.0-rc1, 5.7.0-rc2 ... until 1-August when 5.7.0 will be released. During a three monthly period, emergency patches on the last "train" will increment the patch level e.g. 5.6.1.

Move all the id and name assembly boilerplate into these convenience methods.

getKey does the same as formatId to its inputs so calls to it can be refactored in the same way.

RDS has the ability to provide SNS notifications when events occur on an RDS cluster, this includes events like failovers, patching, reboots etc.
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_Events.html

This could be integrated with KTLG if it supports forwarding events via an HTTP call or post.

Remove deprecated policy id and name "Stem" variables in createTask macro.

Policies should use the policyResourceId and policyResourceName variables.

Currently we allow a single custom message, the verification message for cognito user pool configuration. There are actually 2 messages that can be sent from a user pool, the verification message to confirm SMS or email addresses and an invite message when an admin signs a user up that contains the password details for the account.

We need to add support for seperate messages on each one.

Changes to stack management are resulting in stacks where the update is unsuccessful but a rollback is completed as being successful overall.

A rollback should be a trigger for marking the process as failed.

At present we have a single log group for an entire ECS cluster but this makes searching logs more difficult than it would be if each service had its own group

To better reflect the purpose of slices, we will change the term used from "Slice" to "DeploymentUnit".

This term more closely aligns to what slices are used for.

The task will change the term expected by the cloud formation template generation framework. Internally the logic will look for "DeploymentUnits" attributes. In order to provide a smooth transition for existing solutions that use "Slices", addDefaults.jq (which adds Id and Name attributes if not present) will be updated to convert "Slices" attributes to "DeploymentUnits" attributes.

As far as scripts are concerned, we will continue to use the letter "s" for now when a deployment unit is provided, but internally the scripts should use the term DEPLOYMENT_UNIT instead of SLICE.

We will also add a -u option where ever there is a -s option to allow us to phase out the use of -s.

Looks like solution template doesn't behave well if no credentials has been defined for the database.

Add support for vpc endpoints as part of the vpc deployment unit for segments.

It will be added automatically by default. We can add a control latter to turn it off if required.

Remove the deprecated -s switch from createTemplate.sh and managestacks.sh

As noted in #99 when a link is referenced the region is currently set based on the context of where the link is being referenced from instead of the region of the linked item. The majority of our components are hosted within the same region for a single product, but as more services are added to the linking functionality this will need to be looked at.

Containers don't need the tier or component name in their name as this is inherent in the service/task name. They can thus be simplified to just the container.Name. As part of this change, move the naming of containers into the createTask macro, which will mean removing the Name attribute from any existing application container files.

While we check acceptance of build commits when promoting code, we don't confirm that the lower environment was generated using the same version of the framework about to be used in the higher environment. This can allow issues to slip through when frameworks are upgraded.

In order to mitigate this,

• include the generation framework version in the metadata of all templates
• when promoting, ensure that all the templates in the lower environment were built with the version of the framework about to be used in the higher environment

Currently the ContentNode Path Attribute is returning the path configuration instead of the determined path which is the intention of the attribute details.

Add a configuration setting to control whether minor version upgrades are permitted in the maintenance window.

Support a per environment default, with it turned off in production environments and enabled otherwise.

When cloudformation processes an update changes can sometimes require the replacement of the resource that is created. When it comes to an RDS instance this is quite common, but its also the place where you want it the least as RDS data often needs to persist

Within the cloudformation configuration you can specify a snapshot identifier, which allows for an RDS instance to be created using the snapshot.

Using this configuration when an update is requested that requires replacement, a template can be provided which specifies a snapshot to restore from. This allows for changes to made on the RDS instance without losing data.

generation framework version - v5

Error occurred during deploy job:

22:42:01 SEVERE: Error executing FreeMarker template
22:42:01 FreeMarker template error:
22:42:01 When calling macro "getOccurrences", required parameter "root" (parameter #1) was specified, but had null/missing value.
22:42:01 
22:42:01 ----
22:42:01 Tip: If the parameter value expression on the caller side is known to be legally null/missing, you may want to specify a default value for it with the "!" operator, like paramValue!defaultValue.
22:42:01 ----
22:42:01 
22:42:01 ----
22:42:01 FTL stack trace ("~" means nesting-related):
22:42:01 	- Failed at: #function getOccurrences(root, deploy...  [in template "common.ftl" in function "getOccurrences" at line 851, column 1]
22:42:01 	- Reached through: #include compositeList?ensure_starts_...  [in template "codeontap/jenkins/workspace/cotw-ktlg/cotp-mitm/cots-integration/4-Deploy/ktlgnp01/composite_resource.ftl" in macro "includeCompositeLists" at line 452, column 25]
22:42:01 	- Reached through: @includeCompositeLists asArray(compos...  [in template "codeontap/jenkins/workspace/cotw-ktlg/cotp-mitm/cots-integration/4-Deploy/ktlgnp01/composite_resource.ftl" in macro "cfTemplate" at line 479, column 9]
22:42:01 	- Reached through: @cfTemplate level="application" compo...  [in template "createApplicationTemplate.ftl" at line 4, column 1]
22:42:01 ----

Caused by non-empty Tasks element in the ECS definition in solution.json file.

• add Mandatory indicator on all attributes - default false
• add frequency indicator - so attribute rules then apply to each unique child of the parent, rather than being expected as direct child attributes of the parent

This will allow explicit configuration in a range of situations where currently we leave the parent model open (e.g. Containers and Links). It will also support aggregation of children e.g. where a common set of links is required, along with some specific ones.

Add mask.

The generation framework contains all the information of what needs to be deployed and what has been deployed. Using this information we should be able to create a template which outputs a blueprint document outlining

• The Tenant structure - Products, Environments, Solutions, Segments, tiers
• The components running within the tenant structure
  • The configuration of the components
  • The current runtime attributes of the components

This blueprint can then be used to generate documentation as required.

Add the alternate name forms currently in includeCompositeLists (resource/start.ftl) to the occurrence structure and switch to using these via the occurrence structure.

Remove the corresponding global variables from includeCompositeLists .

Add the URL and FQDN details for an SPA so that they can be referenced by documentation and other components.

Maintenance windows for RDS define when minor work can be scheduled and performed by AWS.
By default the maintenance window is set by AWS and can be different for each RDS instance. So that we can report and identify maintenance windows from the template configuration we should set the window using the PreferredMaintenanceWindow parameter.

Since maintenance windows can be used on other services, we should have a codeontap maintenance window configuration which staggers across the environments, segments and tiers.

• Environments: staggered across days of the week
• Segments: staggered across half day of the environment day (morning/night)
• Tiers: staggered by the hour of the segment half day

This way updates are applied on a predictable, rolling basis.

While we don't use EC2 based NAT Gateways anymore we would like to make sure that they are patched in the same way as the other instances. The same yum update security and cron job structure needs to be added to NAT EC2 scaling groups

https://github.com/codeontap/gen3/blob/822adfa26e552b4b20ed6fb6f0fd03913de2b01b/aws/templates/segment/segment_vpc.ftl#L652

Lambda has the ability to trigger lambda functions based on a given event that occurs on a Cognito user pool

https://docs.aws.amazon.com/cognito/latest/developerguide/cognito-user-identity-pools-working-with-aws-lambda-triggers.html

This should be reasonably straight forward to add to the framework using links. When a cognito solution is processed, using the link names we whould be able to match the functions to the appropriate cognito action.

Best practice bash scripting suggests scripts shouldn't have an extension.

PBI https://github.com/codeontap/codeontap/issues/8
AWS System Set Manager can be used to automate tasks on Ec2 based or server instances within an environment.

The key task that we want to support is for automated patch management

the RunId is generated to create a run specific Id which can be used as a reference between generation templates within a run.

Currently the runId can contain uppercase and lower case letters along with numbers. To make them more consistent with other Id's generated they should only return lowercase characters

This will allow code to perform back off on processing failure. Needs

sqs:ChangeMessageVisibility and sqs:GetQueueAttributes

Cache should use occurrences.

Both should use the Resources of the occurence.

Some systems connect to a database using a Single URL which specifies all of the configuration properties of the database.

E.g. Django accepts the following format as a URL
DATABASE_URL=postgres://user:password@dbfqdn:5432/dbname

We have all the details available to create this URL string and it is used by other frameworks as well.
This also allows us to set and pass the database engine details to the application as well.

When a replace template is processed if it has no changes on the current deployment a failure is being raised which cancels the rest of the manage environment or deploy. Instead it should notify that no changes are required and not fail.

This is to protect against memory leaks or processes whose memory footprint grows over time (e.g. more user data means more memory consumed).

The default will be two times the reservation. If an explicit maximum memory figure is provided, the default will not apply.

When it goes GA,

• review/update the startup repo to accommodate both variants
• update templates to use Linux 2 for new builds, but leave existing ones as they are

Use occurrence state to determine queue id/name

Currently the segment/solution/application templates are getting quite large and unwieldy.

In order to make them more manageable, I want to split the support for each slice/resource type into its own freemarker template, and assemble these dynamically, in the same way we are currently assembling the ECS container definitions.

This will also permit new AWS resource types to be dynamically added, allowing a new type to be developed and then migrated into the gen3 repo.

We'll do createSegmentTemplate.ftl as a test of the idea.

Under templates, we will have a directory for segment, solution, and application. We will out each slice into its own file "segment_*.ftl" and save in the segment directory. We will also

Each file will handle one slice, and will generate the resulting outputs. Each will include a case structure with a "mode" to select between "definition" and "outputs".

If required, we can add a start/end file as well. Finally all the definitions/macros at the start can be moved to their own include as they are mostly reusable across the different template types. We may need to adjust this file slightly to check for variable presence in some cases.

setContext.sh will also need to be modified to assemble the segmentList in the same manner as it does for container definitions, with the presence of a definition at the segment, then solution level overriding the same file in the generation framework. This permits modifications to be tested in a project before incorporating them more generally into the framework.

Using the Blueprint from #132 create an interactive diagram which shows the components that have been deployed or configured. When the components are selected display the deployment attributes and the configuration of the component.

• Use HEREDOCS for usage
• Use HERESTRINGS instead of "echo ... | "
• Use input redirection not "cat" command for a single input file
• Direct errors to stderr (>&2)

When processing environment variables for Lambda and ECS the environment variable for the Subcomponent is currently being set as a hash

            "Environment": [
              {
                "Name": "SUBCOMPONENT",
                "Value": {
                  "Id": "app",
                  "Name": "app"
                }
              },

the environment variables can only be set to string so this is causing a failure

Add an "all", "produce" and "consume" role to the sqs state.

Permissions can then be configured in the solution for sqs rather than needing explicit entries in the container fragment

Provide a single command that allows access to all codeontap cli functions. This is a precursor to dockerising codeontap (see #130) This script will become the default entrypoint.

It will provide switches for all the usual suspects (TENANT, PRODUCT, ENVIRONMENT, SEGMENT, TIER, COMPONENT, DEPLOYMENT_UNIT before checking for a verb and a noun. Subsequent switches will be specific to the verb/noun combination.

The command will try to find the root of the context tree and then work from there. When used in docker, the working directory will be indir, which is expected to be mapped as a volume to the root of the tree to be processed. We will add a marker fil in the image so that if indir isn't mapped, we can put out an error.

Allows other services to discover the published API Docs URL and as a result of this the swagger spec that the API Gateway uses