Remove any legacy or unused vars and secrets from the Task Definition

Summary

Add a CloudWatch alarm that is triggered when there is a high number of failed Cognito sign-in attempts to catch account brute forcing attempts.

Steps

1. Update the authorize method to log the Cognito exception type to the Form's CloudWatch log group.
2. Create a metric filter and CloudWatch alarm based on this filter:

resource "aws_cloudwatch_log_metric_filter" "cognito_signin_exceeded" {
  name           = "CognitoSigninExceeded"
  pattern        = "NotAuthorizedException: Password attempts exceeded" # adjust based on logging in step 1 
  log_group_name = var.ecs_cloudwatch_log_group_name

  metric_transformation {
    name          = "CognitoSigninExceeded"
    namespace     = "forms"
    value         = "1"
    default_value = "0"
  }
}

resource "aws_cloudwatch_metric_alarm" "cognito_signin_exceeded" {
  alarm_name          = "CognitoSigninExceeded"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = "1"
  metric_name         = aws_cloudwatch_log_metric_filter.cognito_signin_exceeded.metric_transformation[0].name
  namespace           = aws_cloudwatch_log_metric_filter.cognito_signin_exceeded.metric_transformation[0].namespace
  period              = "60"
  statistic           = "Sum"
  threshold           = "10" # this could also be adjusted depending on what we think is a good threshold
  treat_missing_data  = "notBreaching"
  alarm_description   = "Cognito - multiple failed sign-in attempts detected."

  alarm_actions = [var.sns_topic_alert_warning_arn]

  tags = {
    (var.billing_tag_key) = var.billing_tag_value
    Terraform             = true
  }
}

Test

Perform multiple failed login attempts and expect the alarm to trigger.

Implement resources to resolve following checkov policy "Ensure Domain Name System (DNS) query logging is enabled for Amazon Route 53 hosted zones"

Search for "CKV2_AWS_39" in the infra repository in order to see places that requires modifications.

Implement resources to resolve following checkov policy "Ensure IAM policies does not allow write access without constraints"

Search for "CKV_AWS_111" in the infra repository in order to see places that requires modifications.

The templates Lambda function currently returns the bearerToken.

For security purposes, the bearerToken should only be returned using the /api/id/[form]/bearer API via the nextJS client.

• Bump devcontainer version
• Bump Terraform plan/apply workflow versions

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Pending Approval

These branches will be created by Renovate only once you click their checkbox below.

• chore(deps): update actions/checkout action to v4
• chore(deps): update actions/github-script action to v7
• chore(deps): update google-github-actions/release-please-action action to v4
• chore(deps): update public.ecr.aws/lambda/nodejs docker tag to v20
• chore(deps): update terraform github.com/cds-snc/terraform-modules to v9
• fix(deps): update dependency json2md to v2
• fix(deps): update dependency notifications-node-client to v8
• fix(deps): update dependency uuid to v9
• 🔐 Create all pending approval PRs at once 🔐

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• chore(deps): update all non-major docker images (localstack/localstack, localstack/localstack-pro, mcr.microsoft.com/vscode/devcontainers/base, public.ecr.aws/lambda/nodejs)
• chore(deps): update all non-major github action dependencies (actions/checkout, actions/create-github-app-token, aws-actions/amazon-ecr-login, aws-actions/configure-aws-credentials)
• chore(deps): update all patch dependencies (@types/aws-lambda, @types/node, @types/uuid, axios, github.com/cds-snc/terraform-modules, random, typescript)
• fix(deps): update all minor dependencies (@aws-sdk/client-dynamodb, @aws-sdk/client-lambda, @aws-sdk/client-rds-data, @aws-sdk/client-s3, @aws-sdk/client-secrets-manager, @aws-sdk/client-sqs, @aws-sdk/lib-dynamodb, @types/node, aws, font-awesome, hashicorp/terraform, typescript)
• chore(deps): lock file maintenance
• Click on this checkbox to rebase all open PRs at once

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

As file uploads become more and more popular the runtime for Lambda functions is also increasing. The current default is 3 seconds however there has recently been an instance of the timeout being hit when processing a form with 2 pdf files.

Increase Lambda Execution timeout to 5 min (300 seconds)

We currently have an issue keeping two separate terraform repos as moving changes through staging to production is cumbersome and error prone.

Terragrunt will allow for the declaration of separate stand alone modules allowing for easy injection of variables that differ between the staging and production environment.

Requires complete refactoring of terraform code for extraction of modules and variables.

Project that have already gone this path:
Notifications
Covid Alert Metrics

Hi! I saw that you're using tfswitch. It does not support Terragrunt and OpenTofu, so, I suggest you to switch to tool that can manage all of these tools together: https://github.com/tofuutils/tenv

Related to this discussion

We should use an Alpine based Docker image (NodeJS compatible) instead of the default AWS Lambda NodeJS one. This way we would reduce the size of our Lambda images. Furthermore, we would benefit from a runtime environment that has way less opened vulnerabilities.

More information in the discussion linked at the beginning.

Add an archive that stores processed form submissions for a set amount of time before purging expired submissions. If there is ever an issue with a client (system crash, inadvertent deletion, etc.) we need a process to be able to provide the client with form submission backups.

Potential solution:

• Once a form submission is processed move the raw submission to an S3 bucket that is set to purge entries older than x days (check with policy for exact value).

We had a case where the response archiver lambda was timing out in both Production and Staging and there was no way for us to know that other than logging into the AWS admin console and manually checking the logs.

Create CloudWatch alarms so that a notification is sent on Slack when a Lambda times out.

Ran into this while testing a Branding upload form on Staging.

If you're logged in on Staging, you can see the form here: https://forms-staging.cdssandbox.xyz/en/form-builder/edit/clfgzqv7705358bx84qjps43y

Alternatively, here's a JSON form definition you can use: https://github.com/cds-snc/platform-forms-client/pull/1755/files

If you try to upload a file with spaces (or presumably other special characters that require escaping), the following error will cause it to fail in the reliability queue: Failed to retrieve files from reliability storage because of following error: Request path contains unescaped characters..

Here is a form submission that failed:

{"formID":"clfgymbb801888bx8jgnaqqif","language":"en","responses":{"1":"CA","2":"CAFR","3":"https://canada.ca/","4":"https://canada.ca/","7":"form_attachments/2023-03-20/13830c12-bce6-4817-a576-15e68683924e/Screenshot 2023-03-20 at 11.09.23 AM.png","8":"form_attachments/2023-03-20/2ab76646-1dc4-48c5-9659-dd2096587a2a/Screenshot 2023-03-20 at 11.09.23 AM.png"},"deliveryOption":{"emailAddress":"[[email protected]](mailto:[email protected])"}}

Summary

Checkov has identified the following issues with the RDS cluster:

- CKV2_AWS_8  # RDS enable Query Logging
- CKV2_AWS_27 # RDS setup AWS Backup plan 

Once the existing AWS resources have been imported into the new Terraform state, we should attempt to fix them.

Related

• #101

When completing a form with validated fields, if the validation errors are triggered on any field the Dynamic Row only renders a single row instead of the x number of rows that were completed before the submission attempt.

Before Validation	After Validation

Acceptance Criteria:

• On render after error validation the dynamic row should appear as it did before the submission event
• All error references are able to reach their intended fields (For example a dynamic row with more than one row should still be reachable by the error summary at the top of the page).

Create a filter on the github action that ensures it only creates PR Review Environments for PR's that do not include database migrations/changes.

Implement resources to resolve following checkov policy "Ensure no IAM policies documents allow "*" as a statement's resource for restrictable actions"

Search for "CKV_AWS_356" in the infra repository in order to see places that requires modifications.

Summary

The following steps will be required to migrate the existing AWS resources from the Terraform remote state to the new Terragrunt module remote states.

Staging

• Merge new changes from forms/aws into Terragrunt modules
• List resources in Terraform state
• Taint the ECS service
• Terraform import all resources into respective Terragrunt module remote states
• Run Terragrunt apply for all modules and ensure no unexpected changes
• Remove forms/aws folder and terraform.yml workflow from repo

Production

• Compare Production and Staging Terraform and merge any changes required
• List resources in Terraform state
• Taint the ECS service
• Terraform import all resources into respective Terragrunt module remote states
• Run Terragrunt apply for all modules and ensure no unexpected changes
• Archive forms-production-terraform repo

Finish

• Rename this repo to forms-terraform

Notes

The aws_ecs_service.form_viewer must be tainted and recreated, otherwise CodeDeploy prevents any changes to it.

cd "env/$ENV/app"
terragrunt taint aws_ecs_service.form_viewer

⚠️ This will cause a few minutes of downtime in the environment.

Related

• #28

Context:
pg client throw an unhandled error with connection terminated unexpectedly .
throw er; // Unhandled ‘error’ event Error: Connection terminated unexpectedly

attempt PR to fix this issue PR

Thread

Summary

Currently we only have health checks from the CDS status checker, which performs a health check every 10-15 minutes.

We should add Route53 health checks and CloudWatch alarms on these checks so that we have health checks every 30 seconds and are alerted in our ops channel if the site goes down.

Related to this discussion

When implementing containerized Lambda functions we had a discussion concerning the mutability of our ECR repositories. The recommendation is to make them immutable so that there is no chance an image is being mistakenly overridden.
In our current infra, for simplicity, we decided to have all Lambda function point to the tag named latest and every time we deploy a new version we tag it with both the commit SHA and latest tags.
There is an ongoing discussion in this ticket aws/containers-roadmap#878 to see if AWS can support immutability expect for latest which would become some kind of a pointer to the newest image. If this gets implemented then we would be able to tweak our implementation so that we switch our repositories to being immutable.

More information in the discussion linked at the beginning.

Transition our Redis cluster from using a consistent reserved capacity (cache.t2.micro) to a serverless engine.

Pros:

• Reduced costs
• Ability to scale independently
• Leverage latest Redis engine (7.2)
  • Allows to leverage new JSON type instead of converting form configurations to strings.

Cons:

• Request application mitigation so that there is no downtime during the Production release.

There is currently no warning when the Audit Log deadletter queue receive messages. To ensure no logs are lost a warning should be issued or a redrive should be enabled.

Leverage the ability of the WAF to inject headers into a request so that the Application is aware the request is from outside the country and can trigger alerts as required.

Acceptance Criteria:

• WAF adds specific header to the request when the request is detected from outside of the accepted Geo Zone.
• Application verifies header during the JWT callback to ensure that any authenticated action is done from within the accepted Geo Zone.
• Application produces different alarms based on forbidden action:
  • Sign In at Cognito Level from outside Geo Zone
  • Sigin In Mfa level from outside Geo Zone
  • Authenticated action from outside Geo Zone.

Problem: Currently end users will either be presented with a 500 page when the GCForms application is no longer functioning or is undergoing maintenance.
Work: Add a maintenance page to inform users that the application is not available

Involves:

• Creating a route53 check that verifies if the app is running
• Created a CloudFront CDN end point
• Create a primary and fallback route53 entry for GCForms
• Create a static site (en and fr) that users will be directed to when the route53 entry is in fallback mode

Description

Currently the ElastiCache Redis replication group does not have encryption enabled in transit or at rest. This has a low impact because the data is only for feature flags, but it does cause the following 3 checkov failures around encryption and authorization:

- CKV_AWS_29  # Elasticache enable at-rest data encryption
- CKV_AWS_30  # Elasticache enable in-transit data encryption
- CKV_AWS_31  # Elasticache enable in-transit data encryption and auth token  

Once the Terragrunt conversion has been applied to Staging, we should check the impact of enable at-rest and in-transit encryption.

Notes

Terraform settings for the AWS provider:

• elasticache_replication_group#at_rest_encryption_enabled
• elasticache_replication_group#transit_encryption_enabled

Related

• #98

Summary

Create a Lambda function that triggers a scan of new objects created in an S3 bucket. This will be a proof-of-concept for the proposed asynchronous solution in Design Doc 008: File upload for API and UI + content scanning:

Acceptance criteria

• The function should be triggered automatically by S3 s3:ObjectCreated:* events.
• The S3 object that triggered the scan should be updated with metadata related to the scan: IN_PROGRESS and FAILED_TO_SCAN.
• The function should also respond to SNS event triggers from the Scan Files service to update the S3 object metadata with the scan result.

Summary

Enabling Shield Advanced will provide DDoS protection to our resources.

An example of how to do this can be see in Articles:

• Enroll resources
• Add DDoS alarms

Notes

There is no additional cost for this - the subscription is paid once at the org account level.

Implement resources to resolve following checkov policy "Ensure RDS Clusters are encrypted using KMS CMKs"

Search for "CKV_AWS_327" in the infra repository in order to see places that requires modifications.

This will bring some QOL changes to our PRs that should improve the reviewing experience.

https://github.com/cds-snc/terraform-plan

The Staging and Production environments received multiple Fuzzy attacks per day with requests similar to the following:
-Requested and resolved page mismatch: /%0ASet-Cookie%3Acrlfinjection/.. /

• Requested and resolved page mismatch: /en/./RestAPI/Connection /en/RestAPI/Connection
• Requested and resolved page mismatch: /./RestAPI/LogonCustomization /RestAPI/LogonCustomization

Ensure that only known and valid url paths are allowed through the Firewall.
AWS Resources:

• WAF Rules
• WAF Regex Sets

Acceptance Critera

• WAF rule blocks invalid or non-existent paths before reaching the application

Summary

Add the S3 scan object Terraform module to the forms-${var.env}-vault-file-storage bucket to enable ClamAV file scanning on all newly created S3 objects.

The module behaves as follows:

1. When an S3 object is created in the bucket, it triggers a scan using Scan Files ClamAV endpoint.
2. This marks the S3 object scanning status as in_progress using object tags.
3. Once Scan Files has completed the scan, it publishes the scan verdict to an SNS topic, which causes the lambda to update the S3 object tag with the scan verdict.

Related

• #219

When fetching the template during reliability queue processing also retrieve the delivery method. This ensures that if a user needs to change an email address due to technical problems that responses are not left in limbo with the old information.

TF:

• Modify reliability queue to fetch delivery option when pulling template from DB
• getTemplateFormConfig in /lambda/reliability/lib/templates.js
  App:
• Remove inclusion of deliveryOption in Lambda Call in api/submit
• Remove unused function getTemplateDeliveryOptionByID in lib/templates

PR #336 added the functionality to store Audit Logs in DynamoDB. The logs are currently set with a TTL of 1 year, which means after 1 year they'll be automatically deleted from the DynamoDB database.

Before that event occurs an archiving solution that reads from the DynamoDb stream for the AuditLogs table that filters on TTL deletions must be put in place in order to save the deleted items in S3.

See AWS reference material for solution: Reference material

TFSec just released an official github action that we should change to, the triat action has some issues regarding how it actually works.

https://github.com/aquasecurity/tfsec-pr-commenter-action

This action will also comment back issues into the PR.

Remove egress from ECS task over port 443. The NextJS server does not need to initiate calls to external services.

Summary

Checkov identified the following issue with the load balancer:

- CKV_AWS_91 # Load balancer enable access logging

Once the existing AWS resources have been imported into the new Terraform state, we should attempt to fix them.

Related

• #107

Implement resources to resolve following checkov policy "Ensure Domain Name System Security Extensions (DNSSEC) signing is enabled for Amazon Route 53 public hosted zones"

Search for "CKV2_AWS_38" in the infra repository in order to see places that requires modifications.

Make sure there is a protection in place to avoid any unintentional DynamoDB table deletion

If ever one of our outbound service calls (Notify, etc.) are down the form submissions will eventually be sent to the Dead Letter Queue. (DLQ). In a situation where Notify can be down for a few hours that could mean 100's of form submissions in the DLQ. A lambda is needed in order to automatically process the DQL and resend those messages back through the reliability queue once Notify service is active again. This lambda would be manually triggered by an AWS admin/dev once confirmed the DQL is ready is playback.

Implement resources to resolve following checkov policy "Ensure that the Execution Role ARN and the Task Role ARN are different in ECS Task definitions"

Search for "CKV_AWS_249" in the infra repository in order to see places that requires modifications.

Add a general rate-limiting rule to complement our post-request rate-limiting rule.

Investigation is required to determine an appropriate cutoff value.

Summary

Currently we do not have load balancer access logs setup. As a result, it's difficult to trace a user request to our service into the ECS form viewer task logs.

This will be enabled as part of the Cloud Based Sensor PR in #173.

On every terraform apply a new ECS Task is created and the older versions remain in AWS. This can lead to thousands of ECS Task Definitions existing in our AWS account with the frequency that we push.

Implement a solution that only keeps the last 5 ECS Task Definitions and removes all previous versions.

Now that the IRCC project is confirmed to no longer move forward we need to remove this temporary functionality. While the mailing list functionality may come back in the future it will need to be redesigned.