Relates #33.
If role_arn is configured on the provider, pass its value as RoleArn to CloudAPI operations.

Description

Relates hashicorp/terraform-plugin-framework#63.
As of v0.1.0 terraform-plugin-framework does not support terraform-plugin-log.

Definition of Done

Once upstream support is added, enhance this provider and remove calls to log.Printf.

Description

Relates hashicorp/terraform-plugin-framework#68.

As of v0.1.0 terraform-plugin-framework does not support per-resource, per-operation timeouts.
Once upstream support is added, enhance this provider.
In the meantime, ensure that the Cloud API waiter timeout values are high enough to cover all supported resource operations.

Definition of Done

CloudFormation properties that are not required but are create-only should be emitted as Optional/Computed/ForceNew attributes.
For example CloudWatch Logs log group name.
Currently we are missing the Computed flag on these attributes.
ForceNew requires #44.

Description

Given a source directory of CloudFormation Resource Schema files from #15, an initial code generator should be written that creates corresponding Terraform Resource files. These files can use the output of #7 (or an empty/placeholder map) for schema and write empty/placeholder CRUD handler definitions (which will be more filled in via #8).

Dependent on: #17

Definition of Done

• Running a generation command (e.g. make gen and/or go generate ./...) (re)creates all Terraform Resource files based on available CloudFormation Resource Schema files.
• Ensure CI verifies that the code generation is synchronized on pull requests and the main branch

Description

Currently we have no support for generating data source schemas.
Eventually we will want to add both singular and plural data sources (as Cloud API supports List operations).

Definition of Done

• Resources with get by id functionality are supported with a singular datasource

Description

Practitioners using the CloudAPI-based provider should be able to declare the AWS Region for all resources/operations at the provider level. Currently the AWS Region is hardcoded to us-west-2 for the beta period. This configuration should be required as there will be no default after beta.

Proposed Configuration

provider "awscloudapi" {
  region = "us-west-2"
}

Otherwise if not provided, reads AWS_DEFAULT_REGION environment variable.

Definition of Done

• Provider schema includes region attribute. If configured, CloudAPI operations are called against that region's endpoint.
• Supports AWS_DEFAULT_REGION environment variable. If configured, CloudAPI operations are called against that region's endpoint.
• If both region provider configuration and AWS_DEFAULT_REGION environment variable are provided, configuration takes precedence.

Description

CloudAPI operations support an optional RoleArn parameter, which is an IAM Role that is used to do the actual provisioning (otherwise it defaults to the principal calling the operation). We should implement support for an optional string argument at the provider level, which can be passed through to all resource-level CloudAPI operations.

Proposed Configuration

provider "awscloudapi" {
  role_arn = "arn:aws:iam::123456789012:role/CloudAPIExecutionRole"
}

Definition of Done

• Provider schema includes role_arn attribute. If configured, all CloudAPI operations are called with the RoleArn parameter and this configuration value. Should provide IAM Role ARN format validation (e.g. arn.Parse(), service equals iam, etc).

Description

Relates hashicorp/terraform-plugin-framework#53.
As of v0.1.0 terraform-plugin-framework does not support Terraform Sets.
This limits the resource schemas that can be generated.
Once upstream support is added, enhance this provider.

Definition of Done

Description

Ensure that from day one of alpha the GitHub repository configuration matches HashiCorp standards and as much as possible is entirely managed by Terraform. Where possible these should be consistent with the existing provider assuming that standard makes sense.

Definition of Done

• labels
• issue templates

Description

The CloudFormation resource type schemas write-only. These are properties, such as passwords, that can be specified on resource creation but aren't returned when reading the current state of the resource.

In this provider's generic Read handler, the new Terraform state value is constructed from the returned CloudFormation ResourceModel without regard for any write-only properties.
Write-only properties should be copied from the prior Terraform state value into the new state value.

This will involved capturing the paths to all write-only properties during schema code generation and retaining this information in the generic resource type structure.

Definition of Done

Description

We currently use the terraform-plugin-sdk's resource.StateChangeConf structure and associated WaitForState method to poll the CloudAPI GetResourceRequestStatus method waiting for a resource operation to complete.
The AWS Go v2 SDK has a powerful waiter mechanism but as of today the CloudAPI preview SDKs we have received have not had waiters implemented for GetResourceRequestStatus.
It looks like we should be able to implement our own waiter for this operation.
Use for example NewTypeRegistrationCompleteWaiter as a guide.

Definition of Done

• Implement AWS Go v2 SDK-native waiter

Relates #34.

To use the current terraform-plugin-sdk acceptance test framework a top-level attribute named id must be declared in the resource schema. This is required for the state shim, otherwise errors are reported:

TF_ACC=1 go test ./internal/aws/logs -v -count 1 -parallel 20 -run=TestAccLogGroup_ -timeout 180m
=== RUN   TestAccLogGroup_basic
=== PAUSE TestAccLogGroup_basic
=== CONT  TestAccLogGroup_basic
    testing_new_config.go:111: no "id" found in attributes
    testing_new.go:53: no "id" found in attributes
--- FAIL: TestAccLogGroup_basic (3.70s)

The resource code generation should produce an attribute

		"id": {
			Description: `The unique resource ID.`,
			Type:        types.StringType,
			Computed:    true,
		},

and the generic resource handlers should populate this field in state.

Relates: #38.
Relates: hashicorp/terraform-plugin-framework#80.

Description

This provider implements generic CRUD handlers for all supported resource types.
The single code paths for each operation give us the opportunity to enable metrics consistently.
Consider using https://github.com/armon/go-metrics.

Definition of Done

Description

In theory we can generate acceptance tests based on the CloudAPI schema definition in order to fully validate the behavior of the generated provider and protect against both local and upstream regression.

The terraform-plugin-framework does not currently provide testing support, however as the existing SDK relies on binary testing it should be possible to use the terraform-plugin-sdk to enable these tests to be run.

However, given the generated nature of the provider, and that the generation code is really the subject under test we may choose to rely mainly on unit tests. The detailed per-attribute, full lifecycle tests that we do for the existing provider, may not be appropriate given the vast range of services on offer from AWS and that the provider code is meant to be generic across services.

Definition of Done

• We understand whether the terraform-plugin-sdk will allow us to test this provider.
  That we have in Internal RFC (with team alignment) which specifies:
• Our testing strategy for the generated provider
• If we choose to pursue generated acceptance tests, an agreed scope for 0.1, 0.2 and 1.0 release.

Description

Validate that we can test the existing release process and that it works for this new provider.

Check with Registry Team and Release team what needs to be done to ensure that the first release works, and how it can be testing without making an artifact public.

Decide what the process is for pre-GA builds vs post.

Some notes from Megan:

• https://hashicorp.slack.com/archives/CBG2PFGCE/p1622835922047000?thread_ts=1622814368.041900&cid=CBG2PFGCE (i think this was for logo and display name in the registry)
• this was the changelogs-links.sh change i was mentioning the other day: https://github.com/hashicorp/terraform-provider-googleworkspace/pull/69/files
• then just make sure your repo is public (i'm not sure this is necessary? but alex thinks it is)
• Make sure Terraform-Release-Bot has access to your repo
  Post Release GitHub Actions
• changelog-newversion

Builds should be repeatable, and be one click.

Relates #35

Definition of Done for Testing Release Tooling:

• Create Release GithubAction
• Create Post-Release GithubAction (generates new changelog etc.)
• Ensure repo has access to necessary environment vars per TF-279
• configure HC-releases with a private (or public) bucket/env for verifying released assets

Additional Tasks before day of release:

• Raise PR against registry for logo/display-name

Description

Augment the code generation to include placeholder Create, Read, and Delete functions for all resources. In particular, the Create function should call CreateResource operation with a progress waiter and d.SetId() with the resource type and identifier. The Read function should perform GetResource operation with error checking and trigger resource recreation when receiving the appropriate resource not found error. The Delete function can perform a DeleteResource operation with a progress waiter.

Description

The CloudAPI supports operation cancellation:

You can use the CancelResourceRequest API to request the cancellation of an IN_PROGRESS request, passing the RequestToken from the CreateResource, UpdateResource, or DeleteResource APIs.

Consider supporting this functionality when context.Context signals a user cancellation (^C).

The resource schema code generation in internal/provider/generators/resource/main.go need to be reworked to capture attribute paths for write-only properties (#42) and to correctly determine computed and force-new attributes.
Go back to a version of the code sketched out in #7 (comment) to generate the Schema as a single structure.

Description

To ensure consistency and improve error messaging, standardized warning and error diagnostics can be created.

Proposed Implementation

func ResourceNotFoundAfterCreationDiag(err error) diag.Diagnostic {
	return diag.Diagnostic{
		Severity: diag.Error,
		Summary:  "AWS Resource Not Found After Creation",
		Detail:   fmt.Sprintf("After creating the AWS resource and attempting to read the resource, the API returned a resource not found error. This is typically an error with the Terraform resource implementation. Original Error: %s", err.Error()),
	}
}

func ResourceNotFoundWarningDiag(err error) diag.Diagnostic {
	return diag.Diagnostic{
		Severity: diag.Warning,
		Summary:  "AWS Resource Not Found During Refresh",
		Detail:   fmt.Sprintf("Automatically removing from Terraform State instead of returning the error, which may trigger resource recreation. Original Error: %s", err.Error()),
	}
}

func ServiceOperationEmptyResultDiag(service string, operation string) diag.Diagnostic {
	return diag.Diagnostic{
		Severity: diag.Error,
		Summary:  "AWS SDK Go Service Operation Empty Response",
		Detail:   fmt.Sprintf("Calling %s service %s operation returned missing contents in the response. This is typically an error with the API implementation.", service, operation),
	}
}

func ServiceOperationErrorDiag(service string, operation string, err error) diag.Diagnostic {
	return diag.Diagnostic{
		Severity: diag.Error,
		Summary:  "AWS SDK Go Service Operation Unsuccessful",
		Detail:   fmt.Sprintf("Calling %s service %s operation returned: %s", service, operation, err.Error()),
	}
}

func ServiceOperationWaiterErrorDiag(service string, operation string, err error) diag.Diagnostic {
	return diag.Diagnostic{
		Severity: diag.Error,
		Summary:  "AWS SDK Go Service Operation Incomplete",
		Detail:   fmt.Sprintf("Waiting for %s service %s operation completion returned: %s", service, operation, err.Error()),
	}
}

Can drop Diag in function naming if moved into its own package.

Definition of Done

• The following known error situations have diagnostic wrappers created:
  • AWS Resource Not Found After Creation (error)
  • AWS Resource Not Found During Refresh (warning)
  • AWS SDK Go Service Operation Empty Response (error)
  • AWS SDK Go Service Operation Incomplete (error)
  • AWS SDK Go Service Operation Unsuccessful (error)
• Any CRUD handlers are converted

The Terraform HCL v2 SDK no longer implements (or enforces) an implicit id attribute for every resource.
As the Cloud API has the idea of a unique resource instance identifier, generate an id attribute for every resource type and map it to the Cloud API unique ID.

Related: #28.

The Cloud API (currently) names this as Identifier. For consistency, generated resources will name this attribute identifier.

Description

Given an input CloudFormation Resource Schema, generate the corresponding Terraform Resource Schema. All PascalCase naming must be converted to snake_case.

Description

To avoid practitioner confusion we don't want the default provider configuration to conflict with the current AWS Provider (aws_ prefix).

Definition of Done

• Specify a new prefix for this provider.

Obvious suggestions:

• cloudapi_
• awscloudapi_

To ensure the viability of this project, the conversion package can print output representing the built schema structures. This output will closely represent the output necessary for code generation.

Description

To generate a provider with all CloudAPI-compatible CloudFormation Resource Types, the schemas of each resource will need to be consistently readable from remote source(s) (e.g. CloudFormation API and/or GitHub repositories) and likely saved within the codebase. Once those schemas are available locally, then the generator can do its part.

There are currently a few unknowns with this setup as:

• Public CloudFormation Resource Types are not versioned
• It is unclear how the CloudFormation API itself is updated
• CloudFormation API does provide source GitHub repository for each CloudFormation Resource Type, which could be used to potentially simplify configuration
• Source filenames within each of those repositories seems to follow a convention, e.g. “aws-service-thing/aws-service-thing.json”, further investigation would be good

Dependent on: #17

Proposed Functional Requirements

• Configuration Specification (proposed dependency)
  • Terraform Resource Type
  • Source
    • CloudFormation Registry
    • Git
  • Version
• Git support (proposed dependency)
• JSON Schema handling (proposed dependency)
  • Parse
  • Validate

Proposed Configuration

sources = [
  {
    resource_type = "aws_logs_log_group"
    git_source    = "github.com/aws-cloudformation/aws-cloudformation-resource-providers-logs//aws-logs-loggroup/aws-logs-loggroup.json"
    version       = "a0363cb2ff6fd29e2e2312a116470baf4774e219"
  },
]

Definition of Done

• Given a configuration file, loop through all sources to download all CloudFormation Resource Schemas consistently into a local directory in the repository matching the provided resource_type.
• CI should be setup to verify out of sync configuration and codebase cache on pull requests and main branch.

Currently this provider is using a custom version of the AWS Go v1 SDK.
This major version of the SDK is on a path towards EOL.
AWS will soon provide us with a custom version of the AWS Go v2 SDK.
Once available, migrate the Cloud API calls.
CloudFormation API client bootstrap and provider configuration will also need to be migrated.
Where possible, share implementation patterns with the similar proposed work in the AWS Provider.
If feasible, use an AWS Go v2 SDK-enabled version of the aws-sdk-go-base package.

Definition Of Done

• All dependencies on the AWS Go v1 SDK removed
• No test regressions
• Provider Configuration and Documentation

• hashicorp/aws-sdk-go-base#62

Definition

This codebase is based off terraform-provider-scaffolding which gets us started, but we will need to determine where provider-specific files such as CloudFormation Resource Schemas, generated Terraform Resource files, and various internal packages/tooling should reside.

Proposed Layout

internal/provider/
|- provider(_test).go
 - resource_{RESOURCE_TYPE}_gen.go # Generated Terraform Resource files
internal/service/cloudformation/
|- cfjsonpatch/  # TEMPORARY: RFC 6902 customizations for CloudFormation
|- cfschema/     # CloudFormation Resource Schema Go SDK
|- schemas/      # CloudFormation Resource Schemas cached from update tooling
|- schemas-sync/ # CloudFormation Resource Schema updater
 - waiter/       # GetRequestStatus polling

Definition of Done

• README.md or CONTRIBUTING.md updated with details
• The following issues should have explicit details added:
  • #15
  • #16

All future tasks are dependent on being able to ingest the JSON CloudFormation Resource Provider Schema files. This includes but is not limited to validating the JSON schema against the CloudFormation Resource Provider meta schema, unmarshalling the JSON into Go.

We are currently using the strcase Go package to convert CloudFormation property names (e.g. Arn or GlobalReplicationGroupDescription) to their Terraform attribute name equivalents (arn and global_replication_group_description) and vice-versa.
However I have noticed that S3Bucket is converted to s_3_bucket, which is not what we want.

Currently this provider exposes just the role_arn and region configuration options (and doesn't use role_arn), relying on environment variables to specify authentication arguments via the aws-sdk-go-base package.
Determine and implement the minimal usable set of configuration options, a subset of the AWS Provider's configuration options.
Wherever possible use the same argument names as for the AWS Provider.

Depends on #34.

Definition Of Done

Minimal usable set of provider configuration options

• Documented
• Implemented
• Tested

Relates #39.
Relates #54.
Until the Plugin Framework has a working SetNestedAttributes implementation, implement a local, temporary SetNestedAttributes type.
This should unblock unit tests for sets of objects.

Relates #39.
Until the Plugin Framework has a Set implementation, implement a local, temporary Set type.
This should unblock code generation for all AWS supported resources.

I just noticed that properties listed as required in the CloudFormation schema aren't being marked as such in the generated Terraform schema code.
Fix that.

v0.2.0 of terraform-plugin-framework is available.
Use it.

Description

Relates hashicorp/terraform-plugin-framework#17.

A CloudFormation resource type schema supports various constraints on properties. In Terraform these correspond to attribute validation.

As of v0.1.0 terraform-plugin-framework does not support attribute validation.
Once upstream support is added, enhance this provider.

Definition of Done

Generate the resource schemas into per-service packages instead of into a single large package.
Try and follow work in the "main" provider on how resource registration and bootstrapping would work.

Related: #17

https://github.com/hashicorp/terraform-plugin-framework is the next-generation Terraform Plugin framework.
This provider will target this new framework.
Start the work to target v0.1.0 of terraform-plugin-framework.

• The provider must implement the Provider interface
• The generic resource code must implement the ResourceType and Resource interfaces
• Resource code generation must target the Schema and Attribute types

CloudAPI requires a customized RFC 6902 JSON Patch input for updates. Terraform Plugin Go implements state updates in the TBD type which can be queried via TBD. This information must be converted to a JSON prior state, converted into a JSON desired state, and those two JSON documents compared into JSON Patch format.

Description

Add a minimal set of build tooling to allow contributors to build, test, lint and format code.
Where possible share workflows with the AWS Provider.

Definition of Done

• Go Version - M
• Linting
  • Golangcilint
  • Semgrep
  • impilint
  • Markdownlint
  • link-checker
  • misspell
• CI (TeamCity Setup) - Requires new testing account for CloudAPI (Make request via Doormat)

CloudFormation Resource Provider Schemas encode all information about creation-only (ForceNew), write-only (ForceNew), read-only (wholly Computed), and deprecated properties at the root level using a /path/to/property syntax. The conversion package needs to implement lookup functionality to support those resource behaviors.

CloudAPI requires a stringified JSON desired state as input during creation. Terraform Plugin Go represents prior state (configuration) data in a TBD type which is fetched via TBD calls. Implement this conversion (including snake_case to PascalCase) and generate for all resources, which effectively enables very basic resource support.

A CloudFormation resource schema may (in the future) have a version attribute. The CloudAPI may support this field to lock a resource definition to a major version of a resource schema.

Relates: #15.

Description

Given that the current version of terraform-plugin-docs relies on Terraform CLI output (terraform providers schema -json), the tools should work out of the box with the generated provider. This task captures validating that this process will work, and whether are not there are any shortcoming or modifications required to the tool or to the generated provider.

The scope that the tool needs to satisfy in this case is minimal:

• Support resource level documentation
• Support attribute level documentation

Definition of Done

• Validate that the tool works to satisfy the above scope, and if not, document what needs to be done to satisfy it.
• This item does not cover any remediation work.

Relates: #9.

Description

Relates hashicorp/terraform-plugin-framework#34

As of v0.1.0 terraform-plugin-framework does not support ForceNew attributes.
Once upstream support is added, enhance this provider.

Definition of Done

Description

Documentation generation via terraform-plugin-docs can begin when Terraform Resources (and at least their schema) are being created via the code generation. This will allow stakeholders to investigate the current coverage and verbiage for appropriateness in Terraform. The CloudFormation schema providers resource level and property level documentation, but does not provide detailed service level documentation. There is also clear inconsistencies between the level of documentation in each service.

It is expected that the generated documentation will be much more minimal than practitioners are used to seeing in the existing provider.

Definition of Done

• Setup terraform-plugin-docs to run with other generation command(s).
• Ensure CI verifies terraform-plugin-docs has been synchronized on pull requests and the main branch.
• Ensure that the generation steps allow for non-generated documentation files to exist alongside.
• Import documentation

Until #34 is implemented, consider migrating to the latest CloudAPI-enabled Go v1 SDK: https://github.com/hashicorp/aws-sdk-go-private/tree/f-cloudapi-20210715.
Allegedly the new SDK supports standard JSON Patch operations via a new field, PatchDocument - start to use this new field.

Meet with the Kubernetes Alpha provider team, since that provider must deal with dynamic schemas and validation. They likely have valuable input on design considerations and possible functionality that will influence other tasks.

CloudAPI returns resource information in a stringified JSON. Terraform Plugin Go represents state data in a TBD type which is updated via TBD calls currently. Implement this conversion (including PascalCase to snake_case) and generate for all resources, which effectively enables drift detection for configured attributes.

Description

Relates hashicorp/terraform-plugin-framework#33.
As of v0.1.0 terraform-plugin-framework does not support importing existing resources.
Once upstream support is added, enhance this provider.

Definition of Done