Giter Club home page Giter Club logo

tctest's People

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar

Forkers

paultyng mbfrahry magodo tombuildsstuff angie44 breathingdust gdavison koikonom stephybun teowa

tctest's Issues

command/pr: PRs for Test Files Extraneously Append _test

Description

Example: hashicorp/terraform-provider-aws#11616

$ tctest pr 11616
Discovering tests for pr #11616 (https://github.com/terraform-providers/terraform-provider-aws/pull/11616)...
WARN[2020-01-15 21:28:02] unable to fetch tests from file (aws/resource_aws_service_discovery_private_dns_namespace_test_test.go): unable to get content from https://raw.githubusercontent.com/terraform-providers/terraform-provider-aws/a0f59863c0846d87b69e69a11403d1e7fbca144d/aws/resource_aws_service_discovery_private_dns_namespace_test_test.go: HTTP status NOT OK: 404
    TestAccAWSServiceDiscoveryPrivateDnsNamespace
triggering refs/pull/11616/merge for (TestAccAWSServiceDiscoveryPrivateDnsNamespace)...
  bflad@--OMITTED--#Aws_ProviderAwsAlternate
  build 92399 queued! (https://--OMITTED--/viewQueued.html?itemId=92399)

Note the file name aws/resource_aws_service_discovery_private_dns_namespace_test_test.go contains _test_test.go

command/pr, command/results: allow specifying timeout

Currently tctest pr <pr number> -w and tctest results <build number> -w wait for one hour. It would be useful to allow waiting for longer if we know that the tests we are running will take a long time.

Potentially, the -w flag could take an optional duration value.

command/pr: unknown flag: --wait

Version

Doesn't look like version is working as expected:

$ tctest version
tctest v0.1.0-

Using the latest source via:

$ go get github.com/katbyte/tctest/cmd/tctest

Description

The --wait flag for the pr command no longer appears to be working as it did previously:

$ tctest pr --wait 13292 'TestAccAWSCloudFormationStack(_dataSource)?_yaml'
Error: unknown flag: --wait
Usage:
  tctest pr # [test_regex] [flags]
  tctest pr [command]

Available Commands:
  list        attempts to discover what acceptance tests to run for a PR

Flags:
  -h, --help   help for pr

Global Flags:
  -b, --buildtypeid string   the TeamCity BuildTypeId to trigger
      --fileregex string     the regex to filter files by` (default "(^[a-z]*/resource_|^[a-z]*/data_source_)")
  -p, --pass string          the TeamCity password to use (consider exporting pass to TCTEST_PASS instead)
  -r, --repo string          repository the pr resides in, such as terraform-providers/terraform-provider-azurerm
  -s, --server string        the TeamCity server's ur;
      --splittests string    split tests here and use the value on the left (default "_")
  -u, --user string          the TeamCity user to use

Use "tctest pr [command] --help" for more information about a command.

ERRO[2020-07-20 08:54:13] tctest: unknown flag: --wait

$ tctest pr 13292 'TestAccAWSCloudFormationStack(_dataSource)?_yaml' --wait
Error: unknown flag: --wait
Usage:
  tctest pr # [test_regex] [flags]
  tctest pr [command]

Available Commands:
  list        attempts to discover what acceptance tests to run for a PR

Flags:
  -h, --help   help for pr

Global Flags:
  -b, --buildtypeid string   the TeamCity BuildTypeId to trigger
      --fileregex string     the regex to filter files by` (default "(^[a-z]*/resource_|^[a-z]*/data_source_)")
  -p, --pass string          the TeamCity password to use (consider exporting pass to TCTEST_PASS instead)
  -r, --repo string          repository the pr resides in, such as terraform-providers/terraform-provider-azurerm
  -s, --server string        the TeamCity server's ur;
      --splittests string    split tests here and use the value on the left (default "_")
  -u, --user string          the TeamCity user to use

Use "tctest pr [command] --help" for more information about a command.

ERRO[2020-07-20 08:55:57] tctest: unknown flag: --wait

Timeouts/Retries for Flakey Connections

Description

When running on a poor Internet connection (airplane WiFi in my case), trying to run commands can return timeout errors, e.g.

$ tctest pr 10951
Discovering tests for pr #10951 (https://github.com/terraform-providers/terraform-provider-aws/pull/10951)...
Usage:
... omitted for clarity ...
Use "tctest pr [command] --help" for more information about a command.

ERRO[2019-12-02 13:15:16] tctest: pr cmd failed: pr list failed: Error fetching tests: unable to get content from https://raw.githubusercontent.com/terraform-providers/terraform-provider-aws/87f319b2b0cadd4e57c16ebc34183e2f2f9689f4/aws/resource_aws_autoscaling_group_test.go: HTTP error: Get https://raw.githubusercontent.com/terraform-providers/terraform-provider-aws/87f319b2b0cadd4e57c16ebc34183e2f2f9689f4/aws/resource_aws_autoscaling_group_test.go: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

$ tctest pr 10951
Discovering tests for pr #10951 (https://github.com/terraform-providers/terraform-provider-aws/pull/10951)...
pr file scanner error occured: net/http: request canceled (Client.Timeout exceeded while reading body)    TestAccAWSAutoScalingGroup
triggering refs/pull/10951/merge for (TestAccAWSAutoScalingGroup)...
  bflad@--OMITTED--#Aws_ProviderAwsAlternate
  build 85557 queued! (https://--OMITTED--/viewQueued.html?itemId=85557)

It seems like the timeout value is currently hardcoded to 10 seconds in the common package:

tctest/common/http.go

Lines 12 to 14 in 1c20a2d

var Http = &http.Client{
Timeout: time.Second * 10,
}

Proposal

Could we possibly do one of the following (in increasing complexity):

  • Increase the default timeout to 30 or 60 seconds
  • Allow the timeout to be configurable
  • Introduce retry on timeout logic

Thanks!

command/status: Error parsing test status response body: bufio.Scanner: token too long

Description

Some Terraform Provider acceptance testing will generate log lines longer than the default bufio.Scanner limit (64k). I believe those long log lines are related to the AWS Go SDK debug output of binary file uploads, but the cause doesn't matter too much for the purposes of this command as these long log lines should be truncated or skipped.

Example command:

$ tctest status 91578
Error: Error parsing test status response body: bufio.Scanner: token too long
Usage:
  tctest status # [flags]

Flags:
  -h, --help   help for status

Global Flags:
  -b, --buildtypeid string   the TeamCity BuildTypeId to trigger
      --fileregex string     the regex to filter files by` (default "(^[a-z]*/resource_|^[a-z]*/data_source_)")
  -p, --pass string          the TeamCity password to use (consider exporting pass to TCTEST_PASS instead)
  -r, --repo string          repository the pr resides in, such as terraform-providers/terraform-provider-azurerm
  -s, --server string        the TeamCity server's ur;
      --splittests string    split tests here and use the value on the left (default "_")
  -u, --user string          the TeamCity user to use

ERRO[2020-01-10 19:16:37] tctest: Error parsing test status response body: bufio.Scanner: token too long

command/pr, command/list: List individual affected tests

Proposal

Add an option to automatically discover individual tests (rather than all tests in affected files) affected by a PR. (NOTE: This would only be practical for changes to _test.go files.)

Description

For broad PRs and PRs targetting files with many tests, the resulting number of tests to be run grows exponentially. It would be super-handy if an option allowed looking deeper at the actual changes to find specific individual tests affected rather than test families.

Current behavior (82 tests to be run)

% tctest list 194782
Discovering tests for pr #194782 (https://github.com/terraform-providers/terraform-provider-aws/pull/194782)...
    TestAccAWSInstance

Running make testacc TESTARGS='-run=TestAccAWSInstance' results in 82 tests run.

Proposed behavior (1 test to be run)

Perhaps use -i for individual test discovery.

% tctest list -i 194782
Discovering tests for pr #194782 (https://github.com/terraform-providers/terraform-provider-aws/pull/194782)...
    TestAccAWSInstance_RootBlockDevice_KmsKeyArn

Running make testacc TESTARGS='-run=TestAccAWSInstance_RootBlockDevice_KmsKeyArn' results in 1 test run.

Design thoughts

Basic process:

  1. Go through each changed line in the PR
  2. Given a changed line in a _test.go file:
    • Scan back until func declaration
    • If func is TestAcc, add to the list of individual tests
    • If func is anything else (e.g., testAcc...(), find...()), initially, revert to current behavior and add test family to list (this could be analyzed with some sort of magic to identify individual tests but that may not be feasible)
  3. Given a changed line in a file other than _test.go file, revert to current behavior and add corresponding test family(ies) to list

Support for re-running failed tests

Certain tests may be flaky and it would be desirable to just re-run the failed tests instead of everything again. For example, this workflow is required now where the re-run command is copy-pasting elements from the results:

$ tctest pr 12530
Discovering tests for pr #12530 (https://github.com/terraform-providers/terraform-provider-aws/pull/12530)...
    TestAccAWSStorageGatewayNfsFileShare
triggering refs/pull/12530/merge for (TestAccAWSStorageGatewayNfsFileShare)...
  bflad@HOSTNAME#Aws_ProviderAwsAlternate
  build 103995 queued: https://HOSTNAME/viewQueued.html?itemId=103995

$ tctest results 103995
--- PASS: TestAccAWSStorageGatewayNfsFileShare_basic (198.28s)
--- PASS: TestAccAWSStorageGatewayNfsFileShare_ClientList (265.22s)
--- PASS: TestAccAWSStorageGatewayNfsFileShare_DefaultStorageClass (270.50s)
--- PASS: TestAccAWSStorageGatewayNfsFileShare_GuessMIMETypeEnabled (287.48s)
--- PASS: TestAccAWSStorageGatewayNfsFileShare_KMSKeyArn (303.82s)
--- PASS: TestAccAWSStorageGatewayNfsFileShare_tags (315.32s)
--- PASS: TestAccAWSStorageGatewayNfsFileShare_Squash (317.48s)
--- FAIL: TestAccAWSStorageGatewayNfsFileShare_KMSEncrypted (716.07s)
--- FAIL: TestAccAWSStorageGatewayNfsFileShare_NFSFileShareDefaults (768.03s)
--- FAIL: TestAccAWSStorageGatewayNfsFileShare_ObjectACL (775.86s)
--- FAIL: TestAccAWSStorageGatewayNfsFileShare_ReadOnly (775.84s)
--- FAIL: TestAccAWSStorageGatewayNfsFileShare_RequesterPays (791.91s)

$ tctest pr 12530 '(TestAccAWSStorageGatewayNfsFileShare_KMSEncrypted|TestAccAWSStorageGatewayNfsFileShare_NFSFileShareDefaults|TestAccAWSStorageGatewayNfsFileShare_ObjectACL|TestAccAWSStorageGatewayNfsFileShare_ReadOnly|TestAccAWSStorageGatewayNfsFileShare_RequesterPays)'
triggering refs/pull/12530/merge for (TestAccAWSStorageGatewayNfsFileShare_KMSEncrypted|TestAccAWSStorageGatewayNfsFileShare_NFSFileShareDefaults|TestAccAWSStorageGatewayNfsFileShare_ObjectACL|TestAccAWSStorageGatewayNfsFileShare_ReadOnly|TestAccAWSStorageGatewayNfsFileShare_RequesterPays)...
  bflad@HOSTNAME#Aws_ProviderAwsAlternate
  build 103998 queued: https://HOSTNAME/viewQueued.html?itemId=103998

It'd be really nice if we could so something like the following instead:

$ tctest run-failed-results 103995
# or
$ tctest pr --failed-only 12530
# or
$ tctest pr --run-failed-results=103995 12530

command/pr: PRs for Deleted Files Extraneously Inspected for Tests

Description

Example: https://github.com/terraform-providers/terraform-provider-aws/pull/11565/files

$ tctest pr 11565
Discovering tests for pr #11565 (https://github.com/terraform-providers/terraform-provider-aws/pull/11565)...
WARN[2020-01-16 01:01:37] unable to fetch tests from file (aws/tagsGeneric_test.go): unable to get content from https://raw.githubusercontent.com/terraform-providers/terraform-provider-aws/cfd95109f1804e21bef3156d324fdb36f8cbccea/aws/tagsGeneric_test.go: HTTP status NOT OK: 404
    TestAccAwsBackupPlan
triggering refs/pull/11565/merge for (TestAccAwsBackupPlan)...
  bflad@--OMITTED--#Aws_ProviderAwsAlternate
  build 92522 queued! (https://--OMITTED--/viewQueued.html?itemId=92522)

command/pr: Auto-detection runs all tests unexpectedly

If a PR deals with a test file that only has a serialized test, tctest runs the entire suite.

For example, if the PR dealt with aws/resource_aws_appmesh_test.go, it runs the entire suite. Abbreviated contents of aws/resource_aws_appmesh_test.go:

func TestAccAWSAppmesh_serial(t *testing.T) {
	testCases := map[string]map[string]func(t *testing.T){
		"Mesh": {
			"basic":        testAccAwsAppmeshMesh_basic,
			"egressFilter": testAccAwsAppmeshMesh_egressFilter,
			"tags":         testAccAwsAppmeshMesh_tags,
		},
		// more serialized tests
	}

	for group, m := range testCases {
		m := m
		t.Run(group, func(t *testing.T) {
			for name, tc := range m {
				tc := tc
				t.Run(name, func(t *testing.T) {
					tc(t)
				})
			}
		})
	}
}

command/results pr: returning error

Steps to Reproduce:

  • ensure test is running/ran in TC
  • run tctest results pr 14006

Expected Output:

Test Results (buildID: 11111, buildNumber: 123asda, branch: pull/14006/merge):
--- PASS: TestAccAWS...

Actual Output:

ERRO[2020-07-01 01:03:39] tctest: no builds parsed from XML response

Support for Returning Latest Results Based on PR Number

Description

Currently the tctest results command requires knowing the build number to fetch the build log. When manually kicking off a few tests at the same time then coming back to them later on, you have to scroll back (or worse if you don't have the scroll back, look up the build number in the UI) to get the build number based off the PR number.

For example:

Let's run some tests and grab some ☕ !

$ tctest pr 11960
Discovering tests for pr #11960 (https://github.com/terraform-providers/terraform-provider-aws/pull/11960)...
    TestAccAWSEFSFileSystem
    TestAccDataSourceAwsEfsFileSystem
    TestResourceAWSEFSFileSystem
triggering refs/pull/11960/merge for (TestAccAWSEFSFileSystem|TestAccDataSourceAwsEfsFileSystem|TestResourceAWSEFSFileSystem)...
  bflad@--OMITTED--#Aws_ProviderAwsAlternate
  build 112393 queued: https://--OMITTED--/viewQueued.html?itemId=112393
$ tctest pr 11809
Discovering tests for pr #11809 (https://github.com/terraform-providers/terraform-provider-aws/pull/11809)...
    TestIpPermissionIDHash
    TestAccAWSSecurityGroupRule
triggering refs/pull/11809/merge for (TestIpPermissionIDHash|TestAccAWSSecurityGroupRule)...
  bflad@--OMITTED--#Aws_ProviderAwsAlternate
  build 112467 queued: https://--OMITTED--/viewQueued.html?itemId=112467
$ tctest pr 12865
Discovering tests for pr #12865 (https://github.com/terraform-providers/terraform-provider-aws/pull/12865)...
    TestAccAWSIAMInstanceProfile
    TestAccAWSIAMPolicy
    TestAccAWSIAMRole
    TestAccAWSUser
    TestAccAWSIAMGroup
triggering refs/pull/12865/merge for (TestAccAWSIAMInstanceProfile|TestAccAWSIAMPolicy|TestAccAWSIAMRole|TestAccAWSUser|TestAccAWSIAMGroup)...
  bflad@--OMITTED--#Aws_ProviderAwsAlternate
  build 112587 queued: https://--OMITTED--/viewQueued.html?itemId=112587
$ tctest pr 13263
Discovering tests for pr #13263 (https://github.com/terraform-providers/terraform-provider-aws/pull/13263)...
    TestAccAwsRamResourceShareAccepter
triggering refs/pull/13263/merge for (TestAccAwsRamResourceShareAccepter)...
  bflad@--OMITTED--#Aws_ProviderAwsAlternate
  build 112836 queued: https://--OMITTED--/viewQueued.html?itemId=112836
$ tctest pr 13268
Discovering tests for pr #13268 (https://github.com/terraform-providers/terraform-provider-aws/pull/13268)...
    TestAccAWSKinesisFirehoseDeliveryStream
triggering refs/pull/13268/merge for (TestAccAWSKinesisFirehoseDeliveryStream)...
  bflad@--OMITTED--#Aws_ProviderAwsAlternate
  build 112837 queued: https://--OMITTED--/viewQueued.html?itemId=112837
$ tctest pr 13270
Discovering tests for pr #13270 (https://github.com/terraform-providers/terraform-provider-aws/pull/13270)...
triggering refs/pull/13270/merge for ()...
  bflad@--OMITTED--#Aws_ProviderAwsAlternate
  build 112838 queued: https://--OMITTED--/viewQueued.html?itemId=112838
$ tctest pr 13271
Discovering tests for pr #13271 (https://github.com/terraform-providers/terraform-provider-aws/pull/13271)...
    TestAccAWSIotRoleAlias
triggering refs/pull/13271/merge for (TestAccAWSIotRoleAlias)...
  bflad@--OMITTED--#Aws_ProviderAwsAlternate
  build 112839 queued: https://--OMITTED--/viewQueued.html?itemId=112839
$ tctest pr 13272
Discovering tests for pr #13272 (https://github.com/terraform-providers/terraform-provider-aws/pull/13272)...
    TestAccAWSGlueJob
triggering refs/pull/13272/merge for (TestAccAWSGlueJob)...
  bflad@--OMITTED--#Aws_ProviderAwsAlternate
  build 112840 queued: https://--OMITTED--/viewQueued.html?itemId=112840

That ☕ was delicious! Now let's get test results... hmm let me try to go get all those build numbers... 😅 oh this is tedious!

$ tctest results 112393
...
$ tctest results 
...
$ tctest results 
...
$ tctest results 
...
$ tctest results 
...
$ tctest results 
...
$ tctest results 
...
$ tctest results 
...
$ tctest results

It would be great if there was something along these lines to help (in preference order):

  • A command like tctest pr-results --latest ####/tctest latest-pr-results #### to automatically lookup the latest build number for the PR and return the results
  • A command like tctest pr-build --latest ####/tctest latest-pr-build #### to return the latest build number for the PR that could then be fed into tctest results ####, e.g. tctest results (tctest latest-pr-build ####)

It is probably worth mentioning that this ability would also move us closer to being able to call this tool asynchronously for chatops, etc. (since things like GitHub Actions and Lambda Functions have execution timeouts)

Available Workarounds

  1. Run each test in its own terminal with the tctest pr --wait flag.
  2. Run these tests in a for loop with the tctest pr --wait flag.

command/pr: Support waiting for build to finish

Description

Relates to #13.
Will require #6 implementation first.

It would be super awesome if you could specify a --wait flag or similar to the pr command which would wait for the build to queue, start, and finish, and then display the results.

command/pr: No results or errors returned when cancelled due to "cannot find commit"

Version

v0.2.0 (patched with #28)

Description

While kicking off a few tests via tctest pr --wait and merging other things into VCS, ran into a situation where TeamCity cancelled a run with a temporary VCS error. This was however not surfaced back to the client running the testing in any form.

Opening the build:

Screen Shot 2020-05-21 at 8 32 56 AM

Expected Behavior

$ tctest pr --wait 13430
Discovering tests for pr #13430 (https://github.com/terraform-providers/terraform-provider-aws/pull/13430)...
    TestAccAWSLightsailInstance
triggering refs/pull/13430/merge for (TestAccAWSLightsailInstance)...
  bflad@--OMITTED--#Aws_ProviderAwsAlternate
  build 114694 queued: https://--OMITTED--/viewQueued.html?itemId=114694
Waiting for build 114694 status to be 'finished'...
ERROR: Canceled (Error while applying patch; cannot find commit cbc11326b6340439ac52b1baa58f38e1d0ef62dd in the https://github.com/terraform-providers/terraform-provider-aws repository, possible reason: refs/pull/13430/merge branch was updated and the commi...
$

Actual Behavior

$ tctest pr --wait 13430
Discovering tests for pr #13430 (https://github.com/terraform-providers/terraform-provider-aws/pull/13430)...
    TestAccAWSLightsailInstance
triggering refs/pull/13430/merge for (TestAccAWSLightsailInstance)...
  bflad@--OMITTED--#Aws_ProviderAwsAlternate
  build 114694 queued: https://--OMITTED--/viewQueued.html?itemId=114694
Waiting for build 114694 status to be 'finished'...
$

(No errors or results)

command/pr: Auto-Detection Submits Empty Test Pattern When No Tests Found

Version

$ tctest version
tctest v0.2.0-

Description

When the pr command cannot find associated tests, by default it will send a test pattern of () (meaning all tests). For large projects that create real infrastructure, this is very dangerous. This is likely not what the operator intended, or if it is, then the command should likely require an override such as an explicit test pattern to be added.

$ tctest pr 13270
Discovering tests for pr #13270 (https://github.com/terraform-providers/terraform-provider-aws/pull/13270)...
triggering refs/pull/13270/merge for ()...
  bflad@--OMITTED--#Aws_ProviderAwsOrganizations
  build 112841 queued: https://--OMITTED--/viewQueued.html?itemId=112841

command/pr: Support for only running specifically named test patterns (still with auto-detection)

Description

Currently changes to shared files with large amounts of unit testing introduces a large amount of testing noise when running via tctest:

tctest pr 13143
Discovering tests for pr #13143 (https://github.com/terraform-providers/terraform-provider-aws/pull/13143)...
    TestValidateOnceADayWindowFormat
    TestValidateDmsReplicationTaskId
    TestValidateDbParamGroupName
    TestValidateKmsKey
    TestValidateDxConnectionBandWidth
    TestValidateArn
    TestValidateAwsKmsName
    TestValidateCognitoUserPoolEmailVerificationMessage
    TestValidateCognitoUserPoolSmsAuthenticationMessage
    TestValidateIamRoleDescription
    TestValidateTypeStringNullableFloat
    TestValidateLogGroupNamePrefix
    TestValidateDmsReplicationSubnetGroupId
    TestValidateSSMDocumentPermissions
    TestValidateSecurityGroupRuleDescription
    TestValidateLaunchTemplateName
    TestValidateCIDRNetworkAddress
    TestValidateCognitoProviderDeveloperName
    TestResourceAWSElastiCacheReplicationGroupAuthTokenValidation
    TestValidateLambdaPermissionEventSourceToken
    TestValidateEC2AutomateARN
    TestValidateSagemakerName
    TestValidateApiGatewayUsagePlanQuotaSettings
    TestValidateNeptuneEventSubscriptionNamePrefix
    TestValidateDbSubnetGroupName
    TestValidateNeptuneParamGroupNamePrefix
    TestValidateCloudFrontPublicKeyName
    TestValidateAwsKmsGrantName
    TestValidateWafMetricName
    TestValidateCognitoUserGroupName
    TestValidateSecretManagerSecretNamePrefix
    TestValidateLogMetricTransformationName
    TestValidateS3BucketLifecycleTimestamp
    TestValidateSQSFifoQueueName
    TestValidateStepFunctionStateMachineName
    TestValidateAccountAlias
    TestValidateIamRoleProfileNamePrefix
    TestValidateAwsSSMName
    TestValidateCognitoRoleMappingsRulesConfiguration
    TestValidateNeptuneParamGroupName
    TestAccAWSSSMDocument
    TestValidateCloudWatchEventRuleName
    TestValidateAwsAccountId
    TestValidatePolicyStatementId
    TestValidateLogGroupName
    TestValidateIAMPolicyJsonString
    TestValidateEcsPlacementStrategy
    TestValidateDmsCertificateId
    TestValidateNeptuneEventSubscriptionName
    TestValidateDbOptionGroupNamePrefix
    TestValidateCognitoUserPoolEmailVerificationSubject
    TestValidateRoute53ResolverName
    TestValidateLambdaFunctionName
    TestValidateDocDBIdentifier
    TestValidateDbOptionGroupName
    TestValidateOpenIdURL
    TestValidateCognitoSupportedLoginProviders
    TestValidateCognitoIdentityProvidersProviderName
    TestValidateBatchName
    TestValidateCloudFrontPublicKeyNamePrefix
    TestValidateSQSQueueName
    TestValidateDmsEndpointId
    TestValidateCognitoRoleMappingsAmbiguousRoleResolutionAgainstType
    TestValidateCognitoUserPoolId
    TestValidateTypeStringNullableBoolean
    TestValidateLbTargetGroupName
    TestValidateLbTargetGroupNamePrefix
    TestValidateSecretManagerSecretName
    TestAccAWSCognitoIdentityPoolRolesAttachment
    TestValidateCloudWatchDashboardName
    TestValidateLambdaPermissionAction
    TestValidateOnceAWeekWindowFormat
    TestValidateEcsPlacementConstraint
    TestValidateEmrCustomAmiId
    TestValidateIamRoleProfileName
    TestValidateNeptuneSubnetGroupName
    TestValidateDbSubnetGroupNamePrefix
    TestValidateCognitoRoles
    TestValidateElbNamePrefix
    TestValidateNeptuneSubnetGroupNamePrefix
    TestValidateCognitoIdentityPoolName
    TestValidateBatchPrefix
    TestValidateLambdaQualifier
    TestValidateDbEventSubscriptionName
    TestValidateCognitoUserPoolSmsVerificationMessage
    TestValidateLogMetricFilterName
    TestValidateCloudFormationTemplate
    TestValidateDmsReplicationInstanceId
    TestValidateElbName
    TestValidateCognitoIdentityProvidersClientId
    TestValidateAmazonSideAsn
    TestValidateLaunchTemplateId
triggering refs/pull/13143/merge for (TestValidateOnceADayWindowFormat|TestValidateDmsReplicationTaskId|TestValidateDbParamGroupName|TestValidateKmsKey|TestValidateDxConnectionBandWidth|TestValidateArn|TestValidateAwsKmsName|TestValidateCognitoUserPoolEmailVerificationMessage|TestValidateCognitoUserPoolSmsAuthenticationMessage|TestValidateIamRoleDescription|TestValidateTypeStringNullableFloat|TestValidateLogGroupNamePrefix|TestValidateDmsReplicationSubnetGroupId|TestValidateSSMDocumentPermissions|TestValidateSecurityGroupRuleDescription|TestValidateLaunchTemplateName|TestValidateCIDRNetworkAddress|TestValidateCognitoProviderDeveloperName|TestResourceAWSElastiCacheReplicationGroupAuthTokenValidation|TestValidateLambdaPermissionEventSourceToken|TestValidateEC2AutomateARN|TestValidateSagemakerName|TestValidateApiGatewayUsagePlanQuotaSettings|TestValidateNeptuneEventSubscriptionNamePrefix|TestValidateDbSubnetGroupName|TestValidateNeptuneParamGroupNamePrefix|TestValidateCloudFrontPublicKeyName|TestValidateAwsKmsGrantName|TestValidateWafMetricName|TestValidateCognitoUserGroupName|TestValidateSecretManagerSecretNamePrefix|TestValidateLogMetricTransformationName|TestValidateS3BucketLifecycleTimestamp|TestValidateSQSFifoQueueName|TestValidateStepFunctionStateMachineName|TestValidateAccountAlias|TestValidateIamRoleProfileNamePrefix|TestValidateAwsSSMName|TestValidateCognitoRoleMappingsRulesConfiguration|TestValidateNeptuneParamGroupName|TestAccAWSSSMDocument|TestValidateCloudWatchEventRuleName|TestValidateAwsAccountId|TestValidatePolicyStatementId|TestValidateLogGroupName|TestValidateIAMPolicyJsonString|TestValidateEcsPlacementStrategy|TestValidateDmsCertificateId|TestValidateNeptuneEventSubscriptionName|TestValidateDbOptionGroupNamePrefix|TestValidateCognitoUserPoolEmailVerificationSubject|TestValidateRoute53ResolverName|TestValidateLambdaFunctionName|TestValidateDocDBIdentifier|TestValidateDbOptionGroupName|TestValidateOpenIdURL|TestValidateCognitoSupportedLoginProviders|TestValidateCognitoIdentityProvidersProviderName|TestValidateBatchName|TestValidateCloudFrontPublicKeyNamePrefix|TestValidateSQSQueueName|TestValidateDmsEndpointId|TestValidateCognitoRoleMappingsAmbiguousRoleResolutionAgainstType|TestValidateCognitoUserPoolId|TestValidateTypeStringNullableBoolean|TestValidateLbTargetGroupName|TestValidateLbTargetGroupNamePrefix|TestValidateSecretManagerSecretName|TestAccAWSCognitoIdentityPoolRolesAttachment|TestValidateCloudWatchDashboardName|TestValidateLambdaPermissionAction|TestValidateOnceAWeekWindowFormat|TestValidateEcsPlacementConstraint|TestValidateEmrCustomAmiId|TestValidateIamRoleProfileName|TestValidateNeptuneSubnetGroupName|TestValidateDbSubnetGroupNamePrefix|TestValidateCognitoRoles|TestValidateElbNamePrefix|TestValidateNeptuneSubnetGroupNamePrefix|TestValidateCognitoIdentityPoolName|TestValidateBatchPrefix|TestValidateLambdaQualifier|TestValidateDbEventSubscriptionName|TestValidateCognitoUserPoolSmsVerificationMessage|TestValidateLogMetricFilterName|TestValidateCloudFormationTemplate|TestValidateDmsReplicationInstanceId|TestValidateElbName|TestValidateCognitoIdentityProvidersClientId|TestValidateAmazonSideAsn|TestValidateLaunchTemplateId)...
  bflad@--OMITTED--#Aws_ProviderAwsAlternate
  build 111059 queued: https://--OMITTED--/viewQueued.html?itemId=111059

It would be great if tctest provided a command option to control this behavior, such as --filter or --include-name-pattern or --include-file-pattern, beyond the hardcoded specifying the explicit test name patterns to match the auto-detected pattern. 👍 e.g.

$ tctest pr 13143 --include-name-pattern TestAcc
Discovering tests for pr #13143 (https://github.com/terraform-providers/terraform-provider-aws/pull/13143)...
    TestAccAWSSSMDocument
    TestAccAWSCognitoIdentityPoolRolesAttachment
triggering refs/pull/13143/merge for (TestAccAWSSSMDocument|TestAccAWSCognitoIdentityPoolRolesAttachment)...
  bflad@--OMITTED--#Aws_ProviderAwsAlternate
  build 111059 queued: https://--OMITTED--/viewQueued.html?itemId=111059

command/pr, command/results: add option to open TeamCity build in browser

Add the option to open the TeamCity build in the user's browser.

Example usage:

$ tctest pr 1234 --open
Discovering tests for pr #1234 (https://github.com/org/project/pull/1234)...
    TestTesting
triggering refs/pull/1234/merge for (TestTesting)...
  build 123456 queued: https://teamcity.example.com/viewQueued.html?itemId=123456
opening in browser...
$ tctest results 123456 --open
opening in browser...

command/pr: Consistently failing to download files during test discovery

Version

Latest codebase (d4f150a)

Description

Consistently seeing errors like the following:

$ tctest pr 14230 --wait
Discovering tests for pr #14230 (https://github.com/terraform-providers/terraform-provider-aws/pull/14230)...
ERRO[2020-07-23 08:38:53] tctest: pr cmd failed: pr list failed: downloading file aws/resource_aws_s3_bucket_metric_test.go
$  tctest pr 14310 --wait
Discovering tests for pr #14310 (https://github.com/terraform-providers/terraform-provider-aws/pull/14310)...
ERRO[2020-07-23 08:56:13] tctest: pr cmd failed: pr list failed: downloading file aws/resource_aws_instance_test.go

Adding GITHUB_TOKEN to my environment has no effect.

Workaround

Manually specifying the test pattern.

$ tctest pr 14230 'TestAccAWSS3BucketMetric_' --wait
triggering refs/pull/14230/merge for TestAccAWSS3BucketMetric_...
  build 135111 queued: https://--OMITTED--/viewQueued.html?itemId=135111
Waiting for build 135111 status to be 'finished'...
--- PASS: TestAccAWSS3BucketMetric_WithEmptyFilter (1.67s)
--- PASS: TestAccAWSS3BucketMetric_WithFilterPrefix (17.71s)
--- PASS: TestAccAWSS3BucketMetric_basic (21.05s)
--- PASS: TestAccAWSS3BucketMetric_WithFilterMultipleTags (26.73s)
--- PASS: TestAccAWSS3BucketMetric_WithFilterPrefixAndSingleTag (27.17s)
--- PASS: TestAccAWSS3BucketMetric_WithFilterPrefixAndMultipleTags (27.96s)
--- PASS: TestAccAWSS3BucketMetric_WithFilterSingleTag (36.17s)

Debug Logs

$ TCTEST_LOG=debug tctest pr 14230 --wait
Discovering tests for pr #14230 (https://github.com/terraform-providers/terraform-provider-aws/pull/14230)...
DEBU[2020-07-23 08:58:41] fetching data for PR terraform-providers/terraform-provider-aws/#14230...
DEBU[2020-07-23 08:58:42]   checking pr state: open
DEBU[2020-07-23 08:58:42]   filtering files ((^[a-z]*/resource_|^[a-z]*/data_source_))
DEBU[2020-07-23 08:58:42]     aws/resource_aws_s3_bucket_metric.go
DEBU[2020-07-23 08:58:42]     aws/resource_aws_s3_bucket_metric_test.go
DEBU[2020-07-23 08:58:42]     website/docs/guides/version-3-upgrade.html.md
DEBU[2020-07-23 08:58:42]     website/docs/r/s3_bucket_metric.html.markdown
DEBU[2020-07-23 08:58:42]   FOUND 1
DEBU[2020-07-23 08:58:42]   parsing content:
DEBU[2020-07-23 08:58:42]     download aws/resource_aws_s3_bucket_metric_test.go
ERRO[2020-07-23 08:58:42] tctest: pr cmd failed: pr list failed: downloading file aws/resource_aws_s3_bucket_metric_test.go

$ TCTEST_LOG=debug tctest pr 14310 --wait
Discovering tests for pr #14310 (https://github.com/terraform-providers/terraform-provider-aws/pull/14310)...
DEBU[2020-07-23 08:58:27] fetching data for PR terraform-providers/terraform-provider-aws/#14310...
DEBU[2020-07-23 08:58:27]   checking pr state: open
DEBU[2020-07-23 08:58:27]   filtering files ((^[a-z]*/resource_|^[a-z]*/data_source_))
DEBU[2020-07-23 08:58:27]     aws/resource_aws_ebs_volume.go
DEBU[2020-07-23 08:58:27]     aws/resource_aws_ebs_volume_test.go
DEBU[2020-07-23 08:58:27]     aws/resource_aws_instance.go
DEBU[2020-07-23 08:58:27]     aws/resource_aws_instance_test.go
DEBU[2020-07-23 08:58:27]     website/docs/guides/version-3-upgrade.html.md
DEBU[2020-07-23 08:58:27]     website/docs/r/ebs_volume.html.markdown
DEBU[2020-07-23 08:58:27]   FOUND 2
DEBU[2020-07-23 08:58:27]   parsing content:
DEBU[2020-07-23 08:58:27]     download aws/resource_aws_ebs_volume_test.go
DEBU[2020-07-23 08:58:28] test: TestAccAWSEBSVolume_basic(t
DEBU[2020-07-23 08:58:28] test: TestAccAWSEBSVolume_updateAttachedEbsVolume(t
DEBU[2020-07-23 08:58:28] test: TestAccAWSEBSVolume_updateSize(t
DEBU[2020-07-23 08:58:28] test: TestAccAWSEBSVolume_updateType(t
DEBU[2020-07-23 08:58:28] test: TestAccAWSEBSVolume_updateIops(t
DEBU[2020-07-23 08:58:28] test: TestAccAWSEBSVolume_kmsKey(t
DEBU[2020-07-23 08:58:28] test: TestAccAWSEBSVolume_NoIops(t
DEBU[2020-07-23 08:58:28] test: TestAccAWSEBSVolume_InvalidIopsForType(t
DEBU[2020-07-23 08:58:28] test: TestAccAWSEBSVolume_withTags(t
DEBU[2020-07-23 08:58:28] test: TestAccAWSEBSVolume_multiAttach(t
DEBU[2020-07-23 08:58:28] test: TestAccAWSEBSVolume_outpost(t
DEBU[2020-07-23 08:58:28] test: TestAccAWSEBSVolume_disappears(t
DEBU[2020-07-23 08:58:28]     download aws/resource_aws_instance_test.go
ERRO[2020-07-23 08:58:29] tctest: pr cmd failed: pr list failed: downloading file aws/resource_aws_instance_test.go

command/pr, command/list: Fails if a file is deleted when building list of tests

When running the tctest pr <pr number> or tctest list <pr numberr> commands on a PR which deletes a file, it fails with the following error

$ tctest pr 16467
Discovering tests for pr #16467 (https://github.com/terraform-providers/terraform-provider-aws/pull/16467)...
ERRO[2020-11-27 11:19:24] tctest: pr cmd failed: pr list failed: downloading file (aws/internal/tfawsresource/testing_test.go) metadata: GET https://api.github.com/repositories/93444615/contents/aws/internal/tfawsresource/testing_test.go?ref=e9457fd5f715fc381621b558e33018cf7cec0119: 404 Not Found []

If the user provides a test regex, it does not fail

tctest pr 16467 TestAccDataSourceAwsRoute53ResolverRules_basic
triggering refs/pull/16467/merge for TestAccDataSourceAwsRoute53ResolverRules_basic...
  build 12345 queued: https://teamcity.example.com/viewQueued.html?itemId= 12345

Support for Additional Test Patterns via Code Comments

Description

Currently, tctest checks for test patterns in the _test.go filename of modified PR files. Most of the time this is great and covers what should be tested, however there are scenarios where it may be desirable or necessary to use other test patterns:

  • Test declarations in other files: In Terraform Providers, services may only allow 1 resource to exist so testing is serialized across many resources and the test declaration is in a separate shared file.
  • Additional testing desired: In Terraform Providers, resources may also have an associated data source, which will be in a separate test file. Otherwise, there may also be other resources or testing that use the changing functionality.

Proposal

Support the ability to declare additional test patterns via code comments. Go convention is //XXX with no space for programatic comments. Including an additional :add: field to the comment for future proofing (e.g. if we wanted to have the ability to programmatically skip tests too).

Proposal 1

//tctest:add:TestAccExample

Personally, I think this this more desirable since it gives full control to run singular tests.

Proposal 2

//tctest:add:path/to/other/file_test.go

Returns a 404 when querying status of queued build

When a build is queued, running tctest status returns a 404 error, rather than reporting a queued status.

Expected behaviour:

$ tctest status 1234
Build queued

Actual behaviour:

$ tctest status 1234
ERRO[2019-11-27 15:34:52] tctest: HTTP status NOT OK: 404

command/pr, command/list: When the latest commit is not mergeable, tctest should return an error

If the latest commit is not mergeable, tctest will identify tests from the latest mergeable commit. It would be helpful if the pr and list commands would return an error stating that the current PR is not mergeable.

A flag to override this behaviour should be added, which would emit a warning and continue running the command.

Steps to reproduce:

  1. On a project with status checks configured, introduce an error, e.g. a linting error
  2. Run tctest pr
  3. Observe TeamCity running old tests

Proposed behaviour:

$ tctest pr 12345
Discovering tests for pr #12345 (...)
ERROR: PR #12345 is not mergeable

$tctest pr 12345 --ignore-mergeability
Discovering tests for pr #12345 (...)
WARNING: PR #12345 is not mergeable, using last mergeable commit ab3410
...

pr cmd failed: pr list failed: Error fetching tests: unable to get content

When testing a pull request that contains new files for the repository from a fork, it appears tctest is unable to handle the test lookup correctly.

$ tctest pr 8916
Discovering tests for pr #8916 (https://github.com/terraform-providers/terraform-provider-aws/pull/8916)...
Usage:
  tctest pr # [test_regex] [flags]
  tctest pr [command]

Available Commands:
  list        attempts to discover what acceptance tests to run for a PR

Flags:
  -h, --help   help for pr

Global Flags:
  -b, --buildtypeid string   the TeamCity BuildTypeId to trigger
      --fileregex string     the regex to filter files by` (default "(^[a-z]*/resource_|^[a-z]*/data_source_)")
  -p, --pass string          the TeamCity password to use (consider exporting pass to TCTEST_PASS instead)
  -r, --repo string          repository the pr resides in, such as terraform-providers/terraform-provider-azurerm
  -s, --server string        the TeamCity server's ur;
      --splittests string    split tests here and use the value on the left (default "_")
  -u, --user string          the TeamCity user to use

Use "tctest pr [command] --help" for more information about a command.

ERRO[2019-06-10 08:01:24] tctest: pr cmd failed: pr list failed: Error fetching tests: unable to get content from https://raw.githubusercontent.com/terraform-providers/terraform-provider-aws/49cbf754b761ca9f707775a82df87a342fb81f8d/aws/resource_aws_lex_test.go: HTTP status NOT OK: 404

command/status: Support waiting for build to finish

Description

May require #6 implementation first.

It would be super awesome if you could specify a --wait flag or similar to the status command which would wait for the build to finish and then display the results. If the build is complete, no waiting necessary. 😄

make command returning error

Notes:

  • master branch missing GithubRepo and NewGithubRepoFromViper implementations in pr.go

Steps to reproduce:

  • run make or make install

Expected output:

==> building...
cd cmd/tctest && go build -ldflags "-X github.com/katbyte/tctest/version.GitCommit=bd81da6-dirty" . && mv tctest ../../

or

==> installing...
cd cmd/tctest && go install -ldflags "-X github.com/katbyte/tctest/version.GitCommit=bd81da6-dirty" .

Actual output:

==> building...
cd cmd/tctest && go build -ldflags "-X github.com/katbyte/tctest/version.GitCommit=160e6b3" . && mv tctest ../../
# github.com/katbyte/tctest/cmd/tctest/cli
cli/cmds.go:135:20: undefined: NewGithubRepoFromViper
cli/cmds.go:177:17: undefined: NewGithubRepoFromViper
cli/pr.go:17:10: undefined: GithubRepo
cli/pr.go:21:10: undefined: GithubRepo
cli/pr.go:35:10: undefined: GithubRepo
make: *** [build] Error 2

==> installing...
cd cmd/tctest && go install -ldflags "-X github.com/katbyte/tctest/version.GitCommit=160e6b3" .
# github.com/katbyte/tctest/cmd/tctest/cli
cli/cmds.go:135:20: undefined: NewGithubRepoFromViper
cli/cmds.go:177:17: undefined: NewGithubRepoFromViper
cli/pr.go:17:10: undefined: GithubRepo
cli/pr.go:21:10: undefined: GithubRepo
cli/pr.go:35:10: undefined: Git

Does not return all results for subtests

When a test defines subtests, it does not return nested results. e.g.

$ tctest status 87870

--- PASS: TestAccAWSAccessAnalyzer (31.50s)

Whereas the results in TeamCity are

--- PASS: TestAccAWSAccessAnalyzer (31.50s)
    --- PASS: TestAccAWSAccessAnalyzer/Analyzer (31.50s)
        --- PASS: TestAccAWSAccessAnalyzer/Analyzer/basic (8.14s)
        --- PASS: TestAccAWSAccessAnalyzer/Analyzer/disappears (5.73s)
        --- PASS: TestAccAWSAccessAnalyzer/Analyzer/Tags (17.63s)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.