Acceptance tests for disaster recovery of Cloud Foundry

License: Apache License 2.0

Go 86.53% Shell 13.26% HTML 0.21%

disaster-recovery-acceptance-tests's Introduction

disaster-recovery-acceptance-tests (DRATs)

Tests if Cloud Foundry (CF) can be backed up and restored. The tests will back up from and restore to CF_DEPLOYMENT_NAME. Specifically, DRATs adds state to a Cloud Foundry deployment by testing backup and restore during a CF operation such as pushing an app. DRATs backs up the deployment, restores from the backup, and asserts that the state is present after restore.

Prerequisites

Install go

Running DRATs with Environment Variables

Spin up a Cloud Foundry deployment**.
- The backup-restore ops file must be used during deployment. This will also deploy a backup restore VM.
- Other backup-restore ops files will be needed depending on what test cases you want enabled.
Run DRATs against it using one of the following:
- Using an integration config - Recommended
- Using bbl
- Using environment variables - Deprecated

**Cloud Foundry on BOSH Lite is supported.

Test Structure

The system tests do the following:

Calls CheckDeployment(common.Config) on all provided TestCases (to e.g. check if errand-pushed apps are present).
Sets up a temporary local working directory for storing the backup artifact, and CF_HOME directories for all the test cases.
Calls BeforeBackup(common.Config) on all provided TestCases (to e.g. push unique apps to the environment to be backed up).
Backs up the CF_DEPLOYMENT_NAME Cloud Foundry deployment.
Calls AfterBackup(common.Config) on all provided TestCases.
Restores to the CF_DEPLOYMENT_NAME Cloud Foundry deployment.
Calls AfterRestore(common.Config) on all provided TestCases (to e.g. check the apps pushed are present in the restored environment).
Calls Cleanup(common.Config) on all provided TestCases (to e.g. clean up the apps from the backup environment). It will do this even if an error or failure occurred in a previous step
Cleans up the temporary directories created in the setup
If an error occurred during a bbr backup command, DRATs runs bbr backup-cleanup to remove temporary bbr artifacts from your deployment (which would otherwise cause subsequent DRATs runs to fail)

Extending DRATs

DRATs runs a collection of test cases against a Cloud Foundry deployment.

Test cases should be used for checking that CF components' data has been backed up and restored correctly – e.g. if your release backs up a table in a database, that the table can be altered and is then restored to its original state.

Backup and restore of apps is covered by the existing CAPI test case. No new test cases are needed for this – unless you're writing CAPI backup and restore scripts, app backup and restore can be assumed to work.

To add extra test cases, create a new TestCase that follows the TestCase interface.

The methods that need to be implemented are:

CheckDeployment(common.Config) runs first to ensure that the test case could possibly succeed in the deployment DRATs is running against. E.g. An errand-based test case could never succeed in a deployment where the errand has never been run.
BeforeBackup(common.Config) runs before the backup is taken, and should create state in the Cloud Foundry deployment to be backed up.
AfterBackup(common.Config) runs after the backup is complete but before the restore is started. If were monitoring e.g. app uptime during the backup you could use this step to stop monitoring knowing that backup definitely finished.
AfterRestore(common.Config) runs after the restore is complete, and should assert that the state in the restored Cloud Foundry deployment matches that created in BeforeBackup(common.Config).
Cleanup(common.Config) should clean up the state created in the Cloud Foundry deployment to be backed up.

common.Config contains the config for the BOSH Director and for the CF deployments to backup and restore.

Create a new TestCase in test_cases
In testcases/testcase_helper.go, initialise the TestCase and add it to the slice returned by OpenSourceTestCases()

Running DRATs in your CI

We provide tasks to run DRATs with your CI:

A drats task that reads in environment variables
A drats-with-integration-config task that read from an integration config.

Both DRATs tasks establish an SSH tunnel using sshuttle so that they can run from outside the network. Note the tasks will need to be run from a privileged container. You can also find our pipeline definition here

Debugging your DRATs run

DRATs runs multiple interwoven test cases (for app uptime and each of the components under test) so it can be a little tricky to work out what's gone wrong when there's an error or failure. Here are some tips on investigating DRATs failures - please PR in additions to this doc if you think of more tips that might help other teams!

The bbr backup-cleanup command runs if the test run errored during the bbr backup step. If you see an error in the backup-cleanup step, it's likely that a similar problem happened in the backup step which caused the original failure - scroll up to see.
The easiest way to see where the failure / error happened is to look for the nearest STEP statement in the logs

disaster-recovery-acceptance-tests's People

Contributors

Stargazers

Watchers

disaster-recovery-acceptance-tests's Issues

Consider retrying on temporary networking errors in test suite

Intermittently in our CI we see DRATs fail with networking errors similar to the following:

• Failure [557.066 seconds]
backing up Pivotal Cloud Foundry
/tmp/build/f353ecef/src/github.com/pivotal-cf/pivotal-disaster-recovery-acceptance-tests/acceptance_test.go:12
  backups and restores a cf [It]
  /tmp/build/f353ecef/src/github.com/pivotal-cf/pivotal-disaster-recovery-acceptance-tests/vendor/github.com/cloudfoundry-incubator/disaster-recovery-acceptance-tests/runner/runner.go:47

  Expected error:
      <*url.Error | 0xc002b342a0>: {
          Op: "Post",
          URL: "https://api.sys.shine-fin.aws.releng.cf-app.com/routing/v1/router_groups",
          Err: {
              Op: "read",
              Net: "tcp",
              Source: {IP: [10, 254, 1, 98], Port: 48704, Zone: ""},
              Addr: {IP: "4#\xdbS", Port: 443, Zone: ""},
              Err: {Syscall: "read", Err: 0x68},
          },
      }
      Post https://api.sys.shine-fin.aws.releng.cf-app.com/routing/v1/router_groups: read tcp 10.254.1.98:48704->52.35.219.83:443: read: connection reset by peer
  not to have occurred

  /tmp/build/f353ecef/src/github.com/pivotal-cf/pivotal-disaster-recovery-acceptance-tests/vendor/github.com/cloudfoundry-incubator/disaster-recovery-acceptance-tests/testcases/router_group_testcase.go:85

  Full Stack Trace
  	/tmp/build/f353ecef/src/github.com/pivotal-cf/pivotal-disaster-recovery-acceptance-tests/vendor/github.com/onsi/gomega/internal/assertion/assertion.go:75 +0x20e
  github.com/pivotal-cf/pivotal-disaster-recovery-acceptance-tests/vendor/github.com/onsi/gomega/internal/assertion.(*Assertion).NotTo(0xc001e6ab40, 0x9b12e0, 0xdd0130, 0x0, 0x0, 0x0, 0x0)
  	/tmp/build/f353ecef/src/github.com/pivotal-cf/pivotal-disaster-recovery-acceptance-tests/vendor/github.com/onsi/gomega/internal/assertion/assertion.go:48 +0xca
  github.com/pivotal-cf/pivotal-disaster-recovery-acceptance-tests/vendor/github.com/cloudfoundry-incubator/disaster-recovery-acceptance-tests/testcases.(*CfRouterGroupTestCase).AfterBackup(0xc0003280c0, 0xc0001c8220, 0x17, 0xc000235c80, 0x2f, 0xc00023b270, 0x5, 0xc00022b460, 0x20, 0xc0001c8880, ...)
  	/tmp/build/f353ecef/src/github.com/pivotal-cf/pivotal-disaster-recovery-acceptance-tests/vendor/github.com/cloudfoundry-incubator/disaster-recovery-acceptance-tests/testcases/router_group_testcase.go:85 +0x29b
  github.com/pivotal-cf/pivotal-disaster-recovery-acceptance-tests/vendor/github.com/cloudfoundry-incubator/disaster-recovery-acceptance-tests/runner.RunDisasterRecoveryAcceptanceTests.func2()
  	/tmp/build/f353ecef/src/github.com/pivotal-cf/pivotal-disaster-recovery-acceptance-tests/vendor/github.com/cloudfoundry-incubator/disaster-recovery-acceptance-tests/runner/runner.go:76 +0x76b
  github.com/pivotal-cf/pivotal-disaster-recovery-acceptance-tests/vendor/github.com/onsi/ginkgo/internal/leafnodes.(*runner).runSync(0xc00001e720, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
  	/tmp/build/f353ecef/src/github.com/pivotal-cf/pivotal-disaster-recovery-acceptance-tests/acceptance_suite_test.go:15 +0xb3
  testing.tRunner(0xc0004daa00, 0x94c1d8)
  	/usr/local/go/src/testing/testing.go:827 +0xbf
  created by testing.(*T).Run
  	/usr/local/go/src/testing/testing.go:878 +0x35c

Even in a healthy environment, it's not uncommon to hit networking errors occasionally. Could the DRATs test helpers be updated to retry on Temporary errors from the stdlib? General pattern looks something like this from the CF CLI.

DRATs fails with cf-cli 6.32.0

This is a fun one. We were unable to get DRATs to run successfully using cf-cli 6.32.0, but it passes using cf-cli 6.31.0. The tests fail with the following error:

Creating org acceptance-test-org-CJ3Wj-z3vJwlkoMq1CdPsw as admin...
Polling url test-app-opufssk-9km-dxfzgeejuw.dour-fancier.capi.land
OK
Done polling url test-app-opufssk-9km-dxfzgeejuw.dour-fancier.capi.land

Assigning role OrgManager to user admin in org acceptance-test-org-CJ3Wj-z3vJwlkoMq1CdPsw ...
Polling url test-app-opufssk-9km-dxfzgeejuw.dour-fancier.capi.land
Done polling url test-app-opufssk-9km-dxfzgeejuw.dour-fancier.capi.land
OK

TIP: Use 'cf target -o "acceptance-test-org-CJ3Wj-z3vJwlkoMq1CdPsw"' to target new org
FAILED

However, if you comment out one of the test cases (either NewCfAppTestCase or NewAppUptimeTestCase) it passes even with the latest CLI. What we think is happening is the latest CLI changed the ordering of when the config file is written to disk and running cf orgs in the background while the second test case is running seems to be hitting a race condition where the CLI config file is getting overridden.

The CLI says that running multiple cf commands in parallel is not support but you can ask the CLI team for more info. Maybe a quick fix is to curl the root endpoint of the CC API to see whether CC is down rather than using the cf cli.

Flaky test

Hello,

We, the networking program, run drats in our pipeline and we noticed that we keep getting one flaky test.

• Failure [1607.847 seconds]
backing up Cloud Foundry
/tmp/build/b7784222/src/github.com/cloudfoundry-incubator/disaster-recovery-acceptance-tests/acceptance/acceptance_test.go:12
  backups and restores a cf [It]
  /tmp/build/b7784222/src/github.com/cloudfoundry-incubator/disaster-recovery-acceptance-tests/runner/runner.go:52

  Timed out after 1200.000s.
  Command timed out: bbr deployment restore
  Expected process to exit.  It did not.

  /tmp/build/b7784222/src/github.com/cloudfoundry-incubator/disaster-recovery-acceptance-tests/runner/command_line_helpers.go:65

  Full Stack Trace
  	/tmp/build/b7784222/src/github.com/cloudfoundry-incubator/disaster-recovery-acceptance-tests/vendor/github.com/onsi/gomega/internal/asyncassertion/async_assertion.go:147 +0x403
  github.com/cloudfoundry-incubator/disaster-recovery-acceptance-tests/vendor/github.com/onsi/gomega/internal/asyncassertion.(*AsyncAssertion).Should(0xc000200340, 0x8af620, 0xc00039a8e0, 0xc0001fe610, 0x1, 0x1, 0x29)
  	/tmp/build/b7784222/src/github.com/cloudfoundry-incubator/disaster-recovery-acceptance-tests/vendor/github.com/onsi/gomega/internal/asyncassertion/async_assertion.go:48 +0x62
  github.com/cloudfoundry-incubator/disaster-recovery-acceptance-tests/runner.runCommandWithStream(0x845f7e, 0x16, 0x8ab860, 0xc00008ade0, 0x8ab860, 0xc00008ade0, 0xc0001dc1a0, 0x186, 0x0, 0x0, ...)
  	/tmp/build/b7784222/src/github.com/cloudfoundry-incubator/disaster-recovery-acceptance-tests/runner/command_line_helpers.go:65 +0x3a5
  github.com/cloudfoundry-incubator/disaster-recovery-acceptance-tests/runner.RunCommandSuccessfullyWithFailureMessage(0x845f7e, 0x16, 0xc0001dc1a0, 0x186, 0x0, 0x0, 0x0, 0xc000024780)
  	/tmp/build/b7784222/src/github.com/cloudfoundry-incubator/disaster-recovery-acceptance-tests/runner/command_line_helpers.go:31 +0xbe
  github.com/cloudfoundry-incubator/disaster-recovery-acceptance-tests/runner.RunDisasterRecoveryAcceptanceTests.func2()
  	/tmp/build/b7784222/src/github.com/cloudfoundry-incubator/disaster-recovery-acceptance-tests/runner/runner.go:120 +0xaad
  github.com/cloudfoundry-incubator/disaster-recovery-acceptance-tests/vendor/github.com/onsi/ginkgo/internal/leafnodes.(*runner).runSync(0xc00001e8a0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
  	/tmp/build/b7784222/src/github.com/cloudfoundry-incubator/disaster-recovery-acceptance-tests/acceptance/acceptance_suite_test.go:12 +0x63
  testing.tRunner(0xc000134900, 0x85c598)
  	/usr/local/go/src/testing/testing.go:827 +0xbf
  created by testing.(*T).Run
  	/usr/local/go/src/testing/testing.go:878 +0x35c

All of the flakes are identical. And we've gotten it on approximately 20% of the runs on this environment. However, we have another pipeline that runs drats on different env, which has never seen this flake.

Suspected Problem

Request

The environment that we are running it on is a very large, so maybe it need more than 1200s to backup and restore? We were wondering where the 1200s is coming from? Is there a way that we can override it with something larger?

Tagging other team members

@bruce-ricard

if slack is better, feel free to reach out to us in #networking in the cf slack.

What is proper value to GOPATH to run DRATS with environment variables?

I tried to run DRATS with Environment Variables in local.
I checked out disaster-recovery-acceptance-tests and set environments and run script like below.

/disaster-recovery-acceptance-tests# ./scripts/run_acceptance_tests_local.sh
./scripts/run_acceptance_tests_local.sh: line 14: GOPATH: parameter null or not set

Error occurred because GOPATH is not set.
What is proper value to GOPATH?

Credhub test case fails when `destroy_and_redeploy` flag is turned on

Hi @cloudfoundry-incubator/cf-credhub,

We, at the platform recovery team, start seeing the credhub testcase failures when we turn on the flag destroy_and_reploy, which will delete the targeted cf-deployment after taken the backup, redeploy with the same manifest, and then restore. Since the deployment is fresh, the environment variable CF_INSTANCE_GUID will change after restoring to the fresh deployment. As a result of that, the credhub testcase will fail here as error:

• Failure [2927.276 seconds]
backing up Cloud Foundry
/tmp/build/1f5f819e/src/github.com/cloudfoundry-incubator/disaster-recovery-acceptance-tests/acceptance/acceptance_test.go:12
  backups and restores a cf [It]
  /tmp/build/1f5f819e/src/github.com/cloudfoundry-incubator/disaster-recovery-acceptance-tests/runner/runner.go:47

  Expected
      <[]struct {} | len:0, cap:0>: []
  to have length 1

  /tmp/build/1f5f819e/src/github.com/cloudfoundry-incubator/disaster-recovery-acceptance-tests/testcases/credhub_instance_creds_testcase.go:67

  Full Stack Trace
  	/tmp/build/1f5f819e/src/github.com/cloudfoundry-incubator/disaster-recovery-acceptance-tests/vendor/github.com/onsi/gomega/internal/assertion/assertion.go:69 +0x1ed
  github.com/cloudfoundry-incubator/disaster-recovery-acceptance-tests/vendor/github.com/onsi/gomega/internal/assertion.(*Assertion).To(0xc4206d54c0, 0x865480, 0xc4204b3860, 0x0, 0x0, 0x0, 0xc4206d54c0)
  	/tmp/build/1f5f819e/src/github.com/cloudfoundry-incubator/disaster-recovery-acceptance-tests/vendor/github.com/onsi/gomega/internal/assertion/assertion.go:35 +0xae
  github.com/cloudfoundry-incubator/disaster-recovery-acceptance-tests/testcases.(*CfCredhubSSITestCase).AfterRestore(0xc420091040, 0xc420028c20, 0x2, 0xc420026270, 0x25, 0xc420028c22, 0x5, 0xc420022680, 0x14, 0xc420028c27, ...)
  	/tmp/build/1f5f819e/src/github.com/cloudfoundry-incubator/disaster-recovery-acceptance-tests/testcases/credhub_instance_creds_testcase.go:67 +0x249
  github.com/cloudfoundry-incubator/disaster-recovery-acceptance-tests/runner.RunDisasterRecoveryAcceptanceTests.func2()
  	/tmp/build/1f5f819e/src/github.com/cloudfoundry-incubator/disaster-recovery-acceptance-tests/runner/runner.go:138 +0xd60
  github.com/cloudfoundry-incubator/disaster-recovery-acceptance-tests/vendor/github.com/onsi/ginkgo/internal/leafnodes.(*runner).runSync(0xc42001e6c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
  	/tmp/build/1f5f819e/src/github.com/cloudfoundry-incubator/disaster-recovery-acceptance-tests/acceptance/acceptance_suite_test.go:12 +0x64
  testing.tRunner(0xc4202540f0, 0x8335b8)
  	/usr/local/go/src/testing/testing.go:777 +0xd0
  created by testing.(*T).Run
  	/usr/local/go/src/testing/testing.go:824 +0x2e0

This is due to the fact that the credhub test app relies on the environment variable CF_INSTANCE_GUID as part of the key for the credentials.

We think it makes sense to use a different identifier for the stored credential, which will remain the same after redeploying.

Thoughts?

Cheers,
Chunyi && @jamesjoshuahill

Update documentation to include setting release_level_backup to false for components backup and restore to be a no-op

My tests took about 50 minutes to run, I set theFOCUSED_SUITE_NAME=cf-routing but did not set release_level_backup which means the backup and restore scripts for every component run. This makes a very long run for DRATs. It might be better to explicitly ask team to set release_level_backup to false for components you want to exclude. This would make the run much shorter.

/scripts/run_acceptance_tests_local.sh and /scripts/run_acceptance_tests.sh take different env variables

During my testing, I tried to run /scripts/run_acceptance_tests.sh and /scripts/run_acceptance_tests_local.sh. I could run neither

/scripts/run_acceptance_tests_local.sh expects a BOSH_CA_CERT
and
/scripts/run_acceptance_tests.sh expects a BOSH_CERT_PATH for the director-ca-cert.

Failure leaves credhub in dirty state

We had a drats failure in an AfterBackup which caused the tests to exit.

When trying to re-run the tests, we continually get this error:

  Expected
      <[]struct { Name string } | len:2, cap:4>: [
          {
              Name: "/SECRET_PASSWORD/2",
          },
          {
              Name: "/SECRET_PASSWORD/1",
          },
      ]
  to have length 1

  /tmp/build/f353ecef/src/github.com/pivotal-cf/pivotal-disaster-recovery-acceptance-tests/vendor/github.com/cloudfoundry-incubator/disaster-recovery-acceptance-tests/testcases/credhub_instance_creds_testcase.go:67

This seems to be because in the first test run, the credhub_instance_creds_testcase.go ran the AfterBackup and created a second value in credhub. When the tests were run again, the BeforeBackup fails when verifying that only a single entry exists in credhub.

It might be better to have a clean-up of the credhub key at the start of the test to ensure a known starting state.

This is on an older commit of the drats: f73d02f32df7ffa6d9c95c64d8a08d480c6ede8d, but it seems like the problem still exists.

cc @jpalermo @cloudfoundry-incubator/pcf-release-engineering

Expect to see backup and restore only for the component specified in FOCUSED_SUITE_NAME to run

When I ran DRATs locally using the /scripts/run_acceptance_tests_local , and set the FOCUSED_SUITE_NAME = cf-routing, I would expect only the part of the acceptance tests for routing to run but I noticed it ran all test cases. You can see it in the output.

TestcaseHelper OpenSourceTestCasesWithRegex
  returns all cases if no skip or focus provided
  /Users/sanjurtupil/go/src/github.com/cloudfoundry-incubator/disaster-recovery-acceptance-tests/testcases/testcase_helper_test.go:33
•
------------------------------
TestcaseHelper OpenSourceTestCasesWithRegex
  Focusses on a single case
  /Users/sanjurtupil/go/src/github.com/cloudfoundry-incubator/disaster-recovery-acceptance-tests/testcases/testcase_helper_test.go:40
Running cf-nfsbroker test case
•
------------------------------
TestcaseHelper OpenSourceTestCasesWithRegex
  Focusses on multiple cases
  /Users/sanjurtupil/go/src/github.com/cloudfoundry-incubator/disaster-recovery-acceptance-tests/testcases/testcase_helper_test.go:47
Running cf-uaa test case
Running cf-nfsbroker test case
•
------------------------------
TestcaseHelper OpenSourceTestCasesWithRegex
  Excludes a case
  /Users/sanjurtupil/go/src/github.com/cloudfoundry-incubator/disaster-recovery-acceptance-tests/testcases/testcase_helper_test.go:53
Running cf-routing test case
Running app-uptime test case
Running cf-app test case
Running cf-uaa test case
Running cf-networking test case
•
------------------------------
TestcaseHelper OpenSourceTestCasesWithRegex
  Panics if no test cases matching the suite name found
  /Users/sanjurtupil/go/src/github.com/cloudfoundry-incubator/disaster-recovery-acceptance-tests/testcases/testcase_helper_test.go:61
•
Ran 8 of 8 Specs in 0.001 seconds
SUCCESS! -- 8 Passed | 0 Failed | 0 Pending | 0 Skipped PASS

bbr-run-drats tests fail after "go toolchains" update

Hi bbr team,
the bbr-run-drats tests are failing:
https://concourse.wg-ard.ci.cloudfoundry.org/teams/main/pipelines/cf-deployment/jobs/bbr-run-drats/builds/899

The problem seems to be that the "credhub-test-app" now uses go toolchains which require golang 1.21.
https://github.com/cloudfoundry/disaster-recovery-acceptance-tests/tree/main/fixtures/credhub-test-app

The app however uses golang 1.20 as runtime:

cf logs appC6l4lgUqBGKTVbwUo6A7xA --recent
Retrieving logs for app appC6l4lgUqBGKTVbwUo6A7xA in org acceptance-test-org-C6l4lgUqBGKTVbwUo6A7xA / space acceptance-test-space-C6l4lgUqBGKTVbwUo6A7xA as admin...

   2023-12-20T08:21:35.79+0000 [API/0] OUT Added process: "web"
   2023-12-20T08:21:35.80+0000 [API/0] OUT Created app with guid 195540b6-fe0a-4e53-a25d-a91c306d36b5
   2023-12-20T08:21:35.81+0000 [API/0] OUT Applied manifest to app with guid 195540b6-fe0a-4e53-a25d-a91c306d36b5 (---
   2023-12-20T08:21:35.82+0000 [API/0] OUT applications:
   2023-12-20T08:21:35.82+0000 [API/0] OUT - name: appC6l4lgUqBGKTVbwUo6A7xA
   2023-12-20T08:21:35.82+0000 [API/0] OUT path: "/tmp/build/b7784222/src/github.com/cloudfoundry/disaster-recovery-acceptance-tests/fixtures/credhub-test-app"
   2023-12-20T08:21:35.82+0000 [API/0] OUT default-route: true
   2023-12-20T08:21:35.82+0000 [API/0] OUT buildpacks:
   2023-12-20T08:21:35.82+0000 [API/0] OUT - go_buildpack
   2023-12-20T08:21:35.82+0000 [API/0] OUT env: "[PRIVATE DATA HIDDEN]"
   2023-12-20T08:21:35.82+0000 [API/0] OUT )
   2023-12-20T08:21:39.56+0000 [API/1] OUT Updated app with guid 195540b6-fe0a-4e53-a25d-a91c306d36b5 ({"lifecycle"=>{"data"=>{"buildpacks"=>["go_buildpack"]}}})
   2023-12-20T08:21:39.70+0000 [API/1] OUT Updated app with guid 195540b6-fe0a-4e53-a25d-a91c306d36b5 ({"environment_variables"=>"[PRIVATE DATA HIDDEN]"})
   2023-12-20T08:21:42.20+0000 [API/1] OUT Uploading app package for app with guid 195540b6-fe0a-4e53-a25d-a91c306d36b5
   2023-12-20T08:21:45.59+0000 [API/0] OUT Updated app with guid 195540b6-fe0a-4e53-a25d-a91c306d36b5 ({"environment_variables"=>"[PRIVATE DATA HIDDEN]"})
   2023-12-20T08:21:45.71+0000 [API/0] OUT Updated app with guid 195540b6-fe0a-4e53-a25d-a91c306d36b5 ({"environment_variables"=>"[PRIVATE DATA HIDDEN]"})
   2023-12-20T08:21:46.04+0000 [API/0] OUT Creating build for app with guid 195540b6-fe0a-4e53-a25d-a91c306d36b5
   2023-12-20T08:21:46.19+0000 [STG/0] OUT Downloading go_buildpack...
   2023-12-20T08:21:46.19+0000 [STG/0] OUT Downloaded go_buildpack
   2023-12-20T08:21:46.19+0000 [STG/0] OUT Cell f8937c82-8d8f-484c-9cc5-b0d4237db41e creating container for instance a790ca6f-fded-4396-b655-796dcc7992fb
   2023-12-20T08:21:46.55+0000 [STG/0] OUT Security group rules were updated
   2023-12-20T08:21:46.57+0000 [STG/0] OUT Cell f8937c82-8d8f-484c-9cc5-b0d4237db41e successfully created container for instance a790ca6f-fded-4396-b655-796dcc7992fb
   2023-12-20T08:21:46.84+0000 [STG/0] OUT Downloading app package...
   2023-12-20T08:21:46.90+0000 [STG/0] OUT Downloaded app package (285.5K)
   2023-12-20T08:21:47.07+0000 [STG/0] OUT -----> Go Buildpack version 1.10.14
   2023-12-20T08:21:47.07+0000 [STG/0] OUT -----> Installing godep 80
   2023-12-20T08:21:47.07+0000 [STG/0] OUT Download [https://buildpacks.cloudfoundry.org/dependencies/godep/godep_80_linux_x64_cflinuxfs4_20fea317.tgz]
   2023-12-20T08:21:47.37+0000 [STG/0] OUT -----> Installing glide 0.13.3
   2023-12-20T08:21:47.37+0000 [STG/0] OUT Download [https://buildpacks.cloudfoundry.org/dependencies/glide/glide_0.13.3_linux_x64_cflinuxfs4_be64c2ea.tgz]
   2023-12-20T08:21:47.64+0000 [STG/0] OUT -----> Installing dep 0.5.4
   2023-12-20T08:21:47.64+0000 [STG/0] OUT Download [https://buildpacks.cloudfoundry.org/dependencies/dep/dep_0.5.4_linux_x64_cflinuxfs4_a4d7f7ea.tgz]
   2023-12-20T08:21:47.94+0000 [STG/0] OUT -----> Installing go 1.20.11
   2023-12-20T08:21:47.94+0000 [STG/0] OUT Download [https://buildpacks.cloudfoundry.org/dependencies/go/go_1.20.11_linux_x64_cflinuxfs4_7003fc8b.tgz]
   2023-12-20T08:21:53.65+0000 [STG/0] OUT **ERROR** problem retrieving main package name: go: errors parsing go.mod:
   2023-12-20T08:21:53.65+0000 [STG/0] OUT /tmp/app/go.mod:5: unknown directive: toolchain
   2023-12-20T08:21:53.65+0000 [STG/0] OUT
   2023-12-20T08:21:53.65+0000 [STG/0] OUT **ERROR** Unable to determine import path: exit status 1
   2023-12-20T08:21:54.06+0000 [STG/0] ERR Failed to compile droplet: Failed to run finalize script: exit status 12
   2023-12-20T08:21:54.07+0000 [STG/0] OUT Exit status 223
   2023-12-20T08:21:54.53+0000 [STG/0] OUT Cell f8937c82-8d8f-484c-9cc5-b0d4237db41e stopping instance a790ca6f-fded-4396-b655-796dcc7992fb
   2023-12-20T08:21:54.53+0000 [STG/0] OUT Cell f8937c82-8d8f-484c-9cc5-b0d4237db41e destroying container for instance a790ca6f-fded-4396-b655-796dcc7992fb
   2023-12-20T08:21:54.59+0000 [API/1] ERR Failed to stage build: staging failed
   2023-12-20T08:21:56.09+0000 [STG/0] OUT Cell f8937c82-8d8f-484c-9cc5-b0d4237db41e successfully destroyed container for instance a790ca6f-fded-4396-b655-796dcc7992fb

Can you please check?

Desire to run DRATS with an integration config

Hi friends!

We are interested in running DRATS with an integration config instead of environment variables like CATS and the other -ATS tests. Is this something you all are planning in your backlog? Would you like a pull request for this? It would be nice for all the -ATS tests to be set up the same way.

Thanks from the Relint team!

DRATs failing in cf-deployment pipeline

Hi BBR Team,

the DRATs are currently failing in the cf-deployment pipeline:
https://concourse.wg-ard.ci.cloudfoundry.org/teams/main/pipelines/cf-deployment/jobs/bbr-run-drats/builds/1339

It could be related to the removal of the CF CLI v6 from the BOSH cf-cli package:
cloudfoundry/cf-deployment#1184

Can you please check? This is blocking the cf-deployment pipeline. I can provide access to the bbl env, if needed.

Thanks for your support and best regards,

Jochen.

Is there a way to run Drats against a cf deployment with windows cell?

Hi:

When we run drats test against a cf-deployment setup with windows cell enabled, we see errors like

failed to find instances for deployment cf: couldn't find jobs: finding scripts failed on windows2016-cell/0bca09ab-c6ae-4f7e-b9e2-156b41a7f0a9: 'sudo' is not recognized as an internal or external command,

Is there a way to exclude windows cell? what is the right approach running it?
With many thanks.

DRATs cannot run against a credhub-enabled environment

It looks like DRATs expects to pull the CF admin password from the deployment manifest. This doesn't work in credhub-enabled envs as the manifest only contains references to the password (((cf_admin_password))).

We are currently using a fork of DRATs that takes in CF_ADMIN_PASSWORD as an env variable instead. This will break the DRATs flow of backing up from one env and restoring to another unless they both have the same admin password.

/scripts/run_acceptance_tests_local.sh does not run without certain code modifications

I made some modifications to the script locally to be able to run the tests. Here are the modifications I made to the run_acceptance_tests_local.sh script, ideally I would want to be able to run this without modifications. Commenting out everywhere that CF_HOME is used.

39                   //for _, testCase := range testCases {
40                 //      err := os.Mkdir(cfHomeDir(testCase), 0700)
41                 //      Expect(err).NotTo(HaveOccurred())
42                 //}

52                  //      os.Setenv("CF_HOME", cfHomeDir(testCase))

and cleanup

/*
 99                 By("running bbr backup-cleanup")
100                 Eventually(RunCommandSuccessfully(fmt.Sprintf(
101                         "cd %s && %s deployment --target %s --ca-cert %s --username %s --password %s --deployment %s backup-cleanup",
102                         testContext.WorkspaceDir,
103                         testContext.BinaryPath,
104                         config.BoshConfig.BoshURL,
105                         testContext.CertificatePath,
106                         config.BoshConfig.BoshClient,
107                         config.BoshConfig.BoshClientSecret,
108                         config.DeploymentToBackup.Name,
109                 ))).Should(gexec.Exit(0))
110                 */

ci/drats-with-integration-config/task.yml does not expose `DEFAULT_TIMEOUT_MINS`, `FOCUSED_SUITE_NAME`, and `SKIP_SUITE_NAME` params

We need this variables for our drats to run and are not present in the integration config. Please add them back in. :)

Please configure GITBOT

Pivotal uses GITBOT to synchronize Github issues and pull requests with Pivotal Tracker.
Please add your new repo to the GITBOT config-production.yml in the Gitbot configuration repo.
If you don't have access you can send an ask ticket to the CF admins. We prefer teams to submit their changes via a pull request.

Steps:

Fork this repo: cfgitbot-config
Add your project to config-production.yml file
Submit a PR

If there are any questions, please reach out to [email protected].

Docs for env-vars are incorrect

We tried to use the documentation to run the acceptance tests locally. The documentation specifies setting the following environment variables

CF_DEPLOYMENT_NAME - name of the Cloud Foundry deployment to backup and restore
CF_API_URL - Cloud Foundry api url
CF_ADMIN_USERNAME - Cloud Foundry api admin user
CF_ADMIN_PASSWORD - Cloud Foundry api admin password
BOSH_ENVIRONMENT - URL of BOSH Director which has deployed the above Cloud Foundries
BOSH_CLIENT - BOSH Director username
BOSH_CLIENT_SECRET - BOSH Director password
BOSH_CERT_PATH - path to BOSH Director's CA cert
BBR_BUILD_PATH - path to BBR binary
DEFAULT_TIMEOUT_MINS - timeout for commands run in the test. Defaults to 15 minutes.

On running ./scripts/run_acceptance_tests_local.sh we have encountered the following errors:

./scripts/run_acceptance_tests_local.sh: line 8: BOSH_CA_CERT: parameter null or not set

and

./scripts/run_acceptance_tests_local.sh: line 9: BOSH_GW_HOST: parameter null or not set

BOSH_GW_HOST is not included in the documentation at all.

drats is failing due to IaaS flakiness

Persi team runs drats via https://raw.githubusercontent.com/cloudfoundry-incubator/disaster-recovery-acceptance-tests/master/ci/drats/task.yml in our pipeline and since moving to wings we are seeing some flakiness when running drats.

This cf-d we are running drats against:

is a long-lived environment
is also single AZ

It would be nice to have a flake attempts flag similar to what CATs has (which sets the --flakeAttempts on ginkgo) i.e. https://github.com/cloudfoundry/cf-deployment-concourse-tasks/blob/master/run-cats/task.yml#L52

cc/ @paulcwarren @julian-hj

cloudfoundry / disaster-recovery-acceptance-tests Goto Github PK