See the package documentation for detail on what the package is and how to use it.

If you are migrating a service to DBT PaaS, GOV.UK PaaS to DBT PaaS Migration will also be relevant for you.

3.9, 3.10, 3.11 and 3.12.

1. Clone the repository:

   git clone https://github.com/uktrade/platform-tools.git && cd platform-tools
   

2. Install the required dependencies:

   pip install poetry && poetry install && poetry run pre-commit install
   

The following tools are required to run the full test suite.

• checkov
• AWS Copilot

Run poetry run pytest in the root directory to run all tests.

Or, run poetry run tox in the root directory to run all tests for multiple Python versions. See the tox configuration file.

Note: by default the tests are run using multiple processes for speed. When running using multiple processes pdb (python debugger) does not play nicely and will error.

To allow pdb to work correctly, disable multiple processes using the --numprocesses 0 option:

poetry run pytest --numprocesses 0

Amongst other things designed to exercise our tools together, the regression tests will attempt to deploy demodjango to the toolspr environment and run the smoke tests.

At present, this is currently only triggered on merges to the main branch for this code base and on a schedule early every morning.

You can manually trigger a run from the platform-tools-test AWS CodeBuild project.

To test your platform-tools changes before merging, use "Start build with overrides" and set the "Source version" to your branch name.

You may wish to run tests against a demodjango environment other than toolspr.

In order for this to work, you will need to have deployed environment and codebase pipelines for your environment. See the toolspr examples in demodjango-deploy/platform-config.yml.

To run the regression tests against your environment, select "Start build with overrides", navigate to "Additional configuration" in the "Environment" section, and set TARGET_ENVIRONMENT environment variable value to your environment name.

Because we are currently targeting the same environment for all runs and AWS CodeBuild does not support queueing, it is essential that we do not start a regression test run while another is in progress, communicate with the team and check in the platform-tools-test AWS CodeBuild project.

You may want to test any CLI changes locally.

Run poetry build to build your package resulting in a package file (e.g. dbt_platform_tools-0.1.40.tar.gz) in a dist folder. You may need to bump up the package version before doing so.

Copy the package file(s) to the directory where you would like to test your changes, and make sure you are in a virtual environment. Run platform-helper --version to check the installed package version (e.g. 0.1.39).

Run pip install <file> and confirm the installation has worked by running platform-helper --version which would output version 0.1.40 following our example.

Publishing to PyPI happens automatically when a GitHub Release is published. To publish the Python package dbt-platform-helper manually, you will need an API token.

1. Acquire API token from Passman.
   • Request access from the SRE team.
   • Note: You will need access to the platform group in Passman.
2. Run poetry config pypi-token.pypi <token> to add the token to your Poetry configuration.

Update the version, as the same version cannot be published to PyPI.

poetry version patch

More options for the version command can be found in the Poetry documentation. For example, for a minor version bump: poetry version minor.

Build the Python package.

poetry build

Publish the Python package.

Note: Make sure your Pull Request (PR) is approved and contains the version upgrade in pyproject.toml before publishing the package.

poetry publish

Check the PyPI Release history to make sure the package has been updated.

For an optional manual check, install the package locally and test everything works as expected.

• Use Conventional Commits in our pull request titles. These become the release commit messages in the main branch. The advantage of this is that it's clear from a list of commits, what kind of changes they include (fix=patch, feat=minor, !=breaking change) and most importantly whether commits include breaking changes
• Use GitHub Releases to document and manage releases
• Non-breaking changes already in the main branch should be released before merging a breaking change
• Minor and patch releases should always include zero downtime
• Documentation for releases with breaking changes should include the upgrade path
• Where possible the upgrade path should allow for zero downtime, if this is not possible, it should be flagged up big and visible

• Merging to main will trigger a CodeBuild project called platform-tools-test in the platform-tools AWS account to run regression tests on merge to main / pull request created / pull request updated events emitted by GitHub
• We use the release-please GitHub action to create and update a release PR when changes are merged to main
  • The release PR will automatically update the pyproject.toml version number and generate release notes based on the commits merged since the last release
  • Merging the release PR will create a draft GitHub release for the next version with release notes

Publishing a GitHub release should automatically:

• Run the full regression pipeline (currently WIP)
• Trigger a CodeBuild project called platform-tools-build in the platform-tools AWS account to run. This runs the buildspec-pypi.yml file which contains the build steps to publish the new platform-helper package version to PyPI
• Trigger a rebuild of the DBT Platform Documentation, so it includes the latest release documentation (currently WIP)
• Push a notification to the development community via the #developers channel in Slack

Successful completion of the CodeBuild project platform-tools-build in the platform-tools AWS account executes the buildspec-pypi.yml file which publishes the new application version to PyPI. This build process involves the following steps:

• Retrieving a list of prior releases from the dbt-platform-helper PyPI project repository.
• Verifying if the dbt-platform-helper package version in the root pyproject.toml file does not exist in the list of releases obtained from PyPI.
• If the version does not exist in the PyPI release list, the application is built and published to PyPI as a new release with the version stated in the application pyproject.toml file
• Next the script again checks if the updated version number is included in the PyPI release list. If found, it indicates that the new package version exists in PyPI.
• A Slack message is sent to the #developers channel to notify others that the platform-helper tool has been updated

1. Work on a new branch
2. Create a PR and have it reviewed
3. Once approved:
   • If it is a breaking change, you must release any outstanding non breaking changes before merging
   • Merge to main
4. A release PR will automatically be created when changes are merged to main
   • The release PR is updated with next version number and release notes based on the commits since the last release
5. Merge the release PR to create a GitHub release
6. Ensure the release notes contain an upgrade path for any breaking changes
7. Check PyPI for the new published version