See the package documentation for detail on what the package is and how to use it.
If you are migrating a service to DBT PaaS, GOV.UK PaaS to DBT PaaS Migration will also be relevant for you.
3.9, 3.10, 3.11 and 3.12.
-
Clone the repository:
git clone https://github.com/uktrade/platform-tools.git && cd platform-tools
-
Install the required dependencies:
pip install poetry && poetry install && poetry run pre-commit install
The following tools are required to run the full test suite.
Run poetry run pytest
in the root directory to run all tests.
Or, run poetry run tox
in the root directory to run all tests for multiple Python versions. See the tox
configuration file.
Note: by default the tests are run using multiple processes for speed. When running using multiple processes pdb (python debugger) does not play nicely and will error.
To allow pdb to work correctly, disable multiple processes using the --numprocesses 0
option:
poetry run pytest --numprocesses 0
Amongst other things designed to exercise our tools together, the regression tests will attempt to deploy demodjango
to the toolspr
environment and run the smoke tests.
At present, this is currently only triggered on merges to the main
branch for this code base and on a schedule early every morning.
You can manually trigger a run from the platform-tools-test
AWS CodeBuild project.
To test your platform-tools
changes before merging, use "Start build with overrides" and set the "Source version" to your branch name.
You may wish to run tests against a demodjango
environment other than toolspr
.
In order for this to work, you will need to have deployed environment and codebase pipelines for your environment. See the toolspr
examples in demodjango-deploy/platform-config.yml.
To run the regression tests against your environment, select "Start build with overrides", navigate to "Additional configuration" in the "Environment" section, and set TARGET_ENVIRONMENT
environment variable value to your environment name.
Because we are currently targeting the same environment for all runs and AWS CodeBuild does not support queueing, it is essential that we do not start a regression test run while another is in progress, communicate with the team and check in the platform-tools-test
AWS CodeBuild project.
You may want to test any CLI changes locally.
Run poetry build
to build your package resulting in a package file (e.g. dbt_platform_tools-0.1.40.tar.gz
) in a dist
folder. You may need to bump up the package version before doing so.
Copy the package file(s) to the directory where you would like to test your changes, and make sure you are in a virtual environment. Run platform-helper --version
to check the installed package version (e.g. 0.1.39
).
Note
Copying the package file is optional, but recommended. You can keep the package file in the dist
folder and install the package from your directory.
Run pip install <file>
and confirm the installation has worked by running platform-helper --version
which would output version 0.1.40
following our example.
Important
When testing is complete, do not forget to revert the dbt-platform-helper
installation back to what it was; e.g. pip install dbt-platform-helper==0.1.39
.
Publishing to PyPI happens automatically when a GitHub Release is published. To publish the Python package dbt-platform-helper
manually, you will need an API token.
- Acquire API token from Passman.
- Request access from the SRE team.
- Note: You will need access to the
platform
group in Passman.
- Run
poetry config pypi-token.pypi <token>
to add the token to your Poetry configuration.
Update the version, as the same version cannot be published to PyPI.
poetry version patch
More options for the version
command can be found in the Poetry documentation. For example, for a minor version bump: poetry version minor
.
Build the Python package.
poetry build
Publish the Python package.
Note: Make sure your Pull Request (PR) is approved and contains the version upgrade in pyproject.toml
before publishing the package.
poetry publish
Check the PyPI Release history to make sure the package has been updated.
For an optional manual check, install the package locally and test everything works as expected.
- Use Conventional Commits in our pull request titles. These become the release commit messages in the
main
branch. The advantage of this is that it's clear from a list of commits, what kind of changes they include (fix=patch, feat=minor, !=breaking change) and most importantly whether commits include breaking changes - Use GitHub Releases to document and manage releases
- Non-breaking changes already in the
main
branch should be released before merging a breaking change - Minor and patch releases should always include zero downtime
- Documentation for releases with breaking changes should include the upgrade path
- Where possible the upgrade path should allow for zero downtime, if this is not possible, it should be flagged up big and visible
- Merging to
main
will trigger a CodeBuild project calledplatform-tools-test
in the platform-tools AWS account to run regression tests onmerge to main / pull request created / pull request updated
events emitted by GitHub - We use the
release-please
GitHub action to create and update a release PR when changes are merged tomain
- The release PR will automatically update the pyproject.toml version number and generate release notes based on the commits merged since the last release
- Merging the release PR will create a draft GitHub release for the next version with release notes
Publishing a GitHub release should automatically:
- Run the full regression pipeline (currently WIP)
- Trigger a CodeBuild project called
platform-tools-build
in the platform-tools AWS account to run. This runs the buildspec-pypi.yml file which contains the build steps to publish the newplatform-helper
package version to PyPI - Trigger a rebuild of the DBT Platform Documentation, so it includes the latest release documentation (currently WIP)
- Push a notification to the development community via the #developers channel in Slack
Successful completion of the CodeBuild project platform-tools-build
in the platform-tools AWS account executes the buildspec-pypi.yml file which publishes the new application version to PyPI. This build process involves the following steps:
- Retrieving a list of prior releases from the
dbt-platform-helper
PyPI project repository. - Verifying if the
dbt-platform-helper
package version in the root pyproject.toml file does not exist in the list of releases obtained from PyPI. - If the version does not exist in the PyPI release list, the application is built and published to PyPI as a new release with the version stated in the application pyproject.toml file
- Next the script again checks if the updated version number is included in the PyPI release list. If found, it indicates that the new package version exists in PyPI.
- A Slack message is sent to the
#developers
channel to notify others that the platform-helper tool has been updated
- Work on a new branch
- Create a PR and have it reviewed
- Once approved:
- If it is a breaking change, you must release any outstanding non breaking changes before merging
- Merge to
main
- A release PR will automatically be created when changes are merged to main
- The release PR is updated with next version number and release notes based on the commits since the last release
- Merge the release PR to create a GitHub release
- Ensure the release notes contain an upgrade path for any breaking changes
- Check PyPI for the new published version