This repository contains the core management system for Max Planck Institute Data Systems Group's Satellite Data Augmentation Project. It is being developed by DAD (Dream, Aim, Deliver) as part of the collaboration with the Max Planck Institute.

Please fork this repository to your own GitHub account and clone it to your local machine. Please avoid pushing to this repository directory directly, either to the main branch or to any other branch.

gh repo clone <your-username>/kernel-planckster

git remote add upstream https://github.com/dream-aim-deliver/kernel-planckster.git

Then head over to

https://github.com/<your_username>/kernel-planckster/settings/actions

and enable GitHub Actions for your fork by selecting the Allow all actions and reusable workflows option.

# AT THE ROOT OF THE PROJECT

python3 -m venv .venv
source .venv/bin/activate

# Install poetry
pip install poetry

# Install dependencies
poetry install

# Setup pre-commit
pre-commit install
pre-commit run --all-files

# Create a .env file
cp .env.example .env

# Set up environment variables for pytest in pyproject.toml as needed, but the defaults should work

Using docker containers to spin up an SQL database, you can autogenerate migrations with Alembic. From the root of the project, do:

docker compose up -d
alembic upgrade head
alembic revision --autogenerate -m "migration message"
alembic upgrade head
alembic downgrade base
alembic upgrade head
docker compose down

Make sure to fix any errors given by the alembic commands above before executing the next one and commiting the changes.

For development, the environment variable KP_MODE has to be set to development. This is the default value used by the docker-compose.yml and launch.json files.

In development mode, postgres, fastapi and adminer are all running in docker containers. These containers will be started automatically when you start FastAPI launch configuration in VSCode or when you run the dev script in the root of the project, as shown below:

poetry run dev

The containers will be removed when you stop the dev script or stop the FastAPI launch configuration in VSCode.

FastAPI provides the Swagger UI at http://localhost:8000/docs to manually test the API endpoints, and the ReDoc UI at http://localhost:8000/redoc for a documentation view of the endpoints. In development mode, you can access Adminer interface at http://localhost:8080 to check the database. The credentials are:

System: PostgreSQL
Server: db
Username: postgres
Password: postgres
Database: kp-db

To test the object store in dev mode, you can run:

poetry run dev --storage

This will do the same as poetry run dev but also start a minio container:

You can access MinIO at http://localhost:9001 to check the object storage. The credentials are:

Access Key: minio
Secret Key: minio123

You run tests on the command line with

poetry run pytest -s

Or in the VSCode test explorer UI.

In test mode, postgres, fastapi, adminer, and minio are all running docker containers and will be automatically removed once the tests are finished.

DANGLING CONTAINER WARNING: If you are debugging tests in VSCode and notice that your test is failing, which triggers a breakpoint, then DO NOT STOP the debugger. Just press F5 and let the tests finish. The containers will be removed automatically once the tests are finished. Otherwise, you will end up with dangling test containers that will have to be removed manually. To remove these containers manually:

cd tests
docker compose down

In testing mode, you can access the services as follows:

In test mode, you can access Adminer interface at http://localhost:8085 to check the database. The credentials are:

System: PostgreSQL
Server: db
Username: postgres
Password: postgres
Database: kp-db

You can also access MinIO at http://localhost:9002 to check the object storage. The credentials are defined in the pyproject.toml file, under the pytest ini_options section.

You can start the FastAPI production server with:

 cd lib/infrastructure/rest
 export KP_MODE=production
 export KP_DB_URL=postgresql://postgres:postgres@localhost:5432/kp-db
 poetry run dev

We use VSCode as our IDE. If you use VSCode, please install the recommended extensions.

Please use Issues to report any bugs or feature requests.

Once you have been assigned an issue, please create a branch with the following naming convention:

feature-<issue number>-<short description>

We recommend using the provided create-feature-branch utility to create a branch with the correct name. This script will also pull the latest changes from the remote repository and create the new branch for you to work in.

./tools/create-feature-branch <issue number> <short description>

Commit your change. The commit command must include a specific message format:

git commit -m "<component>: <change_message> #<issue number>"

Valid component names are listed in the label list and are usually specified on the issue of the change.

Add additional explanations to the body of the commit, such as motivation for certain decisions and background information. Here are some general rules: https://cbea.ms/git-commit/.

Using multiple commits is allowed as long as they achieve an independent, well-defined, change and are well-described. Otherwise multiple commits should be squashed.

Before submitting a pull request, please:

1. Run pytest, at the root of the project, and fix all the errors:

poetry run pytest -s

2. Run mypy, at the root of the project, and fix all type errors:

poetry run mypy .

3. Run black, at the root of the project

poetry run black .

Push the commit to your forked repository and create the pull request. Try to keep the Pull Request simple, it should achieve the single objective described in the issue. Multiple enhancements/fixes should be split into multiple Pull Requests.

Watch the pull request for comments and reviews. For any pull requests update, please try to squash/amend your commits to avoid “in-between” commits.

If you add a github-recognised keyword in the pull request name or in the pull request description, then the associated issue can be closed automatically once the pull request is merged, e.g.:

<component>: <change_message> Fix #<issue number>