Performance tests integration

• VS: Prepare a dedicated server for hosting the Drone CI server and runner. Create a descriptive DNS record for easy access.
• MINA: On GitHub, create an OAuth app. Navigate to Settings -> Developer settings -> OAuth apps and fill out the form with the homepage URL as <drone ci url> and Authorization callback URL as <drone ci url + /login>.
• MINA: Generate a new client secret on the same page. Be sure to copy the secret as it's only displayed once, upon creation.
• MINA: Click on 'Update App' to save your changes.
• MINA: Copy the client ID and share it with VS along with the client secret. These will be used for authentication with the Drone server.
• VS: Launch the Drone server and runner using the provided ID and secret.
• MINA: Log in to the <drone ci url> using an account that has organizational access to MINA. This will authenticate with the previously created OAuth app. If you are already logged into GitHub, you will be prompted for authorization. Grant organization access to the MINA organization at this stage.
• MINA: On the 'Complete your Drone Registration' screen, you can simply leave the fields empty and click 'SUBMIT'.
• VS: Add the registered user to the Drone CI admins.
• MINA: You will see a list of organization repositories on the dashboard. Click on the MINA repository and then on 'ACTIVATE REPOSITORY'. This will take you to the repository settings.
• VS: In the settings, set the timeout to 4 hours to ensure the performance tests don't prematurely timeout.
• VS: Add the secrets (k8s_config, docker_hub creds) to the secrets

Notes

Permissions

The OAuth scopes granted to Drone are as follows:

• repo
• repo:status
• user:email
• read:org

Despite the breadth of these permissions, they are necessary for Drone CI to perform its designated functions. This is due to GitHub's OAuth scopes structure, which bundles a set of permissions together in each scope. For instance, the 'repo' scope includes both read and write access to repositories. While this may seem extensive, Drone CI uses this access responsibly and only for specific tasks like managing webhooks. Ideally, GitHub's OAuth scopes would be more granular, allowing applications to request only the necessary permissions. It's important to note that while Drone has write access to your repositories, it does not have the ability to delete repositories or manage administrative settings.

Preparation (TODO):

• Create a TODO list for o1labs
• Test the TODO using test accounts and orgs
• Create a namespace in our cluster
• Cleanup the commits for a PR (only include relevant commits to the helm charts and drone)
• Rebase on develop
• Cleanup possible issues after rebasing
• Add modification to .drone.yml that eliminates the need to use charts from our fork
• Launch the pref CI tooling (perf CI FE + aggregator) in the created namespace (minaprotocol-ci-perftest)
• Prepare a fresh server + domain
• Verify the tutorial in https://github.com/openmina/mina-ci is still up to date
• Test a few builds

Remove Drone CI from the performance test CI, so we won't need write access to the MinaProtocol organization repositories. Drone, with its minimal scopes, requires write access to repositories. Drone is also very tightly integrated with source control management (source), so we cannot lower the scopes it requires.

With GitHub Actions, which is tightly integrated with GitHub, the Mina team will only need to include a few secrets into the repository (Docker Hub access, our cluster config), and we won't require any additional permissions.

TODO:

Convert the Drone pipelines in .drone.yml to GitHub workflows:

• Build the Docker image
• Deploy nodes to the cluster
• Run transaction load and network split test
• Clean up deployments after each test instead of leaving them running

Modify the aggregator app, so it won't query Drone for build info. Here, we need to consider the limits GitHub applies to its API:

• Instead of querying GitHub's API for new builds, query the cluster for new deployments
• When a new deployment is seen in the cluster, get the required info once from the GitHub API
• Instead of relying on a finishing step in CI, check the nodes in the cluster directly for the finishing condition (block height)
• Use a configMap to notify the aggregator on status changes and build info
• Watch the configMap for changes, start/stop aggregation based on these notification
• Testing:
  • Test the new notication system
  • Test the whole setup e2e

• Aggregator in separate repo
• Implement pull architecture instead of push
• Aggregate debugger responses: http://1.k8.openmina.com:31308/dbg1/libp2p_ipc/block/latest to follow the latencies
• Create graph representation
• Add info about the created to the aggregator response
• Add node graphql queries
• Aggregate block tracing (receive latencies + block processing time)
• Write tests for the aggregation pure functions (openmina/mina-network-debugger#23) TODO: move the issue to this repo
• Rewrite in redux