open-services-group / byon Goto Github PK

Find agreement on new additional imagestream annotations:

Is your feature request related to a problem? Please describe.
As an ODH Admin, I want to be able to create new notebook images from scratch by committing a notebook file to a repository and opening an issue assigned to a bot, so that I can run the images via Jupyter Notebook Spawner.

Describe the solution you'd like
I want to be able to create as many PRs I want before I decide to rebuild the jupyter notebook image. Then I can open a template issue assigned to a bot that triggers a new image build with my latest changes. The issue should be closed automatically but also include a link to the image that was hosted.

Acceptance Criteria

• #50
• #51
• #54

Additional context
related: open-services-group/scrum#15

We want to create user documentation that highlights what user should expect and also provides them recommendations on best practices.

Acceptance Criteria:

• Add documentation on good practices/recommendations for creating custom images.
• Add documentation for minimal requirements for image that BYON expects

Currently we blindly execute the start-singleuser.sh script within the imported container pod context. Since we have no way to know what's within the script and yet we need to execute it anyways we should try and limit the access for the pod. Let's limit network access to/from the pod.

https://kubernetes.io/docs/concepts/services-networking/network-policies/#default-deny-all-ingress-and-all-egress-traffic

Is your feature request related to a problem? Please describe.
As an ODH Admin, I want to be able to trigger a new image build by opening an issue through a template, so there is little effort on my part to trigger a build and so that the built image shows up on my Jupyter Notebook Spawner on ODH.

Describe the solution you'd like
I want to be able to create a Github issue using a template. This template doesn't need any other input from the user and the bot will know what to do with the issue. All correspondence with the bot will be done through that issue.

Acceptance Criteria

• Create a template issue that has a bot assigned
• Template could have fields to provide custom parameters to the build
• We can deploy this feature/feature-branch to os-climate

Additional context
related to: #49

Thanks @tumido!
Tested. All test cases work as expected, except these 2:

Thoth Station miminal Python 3.8   This image is expected to PASS validation   Failed
                                   false
                                   [{"severity":"error","message":"start-singleuser.sh script took too long to call 'jupyter labhub'"}]

Thoth Station miminal Python 3.8   This image is expected to PASS validation   Failed
                                   false
                                   [{"severity":"error","message":"start-singleuser.sh script took too long to call 'jupyter labhub'"}]

Originally posted by @oindrillac in #29 (review)

Is your feature request related to a problem? Please describe.
Provide tutorials on the use of BYON.

Describe the solution you'd like

• thoth-station/core#308

Describe alternatives you've considered

Additional context

Is your feature request related to a problem? Please describe.
UI relies on Pipelines states and therefore there is a need to agree on these states.

Describe the solution you'd like
Have a definition of the states required by UI and Pipelines in a doc describing what is expected and accessible at each of these states.

Describe alternatives you've considered

Additional context

Supporting JupyterHub Spawner UI scenario and the current RHODS/ODH architecture:

Based on s2i-minimal-notebook, each image should include following prerequisites:

Python packages

Based on requirements.in and Pipfile at:
https://github.com/thoth-station/s2i-minimal-notebook/tree/master/overlays/f34-python39
https://github.com/thoth-station/s2i-minimal-notebook/tree/master/overlays/python36
https://github.com/thoth-station/s2i-minimal-notebook/tree/master/overlays/python38

System packages

• Python
• Git
• Openshift Client

Configuration

• Jupyter classic UI config if notebook is installed and used
• JupyterLab configuration if jupyterlab is installed and used
• Local kubernetes/OpenShift credentials config for oc
• Entrypoint starting either jupyter lab or jupyter notebook however this might be wrapped in many nested scripts (.s2i/bin/run -> builder/run -> start-notebook.sh -> start.sh)

Omitted scenarios

I'm ignoring following scenarios and usecases, because they are either not used at all or they don't cover the base functionality we'd like to ensure in our "pre-flight check":

• Volume syncing
• Start as single user profiles for JupyterHub 1 2
• Start as webdav via Supervisord (with following config)
• Start as a gateway 1 1
• Start as a dask scheduler
• Start as a dask worker

Originally posted by @tumido in #1 (comment)

As DS/User,

I must be able to add & remove components/packages to/from an environment from a curated library directly via the UI.

Related-To: https://issues.redhat.com/browse/RHODS-2211 req 6

According to mocks name and description should be updateable

https://marvelapp.com/prototype/6dcfc3h/screen/85139857

/kind bug
/assign @tumido

Is your feature request related to a problem? Please describe.
Need to check for vulnerabilities and possible security problems in the image.

Describe the solution you'd like

• Introduce advanced validation steps that perform security scanning of imported images to identify vulnerabilities, and offer recommendations.

Describe alternatives you've considered

Additional context

As per the mocks we're currently missing a value to display in the "User" column

https://marvelapp.com/prototype/6dcfc3h/screen/85139859

Populate this annotation as per #18 (comment) spec in the dashboard backend and pipelines definition.

Is your feature request related to a problem? Please describe.
As the bot, I want to respond to an issue I am assigned to and build the repo into a custom notebook image.

Describe the solution you'd like
I want the bot to automatically start building the image and hosting it.

Acceptance Criteria

• That issue assigned to the bot triggers a new custom notebook image build
• Bot handles closing the issue if no issues arise, otherwise it uses the issue as a platform to inform the user of errors and possible avenues to correct the errors.
• Reuse BYON verification, aicoe-ci pipelines

Additional context
part of epic: #49

Discovered packages needs an additional field to support visibility toggle.

Proposed schema:

[
    {"name": "ansible", "version": "5.5.0", "visible": true},
    {"name": "appdirs", "version": "1.4.4", "visible": false}
]

As Operator/Admin,

I must be able to export custom images & store them on a local system for backup purposes

Related-To: https://issues.redhat.com/browse/RHODS-2211 req 7

As Data Scientist,

I must be able to make custom images private (only accessible by the user) or public (accessible to all cluster users).

Related-To: https://issues.redhat.com/browse/RHODS-2211 req 4

Is your feature request related to a problem? Please describe.
Define new ADR that “bot” is a GitHub App

Describe the solution you'd like
I want to create an ADR that defines what a bot is in the context of this EPIC. We are using a Github app as a bases of the bot, which will also use probot most likely.

Describe alternatives you've considered
Other bot considerations are github actions.

Additional context
related: open-services-group/scrum#15

Investigate how to manage imagestream updates. This issue is looking for answers/analysis on:

• If an image in external registry is update, how is this change propagated to Imagestream in Openshift
  • Scheduled import via importPolicy.scheduled on Imagestream
  • Investigate what happened with referencePolicy in OCP 4.x, since it's missing from the docs, but present in the resource manifests (OCP 3.x docs here)
• How/if we can trigger a validation pipeline on image update
  • Investigate if Openshift can trigger pipelinerun from native image trigger
  • Investigate if there are Tekton Triggers for this
• Also Image pull policy plays their role here https://kubernetes.io/docs/concepts/containers/images/#image-pull-policy

Based on my initial screening it seems there's no way to gate when image stream tag import if set to scheduled. Also it's unsure when the cache is cleaned or accessed (is this per node?) and how an image is cached. Investigate what happens when pods are started: if the image is pulled from a remote repository or mirrored to internal container registry on the first start and pulled from there. What happens when pod is scheduled to a different node (again pulling from internal/external registry?) Experiments needed. It effects when we get the updated image and if we know about it or user is served a newer version withouth admin/validation pipeline noticing first.

Besides scheduled import described above, there's no common reactive facility to initiate/trigger notification from remote registries - webhooks etc, this is remote repository specific and must be set on the remote repository side.

As an ODH Admin,
I want to schedule the BYON Pipeline Runs using Pipeline as a Service
so that I can control the scheduling/location of the PipelineRuns

As an ODH Admin,
I want to point my Pipeline as a Service to a specific Git Tag on a specific repository on GitHub,
so that I can control the version of BYON pipelines used.

As an OSG release-engineer,
I want to git-tag a specific commit in a repository on GitHub,
so that I can maintain releases of the BYON Pipelines.

Acceptance Criteria

• access to Red Hat shared KCP instance is granted to op1st-ops/sre
• smaug and os-climate clusters are available to all op1st-ops in KCP
• Pipeline is schedulable via KCP on smaug
• Pipeline Run logs are available from KCP

References

• https://coreos.slack.com/archives/C020QAF9D46/p1652102657293539

Is your feature request related to a problem? Please describe.
As BYON developer,

I want to access BYON + UI directly from Open Data Hub.

Describe the solution you'd like

• Have a deployment of ODH with BYON and UI to be tested.

Describe alternatives you've considered

Additional context
https://issues.redhat.com/browse/ODH-591

During discussions with Jeff about #42 one more feature surfaced -> It would be nice to show R/RStudio version along with Python versions. Maybe other software will come later, let's start with these two and expand the list if needed.

/assign @tumido

Is your feature request related to a problem? Please describe.
I want:

• A stage environment deploying BYON on top of ODH with JupyterHub deployment
• A clean, easy and reproducible procedure how to stand up a dev environment that deploys ODH operator + JupyterHub + BYON for dev purposes

Describe the solution you'd like
For stage environment provide a namespace or cluster in Operate First with a BYON deployment from thoth-station/helm-charts master
For dev environment provide a guide/steps and a kustomize to apply to get a dev environment up and running in no time. A dev file for code ready containers maybe?

Describe alternatives you've considered
n/a

Additional context
n/a

It's valuable to show software included in the image along with packages.

Updates needed to UI Mockup and the changes to be made in the following places to show included software:

• Admin UI - Software Version also shown in Admin UI under Notebook Image settings along with Packages.
• Spawner UI - Show Python version under question mark in spawner UI, along with Software included. Show description under Image Name

Originally posted by @oindrillac in #43 (comment)

As a Data Scientist,

I want to upload my (newly found on the internet) Jupyter Notebook to a web service hosted on my ODH deployment.

I want to address the notebook via a URL, so that this service turns it into a notebook ready to use, and so that I don’t have to install any more modules, and so that each cell of the notebook is runnable.

Related-To: https://issues.redhat.com/browse/RHODS-2211 req 1

As Platform Operator,

I want to install BYON Pipelines from Operator Hub.

Ref: https://labs.consol.de/development/2021/05/03/helm_based_operator.html

Is your feature request related to a problem? Please describe.
As an ODH Admin, I want to check if my repository’s dependencies, security issues, and compatibility are meeting my standards, so I don’t create an insecure image.

Describe the solution you'd like
The bot will run an advise report and comment back important information about the score of the advise report. If there are big problems with the advise run, the bot will hold the build until either a change is made or the user forces the bot to build with a command.

score : The score can be a combination of metrics obtained through the advise report such as: scorecard info, justifications, recommend actions. it could just be a list of justifications
Acceptance Criteria

• Issue template triggers a bot to run a Thoth Advise on the repository before an image is built
• If issues or vulnerabilities exist in the Thoth Advise report, then the bot comments a link to further details on the issues
• If issues or vulnerabilities exist in the Thoth Advise report, users must further allow the bot to make and deploy the image via a command. ex) /approve

Additional context
related: open-services-group/scrum#15

Kubernetes supports native filters by labels when listing resources. In order to simplify parsing the relevant imagestreams and pipelineruns in ODH dashboard backend, let's enrich the pipeline resource and generated imagestream with additional label app.kubernetes.io/created-by: byon

As Platform Operator,

I want to install a tool via Helm charts that allow builds of images that can be used for ML projects in ODH.

Reference: https://github.com/thoth-station/helm-charts

As BYON User,

I want to have a Grafana dashboard template (.json) that I can use to monitor BYON pipelines.

As Operator/Admin,

I have existing images I want to import in ODH. The system must enable users to bring in/import notebook images via the UI (eg. users should not have to edit config files such as YAML files). After import, the images should be available for selection in the UI like out-of-the-box images.

Note: by default, imported images would be available for all cluster users.

Related-To: https://issues.redhat.com/browse/RHODS-2211 req 1

As Data Scientist,

I want to create a new notebook image from scratch by defining the packages/libraries/components needed.

Related-To: https://issues.redhat.com/browse/RHODS-2211 req 2

As DS/User,

I want to have a system with some capability to check dependencies and verify any dependency issues as they are defining the components in a notebook or a new custom image. [OPTIONAL TOOL]

Notes:

• User adds a TensorFlow version and needs to identify dependencies (specific packages and versions needed)
• User adds a TensorFlow version and adds an incompatible version of a dependency (eg. the wrong version of scipy to work with the added TensorFlow version). The user should be able to access a tool in the UI to identify the issue and make corrections.

As Platform Operator,

I want to have a way to configure the deployment of BYON pipelines.

As developers planning to contribute on the byon UI development efforts, we need to determine the minimal requirements, setup, and environment required to start contributing.

UI design (working document) - https://marvelapp.com/prototype/6dcfc3h/screen/85139864

Run a set of tests to verify the import and validation pipeline for edge cases.

Acceptance Criteria:

Come up with a test images to test for edge cases.

• ODH notebooks and invalid edge case notebooks.
• R and Opta planer images (@guimou)
• OS Climate images (@oindrillac)

Test images can be included in https://github.com/open-services-group/byon/blob/main/Makefile

Change the message if the first content check fails - If python is not present it is not an error but rather a warning. Skip further validation in such cases, because the image may be R or something else totally valid.

Change the Dashboard frontend/backend to support toggling the visibility on failed images when only the messages with warning severity are present. Disable the visibility toggle if there are error messages.

This working group extends across multiple teams and we would like to add external collaborators to this repo and project (such as ODH team members). By default this is not possible at the moment, so we propose to :-

1. Add them as members of the org
2. Put them on a wg-byon-collaborators team
3. Add team with write perms to project

Any inputs?

cc: @goern

As Data Scientist/User,

I must be able to start with an existing base notebook image, make changes by adding/removing/updating components and save the resulting image. The resulting image should be available for selection like other notebook images.

As more byon custom notebook images get imported, the Jupyterhub Spawner UI can get quite cluttered. To reduce clutter, add a dropdown and search functionality in Spawner.

As Operator/Admin,

I have added images from the URL and I want to take some actions on these images directly from the UI:

• Ability to rename images (name displayed in “spawner” UI).
• Ability to delete images
• Ability to disable/enable images. Disabled images would still be in the list in the admin view, but would not be visible for users in the spawner page. Admins could enable disabled images.

Is your feature request related to a problem? Please describe.
As an ODH Admin, I want the closing of my template issue to signify that my notebook image is available on my ODH deployment via the Jupyter Notebook Spawner.

Acceptance Criteria

• The bot should give updates on the overall status of the build and deployment.
• The template issue should not close until the image is available in the spawner

Additional context
part of epic: #49