open-services-group / byon Goto Github PK
View Code? Open in Web Editor NEWBring Your Own Notebook (BYON) project repository.
License: GNU General Public License v3.0
Bring Your Own Notebook (BYON) project repository.
License: GNU General Public License v3.0
Find agreement on new additional imagestream annotations:
Status | Annotation key | Values | Description |
---|---|---|---|
Preexisting | opendatahub.io/notebook-image-desc |
string | User facing image description |
Proposed new | opendatahub.io/notebook-image-messages |
string encoded JSON | Validation output (errors and recommendations) |
Preexisting | opendatahub.io/notebook-image-name |
string | User facing image name |
Proposed new, questionable | opendatahub.io/notebook-image-origin |
string | Defines if the image is part of ODH distribution, admin imported or user created. Proposed values Default , Admin and Custom |
Proposed new | opendatahub.io/notebook-image-phase |
string | State of validation process: Validating , Success , Failed |
Preexisting | opendatahub.io/notebook-image-url |
url | Container image url e.g. quay.io/org/image:tag |
opendatahub.io/notebook-image-visible |
"true" or "false" opendatahub.io/notebook-image label, see opendatahub-io-contrib/jupyterhub-singleuser-profiles#217 |
||
Proposed new | opendatahub.io/notebook-image-creator |
string | Kubernetes user name of the person initiating the import |
Is your feature request related to a problem? Please describe.
As an ODH Admin, I want to be able to create new notebook images from scratch by committing a notebook file to a repository and opening an issue assigned to a bot, so that I can run the images via Jupyter Notebook Spawner.
Describe the solution you'd like
I want to be able to create as many PRs I want before I decide to rebuild the jupyter notebook image. Then I can open a template issue assigned to a bot that triggers a new image build with my latest changes. The issue should be closed automatically but also include a link to the image that was hosted.
Acceptance Criteria
Additional context
related: open-services-group/scrum#15
We want to create user documentation that highlights what user should expect and also provides them recommendations on best practices.
Acceptance Criteria:
Currently we blindly execute the start-singleuser.sh
script within the imported container pod context. Since we have no way to know what's within the script and yet we need to execute it anyways we should try and limit the access for the pod. Let's limit network access to/from the pod.
Is your feature request related to a problem? Please describe.
As an ODH Admin, I want to be able to trigger a new image build by opening an issue through a template, so there is little effort on my part to trigger a build and so that the built image shows up on my Jupyter Notebook Spawner on ODH.
Describe the solution you'd like
I want to be able to create a Github issue using a template. This template doesn't need any other input from the user and the bot will know what to do with the issue. All correspondence with the bot will be done through that issue.
Acceptance Criteria
Additional context
related to: #49
Thanks @tumido!
Tested. All test cases work as expected, except these 2:
Thoth Station miminal Python 3.8 This image is expected to PASS validation Failed
false
[{"severity":"error","message":"start-singleuser.sh script took too long to call 'jupyter labhub'"}]
Thoth Station miminal Python 3.8 This image is expected to PASS validation Failed
false
[{"severity":"error","message":"start-singleuser.sh script took too long to call 'jupyter labhub'"}]
Originally posted by @oindrillac in #29 (review)
Is your feature request related to a problem? Please describe.
Provide tutorials on the use of BYON.
Describe the solution you'd like
Describe alternatives you've considered
Additional context
Is your feature request related to a problem? Please describe.
UI relies on Pipelines states and therefore there is a need to agree on these states.
Describe the solution you'd like
Have a definition of the states required by UI and Pipelines in a doc describing what is expected and accessible at each of these states.
Describe alternatives you've considered
Additional context
Supporting JupyterHub Spawner UI scenario and the current RHODS/ODH architecture:
Based on s2i-minimal-notebook, each image should include following prerequisites:
Based on requirements.in
and Pipfile
at:
https://github.com/thoth-station/s2i-minimal-notebook/tree/master/overlays/f34-python39
https://github.com/thoth-station/s2i-minimal-notebook/tree/master/overlays/python36
https://github.com/thoth-station/s2i-minimal-notebook/tree/master/overlays/python38
Package name | Note |
---|---|
notebook |
Provides the classic notebook interface. Either this or jupyterlab has to be present |
jupyterhub |
Provides Spawner. Also transitive dependency from jupyterhub-singleuser-profiles which is referenced later but not installed as a package (weird) |
jupyterlab |
Provides the JupyterLab UI interface. Either this or jupyterhub has to be present |
jupyterlab-requirements |
Requires jupyterlab , it is NOT present in Python 3.6 overlay |
jupyter_kernel_gateway |
This was probably never used. |
jupyter-nbrequirements |
jupyterlab .notebook . Not maintained. |
supervisor |
This was probably used only when WEBDAV is enabled, which I never saw... |
jupyterlab-git |
Extension enabling git. Requires git distribution package. |
jupyterlab-spellchecker |
Extension. This is NOT present in Python 3.6 overlay |
notebook
is installed and usedjupyterlab
is installed and usedoc
jupyter lab
or jupyter notebook
however this might be wrapped in many nested scripts (.s2i/bin/run
-> builder/run
-> start-notebook.sh
-> start.sh
)I'm ignoring following scenarios and usecases, because they are either not used at all or they don't cover the base functionality we'd like to ensure in our "pre-flight check":
Originally posted by @tumido in #1 (comment)
As DS/User,
I must be able to add & remove components/packages to/from an environment from a curated library directly via the UI.
Related-To: https://issues.redhat.com/browse/RHODS-2211 req 6
According to mocks name and description should be updateable
https://marvelapp.com/prototype/6dcfc3h/screen/85139857
/kind bug
/assign @tumido
Is your feature request related to a problem? Please describe.
Need to check for vulnerabilities and possible security problems in the image.
Describe the solution you'd like
Describe alternatives you've considered
Additional context
As per the mocks we're currently missing a value to display in the "User" column
https://marvelapp.com/prototype/6dcfc3h/screen/85139859
Populate this annotation as per #18 (comment) spec in the dashboard backend and pipelines definition.
Is your feature request related to a problem? Please describe.
As the bot, I want to respond to an issue I am assigned to and build the repo into a custom notebook image.
Describe the solution you'd like
I want the bot to automatically start building the image and hosting it.
Acceptance Criteria
Additional context
part of epic: #49
Discovered packages needs an additional field to support visibility toggle.
Proposed schema:
[
{"name": "ansible", "version": "5.5.0", "visible": true},
{"name": "appdirs", "version": "1.4.4", "visible": false}
]
As Operator/Admin,
I must be able to export custom images & store them on a local system for backup purposes
Related-To: https://issues.redhat.com/browse/RHODS-2211 req 7
As Data Scientist,
I must be able to make custom images private (only accessible by the user) or public (accessible to all cluster users).
Related-To: https://issues.redhat.com/browse/RHODS-2211 req 4
Is your feature request related to a problem? Please describe.
Define new ADR that “bot” is a GitHub App
Describe the solution you'd like
I want to create an ADR that defines what a bot is in the context of this EPIC. We are using a Github app as a bases of the bot, which will also use probot most likely.
Describe alternatives you've considered
Other bot considerations are github actions.
Additional context
related: open-services-group/scrum#15
Investigate how to manage imagestream updates. This issue is looking for answers/analysis on:
Imagestream
in Openshift
importPolicy.scheduled
on Imagestream
referencePolicy
in OCP 4.x, since it's missing from the docs, but present in the resource manifests (OCP 3.x docs here)Based on my initial screening it seems there's no way to gate when image stream tag import if set to scheduled. Also it's unsure when the cache is cleaned or accessed (is this per node?) and how an image is cached. Investigate what happens when pods are started: if the image is pulled from a remote repository or mirrored to internal container registry on the first start and pulled from there. What happens when pod is scheduled to a different node (again pulling from internal/external registry?) Experiments needed. It effects when we get the updated image and if we know about it or user is served a newer version withouth admin/validation pipeline noticing first.
Besides scheduled import described above, there's no common reactive facility to initiate/trigger notification from remote registries - webhooks etc, this is remote repository specific and must be set on the remote repository side.
As an ODH Admin,
I want to schedule the BYON Pipeline Runs using Pipeline as a Service
so that I can control the scheduling/location of the PipelineRuns
As an ODH Admin,
I want to point my Pipeline as a Service to a specific Git Tag on a specific repository on GitHub,
so that I can control the version of BYON pipelines used.
As an OSG release-engineer,
I want to git-tag a specific commit in a repository on GitHub,
so that I can maintain releases of the BYON Pipelines.
Acceptance Criteria
References
Is your feature request related to a problem? Please describe.
As BYON developer,
I want to access BYON + UI directly from Open Data Hub.
Describe the solution you'd like
Describe alternatives you've considered
Additional context
https://issues.redhat.com/browse/ODH-591
Is your feature request related to a problem? Please describe.
I want:
Describe the solution you'd like
For stage environment provide a namespace or cluster in Operate First with a BYON deployment from thoth-station/helm-charts master
For dev environment provide a guide/steps and a kustomize to apply to get a dev environment up and running in no time. A dev file for code ready containers maybe?
Describe alternatives you've considered
n/a
Additional context
n/a
It's valuable to show software included in the image along with packages.
Updates needed to UI Mockup and the changes to be made in the following places to show included software:
Originally posted by @oindrillac in #43 (comment)
As a Data Scientist,
I want to upload my (newly found on the internet) Jupyter Notebook to a web service hosted on my ODH deployment.
I want to address the notebook via a URL, so that this service turns it into a notebook ready to use, and so that I don’t have to install any more modules, and so that each cell of the notebook is runnable.
Related-To: https://issues.redhat.com/browse/RHODS-2211 req 1
As Platform Operator,
I want to install BYON Pipelines from Operator Hub.
Ref: https://labs.consol.de/development/2021/05/03/helm_based_operator.html
Is your feature request related to a problem? Please describe.
As an ODH Admin, I want to check if my repository’s dependencies, security issues, and compatibility are meeting my standards, so I don’t create an insecure image.
Describe the solution you'd like
The bot will run an advise report and comment back important information about the score of the advise report. If there are big problems with the advise run, the bot will hold the build until either a change is made or the user forces the bot to build with a command.
score : The score can be a combination of metrics obtained through the advise report such as: scorecard info, justifications, recommend actions. it could just be a list of justifications
Acceptance Criteria
Additional context
related: open-services-group/scrum#15
Kubernetes supports native filters by labels when listing resources. In order to simplify parsing the relevant imagestreams and pipelineruns in ODH dashboard backend, let's enrich the pipeline resource and generated imagestream with additional label app.kubernetes.io/created-by: byon
As Platform Operator,
I want to install a tool via Helm charts that allow builds of images that can be used for ML projects in ODH.
Reference: https://github.com/thoth-station/helm-charts
As BYON User,
I want to have a Grafana dashboard template (.json) that I can use to monitor BYON pipelines.
As Operator/Admin,
I have existing images I want to import in ODH. The system must enable users to bring in/import notebook images via the UI (eg. users should not have to edit config files such as YAML files). After import, the images should be available for selection in the UI like out-of-the-box images.
Note: by default, imported images would be available for all cluster users.
Related-To: https://issues.redhat.com/browse/RHODS-2211 req 1
As Data Scientist,
I want to create a new notebook image from scratch by defining the packages/libraries/components needed.
Related-To: https://issues.redhat.com/browse/RHODS-2211 req 2
As DS/User,
I want to have a system with some capability to check dependencies and verify any dependency issues as they are defining the components in a notebook or a new custom image. [OPTIONAL TOOL]
Notes:
As Platform Operator,
I want to have a way to configure the deployment of BYON pipelines.
As developers planning to contribute on the byon UI development efforts, we need to determine the minimal requirements, setup, and environment required to start contributing.
UI design (working document) - https://marvelapp.com/prototype/6dcfc3h/screen/85139864
Run a set of tests to verify the import and validation pipeline for edge cases.
Acceptance Criteria:
Come up with a test images to test for edge cases.
Test images can be included in https://github.com/open-services-group/byon/blob/main/Makefile
Change the message if the first content check fails - If python is not present it is not an error
but rather a warning
. Skip further validation in such cases, because the image may be R or something else totally valid.
Change the Dashboard frontend/backend to support toggling the visibility on failed images when only the messages with warning
severity are present. Disable the visibility toggle if there are error
messages.
This working group extends across multiple teams and we would like to add external collaborators to this repo and project (such as ODH team members). By default this is not possible at the moment, so we propose to :-
wg-byon-collaborators
teamAny inputs?
cc: @goern
As Data Scientist/User,
I must be able to start with an existing base notebook image, make changes by adding/removing/updating components and save the resulting image. The resulting image should be available for selection like other notebook images.
As more byon custom notebook images get imported, the Jupyterhub Spawner UI can get quite cluttered. To reduce clutter, add a dropdown and search functionality in Spawner.
As Operator/Admin,
I have added images from the URL and I want to take some actions on these images directly from the UI:
Is your feature request related to a problem? Please describe.
As an ODH Admin, I want the closing of my template issue to signify that my notebook image is available on my ODH deployment via the Jupyter Notebook Spawner.
Acceptance Criteria
Additional context
part of epic: #49
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.