Also, pin version in Dockerfile and .gitlab-ci.yaml

Related: SwissDataScienceCenter/renku#400

Description

We should allow the user to specify default server_options, e.g. default_url. These runtime parameters should go in a config.ini file in the project root directory.

Examples of this include:

• Environment resource requirements (e.g., an interactive environment for this project requires 2 GPUs and 16G of memory)
• The URL to connect to (e.g., jupyterlab or rstudio)

Remove user switches:

We should start to enable projects with different default environments. Whether this requires a full separate template or just different templates for the Dockerfile is yet to be determined.

We should investigate if using cookiecutter for our templates is reasonable, and if so, transition all templates to use their syntax/formatting.

An example can be seen in https://github.com/SwissDataScienceCenter/renku-project-template/pull/97/files#diff-d865b06b497efc8c56c63e5bedfc86f6da6a005cdb7d30e3702286f3d918aaa6
Set allow_template_update: truefor a template if updating files from the template in a users project doesn't break anything (only files not touched by a user get updated) Setimmutable_template_files` to a list of files in the template that a user should not modify (as modifying them causes them not to be updated which could lead to discrepancies when updating).

Do this for all our templates, deciding for each which values are appropriate.

renku-python has switched to using dev versions of the scheme X.Y.Z.devN.gGITSHA, with GITSHA being the sha of the git commit the version was made from. Regular release versions are still just X.Y.Z

In the acceptance tests, we already changed to checking out the commit and installing from source if the version contains a git sha.

Our images should be updated accordingly, e.g. this line.

In SwissDataScienceCenter/renkulab-docker#111 we propose to decouple the renku-python versioning from the renkulab base images. Since renku-python is easy to install and update, we should have an environment variable passed to the CI image build to change the version easily while keeping the underlying runtime image the same.

The default docker build fails with

SpecNotFound: /home/jovyan/test.yml is not a valid yaml file.

originally reported by @cchoirat

Currently the description is not handled at all, although we ask for it in our base templates. We should remove it from the templates until this functionality is properly implemented in SwissDataScienceCenter/renku-python#2218 and related issues.

If a project has spaces in the name, the kernel creation fails -- need to escape spaces (or perhaps also other characters?)

According to me, Renku is suffering from the problem that it needs on one hand an exact requirements.txt file with pinned versions, but on the other hand user friendly requirements.txt/solution.

This problem has been solved by pipenv. So I wonder if this should not become a default in renku. From an interface point of view, the user will not have to care about requirements.txt anymore which is a step toward a more friendly experience.

I understand, they might be other consideration at play here. I just wanted to share my opinion.

Would be it possible that the Empty project is the most skinny version of Renku project possible. What I mean is that if I'd like to use Renku neither for Python nor R (I used it for Bash), I would still expect that when I create an empty project through the Renku UI, I'll get a Renku project where I can use Renku commands like: renku run ...., renku dataset create ... and so on. What I'd imagine the project would have is the most basic Renku Dockerfile, .gitlab-ci.yml, .dockerignore, .gitignore and probably .renku folder. Perhaps, the Empty project could be even named in a bit more descriptive way e.g. Empty Renku project but even if we keep the name as it is now, I believe it won't be that much misleading.

Description

When a template is instantiated, the metadata should include a link to the exact template (including version) from which it came.

Description

A project template should include a readme that describes the features of the template and how to change configuration options after instantiation. This readme should become the new project's readme similar to create-react-app.

A suggestion was made to create a src/python directory with the necessary things inside to make pip install src/python/<project-name> work.

This is needed in order to make it easy for users to refactor early.

In SwissDataScienceCenter/renkulab-docker#77 we changed the docker image names - we need to adapt the templates to reflect this.

The latest project templates should be updated to 0.7.6, e.g.

Several R mirrors are out of date with 0.7.5, see SwissDataScienceCenter/renkulab-docker#163

Ideally, at the time of project creation, the version of renku-python installed in a project's docker image would be the same as the version of the client that was used to create the project. This is now easily done by overriding the renku-python version in the Dockerfile. We should make this version templateable so the client that is used to initialize the project can inject the version there.

Use the default jinja filter to make sure that the template stays compatible with older versions of renku-python: https://jinja.palletsprojects.com/en/2.11.x/templates/#default

Description

It should be possible to store project-level configuration. Examples of this include:

• Environment resource requirements (e.g., an interactive environment for this project requires 2 GPUs and 16G of memory)
• The URL to connect to (e.g., jupyterlab or rstudio)

SwissDataScienceCenter/renku-notebooks#304 is proposing to let projects use a default image in cases like courses and workshops where no changes are expected to occur for the docker image.

In this case, we could include a template that doesn't have a Dockerfile and .gitlab-ci.yml and explains in the README.md file how use this template.

This would probably be adapted by the course/workshop instructor to be forked by the participants.

With the updates to JupyterLab 3.0, the default Python version has gone to 3.9.

A few of the Docker image tags are therefore out of date:

E.g. on a fresh v0.8.0 project:

Related to SwissDataScienceCenter/renkulab-docker#177

The problem

When building a docker image, default behavior is to ship the repository, including data and all, as part of the docker context.

Desired solution

Change default behavior. Provide a .dockerignore file, build from a subdirectory, or take advantage of buildkit.

Alternatives

Point to best practices for building docker images, and leave it to users to implement them. This may not be ideal for non-experts.

Description

SwissDataScienceCenter/renkulab-docker#115 introduces a new image based on the current stable version of Julia (1.5.3). The Julia template should be updated to be in sync.

Currently, variables are dictionaries where the key is the variable name and the values represent its description.
We should use an object instead so that we can specify other properties like type, default, and support enumerations.

Instead of summary: short summary added at the beginning of the readme file, we should have something like this:

  - name: summary
    description: short summary added at the beginning of the readme file
    type: string

For enumerations, we should have something like the following:

  - name: plugins
    description: list of plugins to install in the template
    type: enum
    enum: ["none", "VScode", "GIS", "VScode and GIS"]
    default: none

Initial support for strings and enumerations may be enough.
We could also support numbers and booleans for completeness and to have a better validation system, as well as a better UX in the UI.

Collaborators have asked for this base vnc image to be included within the usual Renku Project Stack of ease of access.

The VNC template has matured over the last few months and is now used by many of our imaging-related counterparts. It will also be used in other contexts down the line (GIS etc.). It has been extensively tested and seems on all accounts to be stable, with desirable features like a clipboard, fullscreen and renku branding.

We should integrate this into the Renku project stack.

The standard templates should include python and R templates to choose from.

See for example https://renkulab.io/projects/ableuler/testrr/

The number of subdirectories for each released version might be huge. It is also a bit more difficult to apply and propagate patches from older templates.

Changes to master in this repo could automatically open a PR in the renku-ui repo to change the corresponding line in the values.yaml file of the UI chart. This would ensure that released versions of the ui chart use the up-to-date version of that repo. What do you think @SwissDataScienceCenter/renku-ui-maintainers?

We currently add the image-build tag to the image-build job in the gitlab ci file. At the same time we recommend admins to tag the runners when deploying runners for Renku. We should change the default behaviour to not using tags.

Users don't know where to find latest image.

There's docs in the works, but this is an obvious place to put a link.

Description

At the moment the CI/CD in the projects is configured in such a way that the docker image is built for every single commit. This is not only unnecessary most of the times but it also slows down the time a user has to wait to start an environment, even for the simplest of changes. On the other hand, gitlab-runner resources are occupied.

Proposal

A docker image should be built by default when only selected files are modified: Dockerfile, requirements.txt, environment.yaml.

In the context of the VNC and other targeted use-cases, it would be nice to templatise/modularise the various options, for example like:

To do:

• scope out necessary functionality used in jinja
• see what changes in renku-python are needed, if any
• put together a PoC

We should be able to parametrize the version of underlying R/python - we should allow for whatever options are supported by our base image build matrix.

• update base renku/singleuser image tag
• make wording consistent

Define a folder structure for templates. The initial proposal is that templates are stored in git repositories. In the top level of the repository is a folder per template. E.g., python-basic, python-datascience, r, r-tidyverse, julia, etc.

Inside each folder is a file that contains template metadata (e.g., display name), and the files that need to be rendered to realize the template.

Insert here:

Include template's available variables in the manifest.yaml with a short description.

E.G.

- folder: python-cuda
variables:
  - description: a short description going in the Readme file
  - cuda-version: specify cuda version (only 9.x or 10.x)

For batch execution described in SwissDataScienceCenter/renku/issues/1929, we suggest to create a new template.

Compare to the original template, this one:

• w/ Dockerfile.batch - based on a batch version image SwissDataScienceCenter/renkulab-docker/issues/172
• w/o environment.yml - not to use conda for dependencies
• extra job in .gitlab-ci.yml to build and push the batch image automatically

Such template can be useful for batch execution with commands proposed in SwissDataScienceCenter/renku-python/issues/2213

Dockerfile.batch, for example:

ARG RENKU_BASE_IMAGE=renku/renkulab-batch
FROM ${RENKU_BASE_IMAGE}

# Uncomment and adapt if code is to be included in the image
# COPY src /code/src

# Uncomment and adapt if your R or python packages require extra linux (ubuntu) software
# e.g. the following installs apt-utils and vim; each pkg on its own line, all lines
# except for the last end with backslash '\' to continue the RUN line
#
# USER root
# RUN apt-get update && \
#    apt-get install -y --no-install-recommends \
#    apt-utils \
#    vim
# USER ${NB_USER}

# install the python dependencies
COPY requirements.txt /tmp/
RUN pip install -r /tmp/requirements.txt

# RENKU_VERSION determines the version of the renku CLI
# that will be used in this image. To find the latest version,
# visit https://pypi.org/project/renku/#history.
ARG RENKU_VERSION=0.15.1

########################################################
# Do not edit this section and do not add anything below

RUN if [ -n "$RENKU_VERSION" ] ; then \
    pipx uninstall renku && \
    pipx install --force renku==${RENKU_VERSION} \
    ; fi

########################################################

Extra job in .gitlab-ci.yml, for example:

batch_image_build:
  stage: build
  image: docker:stable
  before_script:
    - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN http://$CI_REGISTRY
  script: |
    CI_COMMIT_SHA_7=$(echo $CI_COMMIT_SHA | cut -c1-7)
    docker build --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA_7-batch -f Dockerfile.batch .
    docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA_7-batch

When creating a new project on the UI, data and notebooks are not created

Description

The .gitlab-ci.yml file pulls the base image to make it possible to transparently distribute bug fixes by using the same tag to reference a corrected version of the base image, but this breaks the expectation that images not be updated beneath the user's feet.

The line that pulls the base image should be removed, and we should have a better solution for notifying the users about updates to the base image. (See SwissDataScienceCenter/renku#632)

There's no widely accepted way to include dependencies in R projects, but one way is to include an install.R file (@cchoirat suggestion) executed by R. We should include this file & lines for installing stuff from the file in the minimal R template.

file to add:

install.R

lines for docker:

COPY install.R /tmp/install.R
RUN R -f /tmp/install.R

renku-python is adding functionality to optionally keep certain filepaths (including wildcard file matching) out of LFS (see: SwissDataScienceCenter/renku-python#1210)

This is useful for when you have code files in a dataset, or want to diff data files that you know are small, that are output from a renku run.

We should add a template that includes *.ipynb for starters, which is currently hardcoded functionality.

At the moment when I open a terminal window in an Interactive Env the prompt looks like this:

jovyan@jupyter-jakub-2echrobasik1-kubas-2ddatascience-dbash-2d335595ca:/work/kubas-datascience-in-bash$ █

Could we shorten/simplify that to smth as simple as
renku%> █
or
kubas-datascience-in-bash%> █ ?

Previously we had a workaround for creating the .Rproj file in the entrypoints for the R dockerfile, but we decided it should go into the template. However, until now, file names could not be templated. Now that we have SwissDataScienceCenter/renku-python#1271, we can include the templated name.

renku.ini file in all templates contains a [renku "interactive"] section that is not supported by renku-python anymore. It should be renamed to [interactive].

When you create a new project and start an interactive environment it would be nice to have git bash-completion pre-installed already in the image so you can use tab completion for all the git commands. I successfully installed them by adding RUN sudo apt-get install git-core bash-completion --assume-yes to my project's Dockerfile.