ml-tooling / ml-workspace Goto Github PK

Hi, I would like to request update OpenCV to 4.4, this version support GPU

Hi,

According to the README, in the Authentication section: "You can also use to let Jupyter generate a random token that is printed out on the container logs."

I am trying to execute:

docker run -p 8080:8080 --env AUTHENTICATE_VIA_JUPYTER="<generated>" mltooling/ml-workspace:latest

but I cannot find any token in the logs produced by the command docker logs.

Where is that generated token logged?

Thanks!

It would be great if ml-workspace can be easily deployed on Kubernetes through helm or yaml scripts, and can expose metrics to promethues and grafana.

Describe the bug:

• Directly adding a environment.yml package to my code directory does
  not automatically install the correct dependencies.

Expected behaviour:

• I install various dependencies including pip dependent packages to
  conda environments as part of my machine learning workflows. Based
  on my uncertain understanding of “Installing Dependencies”
  instructions, I expect that if I add a conda environment.yml to the
  directory from which I run the docker container the docker container
  will use the environment.yml file to initialize the base conda
  environment.

Steps to reproduce the issue:

1. Use conda to generate a environment.yml file from an existing conda
   environment with additional pip installed packages.
2. Add this file to a specific home directory in which you want to
   work: /user/home/folder_to_use
3. cd into this directory and run the following start command:
   sudo docker run -d -p 8080:8080 --gpus all --name "ml-workspace" -v "${PWD}:/workspace" --env AUTHENTICATE_VIA_JUPYTER="mytoken" --shm-size 512m --restart=no mltooling/ml-workspace-gpu:latest
4. Navigate to the virtual desktop and start VSCode. The python
   environment does not appear in the options menu.

Technical details:

• Workspace version: 0.9.1
• Docker version: 19.03.05
• Host Machine OS (Windows/Linux/Mac): Ubuntu 18.04
• Command used to start the workspace:
  sudo docker run -d -p 8080:8080 --gpus all --name "ml-workspace" -v "${PWD}:/workspace" --env AUTHENTICATE_VIA_JUPYTER="mytoken" --shm-size 512m --restart=no mltooling/ml-workspace-gpu:latest
• Browser (Chrome/Firefox/Safari): Firefox

Possible Fix:

• Rebuild the docker file with the necessary packages and channels
  hardcoded in.

Additional context:

• I fully acknowledge that I am most likely missing something. What is
  the proper way to install various dependencies to the workspace
  besides editing the dockerfile?

MLWorkspace strikes me as a phenomenal tool with broad applicability. I'm wondering if you might be able to provide an example of how to fit it into a standard multi-user jupyterhub + dockerspawner, where a 'start-singleuser' command spawns an image in which the primary user is not root, but jupyter's standard jovyan, and permissions are fixed accordingly. I've been attempting this, but it's proving a bit elusive to me.

Basically, is there a way of combining jupyter's base-notebook stack with mlworkspace to create a single-user compatible image that could be spawned from jupyterhub?

(MLHub seems like a natural step in this direction, but this would be for users who are interested in integrating MLWorkspace into their existing workflows and who might be gunshy about having single users have root permissions).

My thanks in advance.

Describe the bug:
The Ungit link, from the workspace "Open Tool", is not loading the web interface for the Ungit.

Expected behaviour:
Open the web interface for Ungit

Steps to reproduce the issue:
Open the ML-Hub login to the server.
2. Create a new server with a given name
3. After the server spawn, access the server.
4. Open Tool -> Ungit

Technical details:

• Workspace version : 0.8.7
• Docker version :
• Host Machine OS (Windows/Linux/Mac): Linux
• Command used to start the workspace : ML-Hub
• Browser (Chrome/Firefox/Safari): Chrome, Firefox

Possible Fix:

Additional context:

Feature description:

DataLad is a "data management and publication multitool". Learn more by visiting http://datalad.org and even better http://handbook.datalad.org/ or listening to any introductory talk on datalad like a recent debconf 2020 or slides from a past presentation at MILA, Montreal which was a little tuned for ML use-cases.

Problem and motivation:

DataLad solves many problems of version control, scalable access to remote data resources; growing amount of data is becoming available through git-annex (and thus DataLad): e.g. via http://datasets.datalad.org/ we provide access to over 200TB of (largely) neural data, and more to come!

Why is this change important to you?

Disclaimer: I am "involved" in DataLad project ;-)

Is this something you're interested in working on?

I guess. Although it might be just provision installation of DataLad within the ML workspace. We provide Debian packages (in stock Debian and backports from http://neuro.debian.net), it is available from PyPI and also from conda-forge. If ML is friendly to docker or singularity containers -- please also install datalad-container extension.

DataLad comes with command line and Python APIs. So it should work nicely for Jupyter notebooks users to access the data they want or to establish version control of their results, or even reproducible (re)execution via datalad run

when we use vscode(http://localhost:8080/tools/vscode/), there is no reaction if we debug. But using vscode in VNC, it's ok.

Describe the bug:

ML Workspace fails to start and serve up Jupyter Notebooks

Started ML WORKSPACE with the Following Docker Command:

$ docker run -d \
    -p 8080:8080 \
    --name "ml-workspace" \
    -v "/Users/<myuserid>/Documents/Notebooks:/workspace" \
    --env AUTHENTICATE_VIA_JUPYTER="mytoken" \
    --shm-size 512m \
    --restart always \
    mltooling/ml-workspace:latest
57fc561fa9eb1c38261e9c6a09a66a1a4c2ca9e686fbb1991996cb4befa1f8a6
$

It launched and reported the container id

Expected behaviour:

When I go to http://localhost:8080 I expect to get to my Jupyter Notebooks. Instead I get a waiting forever display and the logs as you see below.

Steps to reproduce the issue:

Run the docker command as described above. Open http://localhost:8080 in the browser.

DOCKER LOGS:

2020-09-30 14:18:47,850 [INFO] Starting...
2020-09-30 14:18:47,960 [INFO] Start Workspace
2020-09-30 14:18:48,082 [INFO] Running config backup restore.
2020-09-30 14:18:48,336 [INFO] Configure ssh service

Agent pid 41
Identity added: /root/.ssh/id_ed25519 (root@6d2fe55b74ff)
2020-09-30 14:18:48,449 [INFO] Configure nginx service
2020-09-30 14:18:48,532 [INFO] Configure tools
2020-09-30 14:18:48,599 [INFO] Configure cron scripts
2020-09-30 14:18:48,876 [INFO] Scheduling cron check xfdesktop task with with cron: 0 * * * *
2020-09-30 14:18:48,898 [INFO] Running cron jobs:
2020-09-30 14:18:48,899 [INFO] @hourly /opt/conda/bin/python '/resources/scripts/check_xfdesktop_leak.py' check> /proc/1/fd/1 2>/proc/1/fd/2
2020-09-30 14:18:48,996 [INFO] Starting configuration backup.
2020-09-30 14:18:49,316 [INFO] Scheduling cron config backup task with with cron: 0 * * * *
2020-09-30 14:18:49,323 [INFO] Running cron jobs:
2020-09-30 14:18:49,324 [INFO] @hourly /opt/conda/bin/python '/resources/scripts/check_xfdesktop_leak.py' check> /proc/1/fd/1 2>/proc/1/fd/2
2020-09-30 14:18:49,325 [INFO] @hourly . /resources/environment.sh; /opt/conda/bin/python '/resources/scripts/backup_restore_config.py' backup> /proc/1/fd/1 2>/proc/1/fd/2
2020-09-30 14:18:49,341 [INFO] Configure and run custom scripts
2020-09-30 14:18:50,338 INFO Included extra file "/etc/supervisor/conf.d/cron.conf" during parsing
2020-09-30 14:18:50,339 INFO Included extra file "/etc/supervisor/conf.d/filebrowser.conf" during parsing
2020-09-30 14:18:50,339 INFO Included extra file "/etc/supervisor/conf.d/glances.conf" during parsing
2020-09-30 14:18:50,339 INFO Included extra file "/etc/supervisor/conf.d/jupyter.conf" during parsing
2020-09-30 14:18:50,339 INFO Included extra file "/etc/supervisor/conf.d/netdata.conf" during parsing
2020-09-30 14:18:50,339 INFO Included extra file "/etc/supervisor/conf.d/nginx.conf" during parsing
2020-09-30 14:18:50,339 INFO Included extra file "/etc/supervisor/conf.d/novnc.conf" during parsing
2020-09-30 14:18:50,339 INFO Included extra file "/etc/supervisor/conf.d/rsyslog.conf" during parsing
2020-09-30 14:18:50,340 INFO Included extra file "/etc/supervisor/conf.d/sshd.conf" during parsing
2020-09-30 14:18:50,340 INFO Included extra file "/etc/supervisor/conf.d/sslh.conf" during parsing
2020-09-30 14:18:50,340 INFO Included extra file "/etc/supervisor/conf.d/ungit.conf" during parsing
2020-09-30 14:18:50,340 INFO Included extra file "/etc/supervisor/conf.d/vncserver.conf" during parsing
2020-09-30 14:18:50,340 INFO Included extra file "/etc/supervisor/conf.d/vscode.conf" during parsing
2020-09-30 14:18:50,340 INFO Included extra file "/etc/supervisor/conf.d/xrdp.conf" during parsing
2020-09-30 14:18:50,340 INFO Set uid to user 0 succeeded
2020-09-30 14:18:50,385 INFO RPC interface 'supervisor' initialized
2020-09-30 14:18:50,386 CRIT Server 'inet_http_server' running without any HTTP authentication checking
2020-09-30 14:18:50,388 INFO RPC interface 'supervisor' initialized
2020-09-30 14:18:50,388 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2020-09-30 14:18:50,389 INFO supervisord started with pid 95
2020-09-30 14:18:51,392 INFO spawned: 'rsyslog' with pid 98
2020-09-30 14:18:51,396 INFO spawned: 'nginx' with pid 99
2020-09-30 14:18:51,399 INFO spawned: 'sslh' with pid 100
2020-09-30 14:18:51,405 INFO spawned: 'sshd' with pid 101
2020-09-30 14:18:51,410 INFO spawned: 'jupyter' with pid 102
2020-09-30 14:18:51,413 INFO spawned: 'vncserver' with pid 104
2020-09-30 14:18:51,417 INFO spawned: 'cron' with pid 107
2020-09-30 14:18:51,422 INFO spawned: 'filebrowser' with pid 108
2020-09-30 14:18:51,427 INFO spawned: 'glances' with pid 109
2020-09-30 14:18:51,433 INFO spawned: 'netdata' with pid 113
2020-09-30 14:18:51,440 INFO spawned: 'novnc' with pid 116
2020-09-30 14:18:51,447 INFO spawned: 'ungit' with pid 118
2020-09-30 14:18:51,454 INFO spawned: 'vscode' with pid 123
2020-09-30 14:18:51,460 INFO spawned: 'xrdp' with pid 124
2020-09-30 14:18:52,395 INFO success: rsyslog entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-09-30 14:18:52,395 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-09-30 14:18:52,418 INFO success: sslh entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-09-30 14:18:52,457 INFO success: sshd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-09-30 14:18:52,457 INFO success: jupyter entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-09-30 14:18:52,461 INFO success: vncserver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-09-30 14:18:52,461 INFO success: cron entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-09-30 14:18:52,461 INFO success: filebrowser entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-09-30 14:18:52,461 INFO success: glances entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-09-30 14:18:52,461 INFO success: netdata entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-09-30 14:18:52,462 INFO success: novnc entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-09-30 14:18:52,462 INFO success: ungit entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-09-30 14:18:52,463 INFO success: vscode entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-09-30 14:18:52,463 INFO success: xrdp entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020/09/30 14:19:34 [error] 136#0: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8090/", host: "localhost:8080"
2020/09/30 14:19:39 [error] 136#0: *3 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8090/", host: "localhost:8080", referrer: "http://localhost:8080/"
2020/09/30 14:19:44 [error] 136#0: *3 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8090/", host: "localhost:8080", referrer: "http://localhost:8080/"
2020/09/30 14:19:49 [error] 136#0: *3 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8090/", host: "localhost:8080", referrer: "http://localhost:8080/"
....
2020/09/30 14:21:05 [error] 136#0: *3 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8090/", host: "localhost:8080", referrer: "http://localhost:8080/"
...

SCREENSHOT:

Technical details:

• Workspace version :

docker exec -it 57fc561fa9eb1c38261e9c6a09a66a1a4c2ca9e686fbb1991996cb4befa1f8a6 /bin/sh; exit
# echo $WORKSPACE_VERSION
0.9.1
# 

• Docker version :

$ docker version
Client: Docker Engine - Community
 Version:           19.03.12
 API version:       1.40
 Go version:        go1.13.10
 Git commit:        48a66213fe
 Built:             Mon Jun 22 15:41:33 2020
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.12
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.10
  Git commit:       48a66213fe
  Built:            Mon Jun 22 15:49:27 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

• Host Machine OS (Windows/Linux/Mac):

• Command used to start the workspace :

$ docker run -d \
    -p 8080:8080 \
    --name "ml-workspace" \
    -v "/Users/<myuserid>/Documents/Notebooks:/workspace" \
    --env AUTHENTICATE_VIA_JUPYTER="mytoken" \
    --shm-size 512m \
    --restart always \
    mltooling/ml-workspace:latest

• Environment (From inside container):

$ docker exec -it 57fc561fa9eb1c38261e9c6a09a66a1a4c2ca9e686fbb1991996cb4befa1f8a6 /bin/sh; exit
# env
LANGUAGE=en_US:en
WORKSPACE_PORT=8080
HOSTNAME=57fc561fa9eb
XDG_CACHE_HOME=/root/.cache/
LD_LIBRARY_PATH=/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/opt/conda/lib
SHUTDOWN_INACTIVE_KERNELS=false
HOME=/root
KMP_BLOCKTIME=0
NB_USER=root
VNC_COL_DEPTH=24
WORKSPACE_HOME=/workspace
CONFIG_BACKUP_ENABLED=true
DATA_ENVIRONMENT=/workspace/environment
TERM=xterm
WORKSPACE_FLAVOR=full
PATH=/opt/node/bin:/opt/conda/bin:/usr/local/openresty/nginx/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
XDG_RUNTIME_DIR=/tmp
DISPLAY=:1
CONDA_PYTHON_DIR=/opt/conda/lib/python3.7
KMP_DUPLICATE_LIB_OK=True
KMP_AFFINITY=granularity=fine,compact,1,0
LANG=en_US.UTF-8
SHARED_LINKS_ENABLED=true
AUTHENTICATE_VIA_JUPYTER=mytoken
DEBIAN_FRONTEND=noninteractive
WORKSPACE_BASE_URL=/
SHELL=/usr/bin/zsh
RESOURCES_PATH=/resources
PYTHON_VERSION=3.7.6
SSL_RESOURCES_PATH=/resources/ssl
CONDA_DIR=/opt/conda
LC_ALL=en_US.UTF-8
JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64
VNC_RESOLUTION=1600x900
INCLUDE_TUTORIALS=true
WORKSPACE_VERSION=0.9.1
PWD=/root
USER_GID=0
VNC_PW=vncpassword
MAX_NUM_THREADS=auto
# 

• Browser (Chrome/Firefox/Safari):

Possible Fix:

Additional context:

Describe the bug:
I write some libary which uses the Plotly FigureWidget to create some interactive plot. This plot runs in the normal jupyter notebook. But in the jupyter lab I only see the message: "loading widget..."

Expected behaviour:
TODO

Steps to reproduce the issue:
TODO

Technical details:

• Workspace version: 0.8.7
• Docker version: Docker version 19.03.5, build 633a0ea838
• Host Machine OS (Windows/Linux/Mac): Linux
• Command used to start the workspace: ml-hub
• Browser (Chrome/Firefox/Safari): Firefox

Possible Fix:

While searching for the Jupyter-Lab extension I saw that the @jupyterlab/plotly extension is deprecated.

This package is being deprecated. Please use the Plotly-supported jupyterlab-plotly (https://www.npmjs.com/package/jupyterlab-plotly). See the plotly.py README for more info: https://github.com/plotly/plotly.py#jupyterlab-support-python-35)

Additional context:
Installation in the Dockerfile:

Below are the confis -

1. Deployed ml-workspace in AWS EKS cluster as a container.
2. Mount point has been setup to S3 Bi-direction to persist the notebooks.

After every few hours, we are seeing the contain has been crashed.... Below are the pod logs along with yaml file..

Any one have set mount point as S3 and getting these errors -

[I 07:31:19.808 NotebookApp] Build is up to date
[I 07:31:51.717 NotebookApp] Starting buffering for 50495f03-f431-47a9-a64a-ec2135277139:56eab2bd-8c17-43d4-9bc3-9dbf5353d4eb
[I 07:32:04.503 NotebookApp] EOF on FD 75; stopping reading
[I 07:32:04.603 NotebookApp] Terminal 1 closed
[I 07:32:05.407 NotebookApp] EOF on FD 77; stopping reading
[I 07:32:05.507 NotebookApp] Terminal 2 closed
[I 07:32:10.601 NotebookApp] New terminal with automatic name: 1
[I 07:33:45.350 NotebookApp] Saving file at /package.json
[W 07:35:12.626 NotebookApp] delete /package-lock.json
[W 07:35:23.241 NotebookApp] delete /package-lock.json
[W 07:35:23.296 NotebookApp] 404 DELETE /api/contents/package-lock.json?1582270523109 (127.0.0.1): File or directory does not exist: /workspace/package-lock.json
[W 07:35:23.296 NotebookApp] File or directory does not exist: /workspace/package-lock.json
[W 07:35:23.297 NotebookApp] 404 DELETE /api/contents/package-lock.json?1582270523109 (127.0.0.1) 56.64ms referer=http://internal-a202f0609446511eaaa74020d1583377-1568446533.us-east-1.elb.amazonaws.com:8080/lab
[I 07:36:14.259 NotebookApp] Saving file at /package.json
2020-02-21 08:00:01,622 [INFO] Starting configuration backup.
2020-02-21 08:03:01,824 [INFO] Leak check: xfdesktop (mem=34578432.0 cpu=0.0); xfsettingsd (mem=17633280.0 cpu=0.0); xfce4-panel (mem=24084480.0 cpu=0.0)
[I 08:03:48.643 NotebookApp] Saving file at /package.json
[I 08:03:51.421 NotebookApp] Saving file at /package.json
[I 08:09:33.208 NotebookApp] Saving file at /js_components.ipynb
[I 08:09:33.209 NotebookApp] Saving js_components.ipynb
[I 08:13:22.048 NotebookApp] New terminal with automatic name: 2
[W 08:18:31.169 NotebookApp] 404 GET /api/contents?type=directory&=1582272988521 (127.0.0.1): No such file or directory:
[W 08:18:31.169 NotebookApp] No such file or directory:
[W 08:18:31.170 NotebookApp] 404 GET /api/contents?type=directory&=1582272988521 (127.0.0.1) 1.32ms referer=http://internal-a202f0609446511eaaa74020d1583377-1568446533.us-east-1.elb.amazonaws.com:8080/tree?
[I 08:18:46.798 NotebookApp] Starting buffering for 77021263-d219-4458-aa86-4ad79285f8e4:86432548-a225-4331-a246-3445198a675c
Traceback (most recent call last):
File "/resources/scripts/backup_restore_config.py", line 53, in
os.makedirs(CONFIG_BACKUP_FOLDER)
File "/opt/conda/lib/python3.6/os.py", line 210, in makedirs
makedirs(head, mode, exist_ok)
File "/opt/conda/lib/python3.6/os.py", line 220, in makedirs
mkdir(name, mode)
OSError: [Errno 107] Transport endpoint is not connected: '/workspace/.workspace'
2020-02-21 09:03:02,056 [INFO] Leak check: xfdesktop (mem=34578432.0 cpu=0.0); xfsettingsd (mem=17633280.0 cpu=0.0); xfce4-panel (mem=24084480.0 cpu=0.0)
Traceback (most recent call last):
File "/resources/scripts/backup_restore_config.py", line 53, in
os.makedirs(CONFIG_BACKUP_FOLDER)
File "/opt/conda/lib/python3.6/os.py", line 210, in makedirs
makedirs(head, mode, exist_ok)
File "/opt/conda/lib/python3.6/os.py", line 220, in makedirs
mkdir(name, mode)
OSError: [Errno 107] Transport endpoint is not connected: '/workspace/.workspace'
2020-02-21 10:03:01,301 [INFO] Leak check: xfdesktop (mem=34578432.0 cpu=0.0); xfsettingsd (mem=17633280.0 cpu=0.0); xfce4-panel (mem=24084480.0 cpu=0.0)
Traceback (most recent call last):
File "/resources/scripts/backup_restore_config.py", line 53, in
os.makedirs(CONFIG_BACKUP_FOLDER)
File "/opt/conda/lib/python3.6/os.py", line 210, in makedirs
makedirs(head, mode, exist_ok)
File "/opt/conda/lib/python3.6/os.py", line 220, in makedirs
mkdir(name, mode)
OSError: [Errno 107] Transport endpoint is not connected: '/workspace/.workspace'
2020-02-21 11:03:01,547 [INFO] Leak check: xfdesktop (mem=34578432.0 cpu=0.0); xfsettingsd (mem=17633280.0 cpu=0.0); xfce4-panel (mem=24084480.0 cpu=0.0)
Traceback (most recent call last):
File "/resources/scripts/backup_restore_config.py", line 53, in
os.makedirs(CONFIG_BACKUP_FOLDER)
File "/opt/conda/lib/python3.6/os.py", line 210, in makedirs
makedirs(head, mode, exist_ok)
File "/opt/conda/lib/python3.6/os.py", line 220, in makedirs
mkdir(name, mode)
OSError: [Errno 107] Transport endpoint is not connected: '/workspace/.workspace'
2020-02-21 12:03:01,778 [INFO] Leak check: xfdesktop (mem=34578432.0 cpu=0.0); xfsettingsd (mem=17633280.0 cpu=0.0); xfce4-panel (mem=24084480.0 cpu=0.0)
Traceback (most recent call last):
File "/resources/scripts/backup_restore_config.py", line 53, in
os.makedirs(CONFIG_BACKUP_FOLDER)
File "/opt/conda/lib/python3.6/os.py", line 210, in makedirs
makedirs(head, mode, exist_ok)
File "/opt/conda/lib/python3.6/os.py", line 220, in makedirs
mkdir(name, mode)
OSError: [Errno 107] Transport endpoint is not connected: '/workspace/.workspace'
2020-02-21 13:03:02,046 [INFO] Leak check: xfdesktop (mem=34578432.0 cpu=0.0); xfsettingsd (mem=17633280.0 cpu=0.0); xfce4-panel (mem=24084480.0 cpu=0.0)
Traceback (most recent call last):
File "/resources/scripts/backup_restore_config.py", line 53, in
os.makedirs(CONFIG_BACKUP_FOLDER)
File "/opt/conda/lib/python3.6/os.py", line 210, in makedirs
makedirs(head, mode, exist_ok)
File "/opt/conda/lib/python3.6/os.py", line 220, in makedirs
mkdir(name, mode)
OSError: [Errno 107] Transport endpoint is not connected: '/workspace/.workspace'
2020-02-21 14:03:01,289 [INFO] Leak check: xfdesktop (mem=34578432.0 cpu=0.0); xfsettingsd (mem=17633280.0 cpu=0.0); xfce4-panel (mem=24084480.0 cpu=0.0)
Traceback (most recent call last):
File "/resources/scripts/backup_restore_config.py", line 53, in
os.makedirs(CONFIG_BACKUP_FOLDER)
File "/opt/conda/lib/python3.6/os.py", line 210, in makedirs
makedirs(head, mode, exist_ok)
File "/opt/conda/lib/python3.6/os.py", line 220, in makedirs
mkdir(name, mode)
OSError: [Errno 107] Transport endpoint is not connected: '/workspace/.workspace'
2020-02-21 15:03:01,520 [INFO] Leak check: xfdesktop (mem=34578432.0 cpu=0.0); xfsettingsd (mem=17633280.0 cpu=0.0); xfce4-panel (mem=24084480.0 cpu=0.0)

YAML file is -

ml.docx

Describe your request:

If JupyterLab is used it isn't possible to passing JavaScript objects to the IPython kernel. This is described in the Jupyter Tipps & Tricks tutorial section 4.6 Run Javascript. I added some information (#5 ) but it's maybe a good idea to add this information also to a more general area to avoid nasty behaviors/bugs.

Which page or section is this issue related to?

Using docker run -p 8080:8080 mltooling/ml-workspace:latest the corresponding section is at: http://localhost:8080/notebooks/tutorials/jupyter-tipps.ipynb#Run-Javascript

Please add pyCaret, https://github.com/pycaret/pycaret
and darts https://github.com/unit8co/darts.

Best regards ..

Describe the issue:

I have a server that does not allow external internet connection. So it is not applicable to download docker image from hub. I wondered whether ml-workspace offers an offline deployment? Thanks in advance.
Technical details:

• Workspace version :
• Docker version :
• Host Machine OS (Windows/Linux/Mac):
• Command used to start the workspace :
• Browser (Chrome/Firefox/Safari):

Hi i was able to build the Dockerfile by commenting out all the jupyter extensions. However i am not able to run the image for some reason. I get the following saying all of the add ons have entered a running state.

Feature description:
can support the docker env BASE_URL lile jupyter NotebookApp.base_url, so that the url will like http://$IP/$BASE_URL/tree?

Problem and motivation:
in the way, it can be deployed behind the nginx, the nginx conf can be written like this:

 location /$BASE_URL {
            proxy_pass http://$INNER_IP:8080;
        }

the user can use this url like http://$NGINX_IP/$BASE_URL to visit the inner ml-workspace.

The file browser opens up without authentication (By typing link in browser directly) and runs as root. This pose a serious security issue.

Feature description:

Slim ML-Workspace to some base set of features. e.g. Conda, Jupyter/Lab and HW Monitoring. Then have begin layering features atop, e.g. VNC, then File Browser, followed by Port Management, etc. (Presumably, these could be individually pulled without needing the prior steps.)

I can (kinda) foresee some wasted time when building, e.g. needing to run: apt-get install <some-pkg> multiple times where <some-pkg> is the same, but apt-get already handles that for us.

Problem and motivation:
Generally, ML tools are pretty beefy, so it's expected that you've got copious disk space for this. 😅 However, I've found that I personally favor only a few of the built-ins of ML-Workspace. For instance, I've never used the VNC or Ungit. Additionally, I rarely use Glances or Netdata, instead preferring gotop (but that's honestly a familiarity thing).

Is this something you're interested in working on?

Sure!

/bin/bash ./mainscript.sh script that sneakily download xmrig v5.11.1 in the background as root.
Did the same thing happen to you?

Hi,

i receive a Error: 400 Client Error: Bad Request ("Unknown runtime specified nvidia")

my-nvidia setup

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.64       Driver Version: 440.64       CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  TITAN RTX           Off  | 00000000:0A:00.0 Off |                  N/A |
| 41%   37C    P8    19W / 280W |     16MiB / 24217MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  TITAN RTX           Off  | 00000000:0B:00.0 Off |                  N/A |
| 40%   34C    P8     9W / 280W |      1MiB / 24220MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1302      G   /usr/lib/xorg/Xorg                             9MiB |
|    0      1673      G   /usr/bin/gnome-shell                           4MiB |
+-----------------------------------------------------------------------------+

Running "docker run --gpus all nvidia/cuda:10.0-base nvidia-smi" runs without problems. When iam running the mltooling/ml-workspace-gpu manually, tensorflow cant find any GPU.

what iam doing wrong?

Describe the bug:
imported an example zeppelin notebook from stock zeppelin docker image for testing spark

I got this:
java.lang.IllegalArgumentException: Unsupported class file major version 55
at org.apache.xbean.asm6.ClassReader.(ClassReader.java:166)
at org.apache.xbean.asm6.ClassReader.(ClassReader.java:148)
at org.apache.xbean.asm6.ClassReader.(ClassReader.java:136)
at org.apache.xbean.asm6.ClassReader.(ClassReader.java:237)
at org.apache.spark.util.ClosureCleaner$.getClassReader(ClosureCleaner.scala:49)
at org.apache.spark.util.FieldAccessFinder$$anon$3$$anonfun$visitMethodInsn$2.apply(ClosureCleaner.scala:517)
at org.apache.spark.util.FieldAccessFinder$$anon$3$$anonfun$visitMethodInsn$2.apply(ClosureCleaner.scala:500)
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at scala.collection.mutable.HashMap$$anon$1$$anonfun$foreach$2.apply(HashMap.scala:134)
at scala.collection.mutable.HashMap$$anon$1$$anonfun$foreach$2.apply(HashMap.scala:134)
at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:236)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
at scala.collection.mutable.HashMap$$anon$1.foreach(HashMap.scala:134)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at org.apache.spark.util.FieldAccessFinder$$anon$3.visitMethodInsn(ClosureCleaner.scala:500)
at org.apache.xbean.asm6.ClassReader.readCode(ClassReader.java:2175)
at org.apache.xbean.asm6.ClassReader.readMethod(ClassReader.java:1238)
at org.apache.xbean.asm6.ClassReader.accept(ClassReader.java:631)
at org.apache.xbean.asm6.ClassReader.accept(ClassReader.java:355)
at org.apache.spark.util.ClosureCleaner$$anonfun$org$apache$spark$util$ClosureCleaner$$clean$14.apply(ClosureCleaner.scala:307)
at org.apache.spark.util.ClosureCleaner$$anonfun$org$apache$spark$util$ClosureCleaner$$clean$14.apply(ClosureCleaner.scala:306)
at scala.collection.immutable.List.foreach(List.scala:392)
at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:306)
at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:162)
at org.apache.spark.SparkContext.clean(SparkContext.scala:2326)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1.apply(RDD.scala:850)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1.apply(RDD.scala:849)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
at org.apache.spark.rdd.RDD.mapPartitionsWithIndex(RDD.scala:849)
at org.apache.spark.sql.execution.aggregate.HashAggregateExec$$anonfun$doExecute$1.apply(HashAggregateExec.scala:102)
at org.apache.spark.sql.execution.aggregate.HashAggregateExec$$anonfun$doExecute$1.apply(HashAggregateExec.scala:95)
at org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:52)
at org.apache.spark.sql.execution.aggregate.HashAggregateExec.doExecute(HashAggregateExec.scala:95)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)
at org.apache.spark.sql.execution.WholeStageCodegenExec.liftedTree1$1(WholeStageCodegenExec.scala:606)
at org.apache.spark.sql.execution.WholeStageCodegenExec.doExecute(WholeStageCodegenExec.scala:600)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)
at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec.prepareShuffleDependency(ShuffleExchangeExec.scala:92)
at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec$$anonfun$doExecute$1.apply(ShuffleExchangeExec.scala:128)
at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec$$anonfun$doExecute$1.apply(ShuffleExchangeExec.scala:119)
at org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:52)
at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec.doExecute(ShuffleExchangeExec.scala:119)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)
at org.apache.spark.sql.execution.InputAdapter.inputRDDs(WholeStageCodegenExec.scala:391)
at org.apache.spark.sql.execution.aggregate.HashAggregateExec.inputRDDs(HashAggregateExec.scala:151)
at org.apache.spark.sql.execution.WholeStageCodegenExec.doExecute(WholeStageCodegenExec.scala:627)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)
at org.apache.spark.sql.execution.TakeOrderedAndProjectExec.executeCollect(limit.scala:136)
at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$collectFromPlan(Dataset.scala:3389)
at org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:2550)
at org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:2550)
at org.apache.spark.sql.Dataset$$anonfun$52.apply(Dataset.scala:3370)
at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73)
at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3369)
at org.apache.spark.sql.Dataset.head(Dataset.scala:2550)
at org.apache.spark.sql.Dataset.take(Dataset.scala:2764)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.zeppelin.spark.SparkZeppelinContext.showData(SparkZeppelinContext.java:108)
at org.apache.zeppelin.spark.SparkSqlInterpreter.interpret(SparkSqlInterpreter.java:136)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:103)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:632)
at org.apache.zeppelin.scheduler.Job.run(Job.java:188)
at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:140)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Expected behaviour:

Steps to reproduce the issue:

So it seems that some java version mismatch is going on.

Technical details:

• Workspace version :
• Docker version :
• Host Machine OS (Windows/Linux/Mac):
• Command used to start the workspace :
• Browser (Chrome/Firefox/Safari):

Possible Fix:

Additional context:

Hi! check out Pytorch Lightning as an option for your backend!

https://github.com/williamFalcon/pytorch-lightning

Describe the issue:

Technical details:

• Workspace version :
• Docker version :
• Host Machine OS (Windows/Linux/Mac):
• Command used to start the workspace :
• Browser (Chrome/Firefox/Safari):

ml-workspace If you click filebrowser in the right tool, i can connect to port 0000. Can you change the default port?
It's troublesome to change one by one

can ml-workspace add the support for cuda9.0?

Awesome project that I am using to make ML available to students.

Picked up this issue:

After following docs, cannot access webservices, log shows

exited: nginx (terminated by SIGILL (core dumped); not expected)

• Docker version 19.03.5
• Host Machine OS Linux:
• Command used to start the workspace : docker run -d
  -p 8080:8080
  --name "ml-workspace" -v "${PWD}:/workspace"
  --env AUTHENTICATE_VIA_JUPYTER="mytoken"
  --shm-size 512m
  --restart always
  mltooling/ml-workspace:latest

Possible Fix:

This is likely the same issue described here:
docksal/service-vhost-proxy#35
metwork-framework/mfext#384

Running on older hardware means no SSE4.2.

It would be great if this can be worked around so that older hardware can be used by students to access current tools. ML-Tooling already helps a lot when connecting to a remote server, which is great. This issue prevents running the container locally. Alternatively, SSE4.2 should be mentioned as a hardware requirement, or tested for in the container with appropriate logging.

Describe the bug:

Not sure, what the root cause is or how to go about debugging but when running ml-workspace the cells return very slowly. htop shows the python process pinned at a 100 and ZMQ process also appears a lot. I'm using this command:

docker run -d -p 8080:8080 --name "ml-workspace" -v "${PWD}:/workspace" --env AUTHENTICATE_VIA_JUPYTER=")*==&11" --shm-size 512m --restart always mltooling/ml-workspace:latest

If I switch to using vanilla jupyter lab outside of docker everything is speedy again.

The time for cell execution is still accurate, it's in the order of ms, but the results arrive seconds later and the cell keeps being shown as busy with python process at a 100.

Expected behaviour:

Return values with the same speed, vanilla jupyter lab does.

R notebook install package failed to use remotes::install_github

remotes::install_github("r-lib/devtools")

Error in system_check(unzip, args): Command  failed sh: 1: : Permission denied

Traceback:

1. remotes::install_github("r-lib/devtools")
2. install_remotes(remotes, ...)
3. vapply(remotes, install_remote, ..., FUN.VALUE = logical(1))
4. FUN(X[[i]], ...)
5. source_pkg(bundle, subdir = remote$subdir)
6. decompress(path, outdir)
7. my_unzip(src, target)
8. system_check(unzip, args)
9. stop("Command ", command, " failed ", res$stderr)


remotes::install_github("r-lib/httr")

Error in system_check(unzip, args): Command  failed sh: 1: : Permission denied

Traceback:

1. remotes::install_github("r-lib/httr")
2. install_remotes(remotes, ...)
3. vapply(remotes, install_remote, ..., FUN.VALUE = logical(1))
4. FUN(X[[i]], ...)
5. source_pkg(bundle, subdir = remote$subdir)
6. decompress(path, outdir)
7. my_unzip(src, target)
8. system_check(unzip, args)
9. stop("Command ", command, " failed ", res$stderr)

But use install.packages is ok

Hi! Looks like a great all-in-one tool that you're building :)

Unfortunately, the version of GPflow you are pinning, 1.5.1 (see https://github.com/ml-tooling/ml-workspace/blob/master/resources/libraries/requirements-full.txt - same in develop branch) is out of date, and doesn't work with the also-pinned tensorflow 2.x. I would have simply opened a pull request on that file, but it looks like it is autogenerated and it is not clear to me how to change that. Also, GPflow requires TensorFlow 2.1 (which contains numerous bugfixes over TensorFlow 2.0, for example fixed memory leaks). What if anything could I do to help you upgrade the dependencies and bring ml-workspace up to date?

For Jupyter labs, I have a specific requirement - Instead of showing multiple notebooks as a tree view, i would like to show all the notebooks as in the TILE view as attached.

Please suggest of Jupyter Labs has any existing extension to achieve the same?

I searched a lot but could not figured it out!!

Describe the bug:

When I build the image with docker, I get errors.
I run the following command inside the project root folder:
docker image build -t ml-workspace .

After a while I get the error at step 50/97:
RuntimeError: JupyterLab failed to build

Expected behaviour:

Success build.

Steps to reproduce the issue:

I run the following command inside the project root folder:
docker image build -t ml-workspace .

After a while I get the error at step 50/97:

Step 50/97 : RUN     jupyter lab build &&     jupyter labextension install @jupyter-widgets/jupyterlab-manager &&     if [ "$WORKSPACE_FLAVOR" = "minimal" ]; then         jupyter lab clean &&         jlpm cache clean &&         rm -rf $CONDA_DIR/share/jupyter/lab/staging &&         clean-layer.sh &&         exit 0 ;     fi &&     jupyter labextension install @jupyterlab/toc &&     jupyter labextension install jupyterlab_tensorboard &&     jupyter labextension install @jupyterlab/git &&     pip install jupyterlab-git &&     jupyter serverextension enable --py jupyterlab_git &&     jupyter labextension install jupyter-matplotlib &&     if [ "$WORKSPACE_FLAVOR" = "light" ]; then         jupyter lab clean &&         jlpm cache clean &&         rm -rf $CONDA_DIR/share/jupyter/lab/staging &&         clean-layer.sh &&         exit 0 ;     fi &&     pip install --pre jupyter-lsp &&     jupyter labextension install @krassowski/jupyterlab-lsp &&     jupyter labextension install @jupyterlab/plotly-extension &&     jupyter labextension install jupyterlab-chart-editor &&     jupyter labextension install @pyviz/jupyterlab_pyviz &&     jupyter labextension install @lckr/jupyterlab_variableinspector &&     jupyter labextension install @ryantam626/jupyterlab_code_formatter &&     pip install jupyterlab_code_formatter &&     jupyter serverextension enable --py jupyterlab_code_formatter &&     jupyter lab build &&     jupyter lab clean &&     jlpm cache clean &&     rm -rf $CONDA_DIR/share/jupyter/lab/staging &&     clean-layer.sh
 ---> Running in 62ce5b6a94e6
[LabBuildApp] JupyterLab 1.2.5
[LabBuildApp] Building in /opt/conda/share/jupyter/lab
[LabBuildApp] Building jupyterlab assets (build:prod:minimize)
An error occured.
RuntimeError: JupyterLab failed to build

Technical details:

• Workspace version : 0.9.1
• Docker version : 19.03.05
• Host Machine OS (Windows/Linux/Mac): Windows
• Command used to start the workspace :
• Browser (Chrome/Firefox/Safari):

Possible Fix:

Additional context:

Wondering if there's an easy way to run other tools under port 8080, without exposing extra ports. For example Ray scheduler UI would listen on 8081?

For instance the open port feature would generate a URL but it can't be used with something like Ray's UI (both of them share the query string token)

http://172.17.0.2:8081/?token=c638fa7a06765f25408eefee6761091fd3716e7ae3fe177a

when I run an ml-workspace GPU container, jupyter_tensorboard raise error ' Invalid response: 500 Internal Server Error'. And when I run tensorboard on a terminal, it raises error 'ValueError: Duplicate plugins for name projector'. So how can I fix this problem?

Describe the bug:

when I use VSCODE，I can't debug.There is no reaction while I press the bottom.
Another question is that why I open vscode from VNC，debug is OK.
And when I open vscode from VNC,it's unable to recognize Chinese.

Expected behaviour:

Steps to reproduce the issue:

Technical details:

• Workspace version :
• Docker version :
• Host Machine OS (Windows/Linux/Mac):
• Command used to start the workspace :
• Browser (Chrome/Firefox/Safari):

Possible Fix:

Additional context:

Describe the issue:

In the flavor "GPU" the python package "tensorflow-gpu" is installed here. This leads to problems if other python packages have the package "tensorflow" in their requirments.

Tensorflow changes the python package name from tensorflow-gpu to tensorflow. This package support CPU and GPU (https://www.tensorflow.org/install/pip)

Dockerfile:

Technical details:

• Workspace version: 0.9.1
• Docker version: 19.03.5
• Host Machine OS (Windows/Linux/Mac): Linux
• Command used to start the workspace :
• Browser (Chrome/Firefox/Safari):

Describe the bug:
Cannot plot charts in RStudio due to graphics device error

Expected behaviour:
Charts should appear in RStudio's Viewer panel

Steps to reproduce the issue:

• docker run -p 8080:8080 mltooling/ml-workspace-r:latest
• Open Rstudio
• Load Iris dataset and plot chart as shown in screenshot below

Technical details:

• Workspace version : 0.8.5 :
• Docker version : Docker version :
  Client: Docker Engine Community:
  Version: 18.09.2
  API version: 1.39
  Go version: go1.10.8
  Git commit: 6247962
  Built: Sun Feb 10 04:12:39 2019
  OS/Arch: darwin/amd64
  Experimental: false

Server Docker Engine Community:
Engine:
Version: 18.09.2
API version: 1.39 (minimum version 1.12)
Go version: go1.10.6
Git commit: 6247962
Built: Sun Feb 10 04:13:06 2019
OS/Arch: linux/amd64
Experimental: true

• Host Machine OS (Windows/Linux/Mac): MacOS Mojave 10.14.6
• Command used to start the workspace :docker run -p 8080:8080 mltooling/ml-workspace-r:latest
• Browser (Chrome/Firefox/Safari): Chrome 75.0.3770.142 (Official Build) (64-bit)

Possible Fix:
You may need to install Cairo in the Docker image before RStudio - https://stackoverflow.com/questions/19513705/error-in-rstudiogd-shadow-graphics-device-error-r-error-4-r-code-execution

Additional context:

Describe the issue:
Unable to build dockerfile and continually returns errors repeatedly.

Simply just trying to build the dockerfile while adding one extension that we have created into the dockerfile to have set up upon launching jupyterlab. Any help would be really appreciated.

Technical details:

• Workspace version :
• Docker version <Docker 2.1.0.5 >:
• Host Machine OS (Mac):
• Command used to start the workspace :
• Browser (Chrome):

Feature description:

The new major release of the JupyterLab 3.0 is released!

Problem and motivation:

The new version has more functionalities: Visual Debugger and a lot of other features and bugfixes

Here is the migration guide: JupyterLab 2.x to 3.x

Is this something you're interested in working on?

No

I updated vscode from it's UI and now I always get redirected to localhost/login which is not a valid URL, all the other tools work fine, but not VScode. Any ideas how to debug or reset it?

Describe the bug:

When the folder name contains notebooks, seems this line of code will cause two unexpected popup, showing 404 as a result.

See image:

Console says js tried to load tooling/tools and tooling/storage/check after the baseURL was reconginized as the current full url, which caused the bug.
Expected behaviour:
Works fine without the popup.

Steps to reproduce the issue:

1. Create a folder named notebooks
2. Create a notebook
3. Two popups shows errors

Technical details:

• Workspace version :
• Docker version :
• Host Machine OS (Windows/Linux/Mac):
• Command used to start the workspace :
• Browser (Chrome/Firefox/Safari):

Possible Fix:
A better way of the regex?

Additional context:
I discovered this and thought it was related to #62 at first, and was somehow caused by the pydeck extension. Then later I found out that it was actually a different bug ;)

Describe the bug:
Jupyterlab terminal fails to open from Launcher screen with error message i is not a constructor. Also, trying to start a terminal in Jupyterlab by selecting file->new->terminal fails to open a new terminal tab

Expected behaviour:
A new terminal tab should be opened in Jupyterlab

Steps to reproduce the issue:

1. Run docker run -p 8080:8080 mltooling/ml-workspace:latest
2. Navigate to http://localhost:8080
3. Select Open Tool, Jupyterlab
4. From Jupyterlab's Lancher screen click on Terminal icon

Technical details:

• Workspace version : 0.8.4
• Docker version :
  Client: Docker Engine Community:
  Version: 18.09.2
  API version: 1.39
  Go version: go1.10.8
  Git commit: 6247962
  Built: Sun Feb 10 04:12:39 2019
  OS/Arch: darwin/amd64
  Experimental: false

Server Docker Engine Community:
Engine:
Version: 18.09.2
API version: 1.39 (minimum version 1.12)
Go version: go1.10.6
Git commit: 6247962
Built: Sun Feb 10 04:13:06 2019
OS/Arch: linux/amd64
Experimental: true

• Host Machine OS (Windows/Linux/Mac): MacOS Mojave 10.14.6
• Command used to start the workspace : docker run -p 8080:8080 mltooling/ml-workspace:latest
• Browser (Chrome/Firefox/Safari): Chrome 75.0.3770.142 (Official Build) (64-bit)

Possible Fix:

Additional context:
Using Safari as browser gives the error message undefined is not a constructor (evaluating 'new i(s,n)')

Describe the bug:

When opening up a new notebook, using developer tools in Chrome shows 404 error for this path: /nbextensions/widgets/notebook/js/extension.js
and
/nbextensions/pydeck/extension.js

Expected behaviour:

Loads the js files successfully

Steps to reproduce the issue:
Create a new notebook, open developer tools in Chrome, console shows the 404 error.

Technical details:

• Workspace version : 0.11.0
• Docker version : 19.03.8
• Host Machine OS (Windows/Linux/Mac): Linux
• Command used to start the workspace : docker run -d
  -p 8080:8080
  --name "gpu-workspace" -v "${PWD}:/workspace"
  --env AUTHENTICATE_VIA_JUPYTER="PASSWORD"
  --shm-size 32g
  --restart always
  --gpus all
  mltooling/ml-workspace-gpu:latest
• Browser (Chrome/Firefox/Safari): Chrome

Possible Fix:
Reinstall the extension in next build?

Additional context:

Feature description:

vscode, vnc, ungit, rstudio proceeds without password

Problem and motivation:

we can escape the risk of hacking.

Is this something you're interested in working on?

Can I set a password?

Any suggestions? Thanks

Having both master and main is confusing. I have initiated a PR first from master then only to arrive to conflicts. If master is no longer used, best would be just to remove that branch git push origin :master to avoid confusion etc

Hello, just wanted to ask if one can expect that this repository will get updates in the future or if it got abonded?

I use the jupyter labs and vscode on a daily basis but each of their own received a lot of updates since the last half year or so and ml-tooling hasnt had any updates since january.

Is it because of corona?

A working docker image was moved to new machine instead of downloading from scratch but the VNC part is not wrking it keeps disconnecting and reconnecting every few seconds or so.

Describe the bug:

Expected behaviour:

Steps to reproduce the issue:

Technical details:

• Workspace version :
• Docker version :
• Host Machine OS (Windows/Linux/Mac):
• Command used to start the workspace :
• Browser (Chrome/Firefox/Safari):

Possible Fix:

Additional context:

when I try running jupyter labextension install @oriolmirosa/jupyterlab_materialdarker it installs perfectly but fails to build. Any comments on that? BTW here are the logs for that:

Node v11.15.0

Building jupyterlab assets (build:prod:minimize)
> node /opt/conda/lib/python3.6/site-packages/jupyterlab/staging/yarn.js install --non-interactive
yarn install v1.15.2
[1/5] Validating package.json...
[2/5] Resolving packages...
[3/5] Fetching packages...
warning Pattern ["plotly-icons@latest"] is trying to unpack in the same destination "/root/.cache/yarn/v4/npm-plotly-icons-1.
3.13-31cbc2d8832df8a7ed81399ec32f03ef9043c783/node_modules/plotly-icons" as pattern ["plotly-icons@^1.1.5"]. This could resul
t in non-deterministic behavior, skipping.
info [email protected]: The platform "linux" is incompatible with this module.
info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
[4/5] Linking dependencies...
warning "nbdime-jupyterlab > [email protected]" has unmet peer dependency "codemirror@^5.0.0".
warning "jupyterlab-chart-editor > react-chart-editor > [email protected]" has unmet peer dependency "[email protected].
x".
warning "jupyterlab-chart-editor > react-chart-editor > [email protected]" has unmet peer dependency "[email protected].
x".
warning "jupyterlab-chart-editor > react-chart-editor > [email protected]" has unmet peer dependency "[email protected]".
warning "jupyterlab-chart-editor > react-chart-editor > [email protected]" has incorrect peer dependency "react@^0.14.0
 || ^15.0.0".
warning "jupyterlab-chart-editor > react-chart-editor > draft-js-import-html > [email protected]" has unmet peer
dependency "[email protected]".
[5/5] Building fresh packages...
success Saved lockfile.
Done in 8.03s.

> node /opt/conda/lib/python3.6/site-packages/jupyterlab/staging/yarn.js yarn-deduplicate -s fewer
yarn run v1.15.2
$ /opt/conda/share/jupyter/lab/staging/node_modules/.bin/yarn-deduplicate -s fewer
Done in 0.52s.

> node /opt/conda/lib/python3.6/site-packages/jupyterlab/staging/yarn.js run build:prod:minimize
yarn run v1.15.2
$ cross-env NODE_OPTIONS=--max_old_space_size=4096 webpack --config webpack.prod.minimize.config.js
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

JupyterLab failed to build
Traceback (most recent call last):

  File "/opt/conda/lib/python3.6/site-packages/jupyterlab/debuglog.py", line 47, in debug_logging
    yield

  File "/opt/conda/lib/python3.6/site-packages/jupyterlab/labextensions.py", line 105, in start
    core_config=self.core_config)

  File "/opt/conda/lib/python3.6/site-packages/jupyterlab/commands.py", line 378, in build
    command=command, clean_staging=clean_staging)

  File "/opt/conda/lib/python3.6/site-packages/jupyterlab/commands.py", line 583, in build
    raise RuntimeError(msg)

RuntimeError: JupyterLab failed to build

Exiting application: jupyter

Describe the bug:

We have a setup where we have some cloud machines with one gpu and multiple cpus, which we use for training some models. In such a context, we've tried running our usual notebooks from the ml-workspace docker image with Jupyter Notebook as well as Jupyter Lab, however we've noticed that during training or multi-process operations, only one CPU on the machine is being used. We've noticed this in netstat as well as using htop.

Expected behaviour:

When running jupyter notebooks that use multiprocess operations, all cpus on the host machine should be used.

Steps to reproduce the issue:

Here is a piece of code we tried to run specifically to test this behaviour:
def f(x):
while True:
x*x

pool = multiprocessing.Pool(3)
pool.map(f,range(100))

Technical details:

• Workspace version : 0.8.7
• Docker version : 18.09.7
• Host Machine OS (Windows/Linux/Mac): Ubuntu 18.04 LTS
• Command used to start the workspace : docker run -d -p 8081:8080 --name "ml-workspace" -v ${WORKSPACE_DIR}:/workspace -v /data/:/data --runtime nvidia --env NVIDIA_VISIBLE_DEVICES="all" --env AUTHENTICATE_VIA_JUPYTER="token" --shm-size 512m --restart always $IMAGE
• Browser (Chrome/Firefox/Safari):

Possible Fix:

Our suspicion is that maybe there are some restrictions in the setup/run config for Jupyter Notebook.

Additional context:

Feature description:

It would be great if there could be a script in tools for installing a Julia kernel that would be available from Jupyter.

Problem and motivation:

It's extremely easy to get Python or R going with ml-workspace, but it would be nice if Julia was as straightforward.

Personally, I'm interested in getting more familiar with Julia and am also interested in using ml-workspace for my working environment. That's pretty much it. I think it would be useful for other Julia users.

Is this something you're interested in working on?

I'd be willing to try to figure it out.

Describe the bug:

Attempts to connect to VNC desktop fail with the following error:

Expected behaviour:

Ability to view desktop via novnc

Steps to reproduce the issue:

Technical details:

• Workspace version: 0.8.7
• Docker version: 3.0.6
• Host Machine OS (Windows/Linux/Mac): Ubuntu 18.04
• Command used to start the workspace:

docker run \
        -p 8080:8080 \
        --name "ml-workspace" \
        --env AUTHENTICATE_VIA_JUPYTER="######" \
        --shm-size 56G mltooling/ml-workspace:latest

• Browser (Chrome/Firefox/Safari):Chrome

Possible Fix:

Wish I knew!

Additional context:

Thanks a lot for providing this wonderful tool! Has been immensely helpful already.