JupyterHub spawner based on libcloud for OpenStack

Home Page: https://libcloudspawner.readthedocs.io

License: Other

Python 83.94% Shell 4.60% Jinja 11.46%

jupyter spawner openstack jupyterhub jupyterhub-spawner

jupyterhub-libcloudspawner's Introduction

About LibCloudSpawner

LibcloudSpawner enables JupyterHub to spwan single-user notebooks server inside fresh cloud instance.

LibcloudSpawner is based on Apache Libcloud which aims to provide an abstraction API for a lot of cloud providers. Apache Libcloud provides the ability to manage resources in AWS, Google Cloud or Azure. But for the moment, only OpenStack is supported.

Overview

A Jupyterhub Spawner start each single-user notebook server. LibcloudSpawner is a spawner that operate with cloud API to manage singleuser notebook server instance life-cycle. Each Jupyterhub user will have one cloud instance for his usage.

Benefits

needs one small instance for JupyterHub server, all other resources are dynamic
single-user servers have real OS, feel free to integrate them in your information system (ActiveDirectory, authentication, mountpoint...)
simple access to VT-io or pci-passthrough devices (GPU)
ability to host third-party services on user instance (VNC, RStudio...)
no need for Docker or Kubernetes

Drawbacks

user server could be very slow to start (depending cloud provider and image size)
no need for Docker or Kubernetes

See the full documentation for informations, use-case and tutorials

jupyterhub-libcloudspawner's People

Contributors

Stargazers

Watchers

Forkers

tannevaled jnavarrof gitter-badger belavicm

jupyterhub-libcloudspawner's Issues

Creating a terminal doesn't open in correct folder

Upon testing the libcloudspawner, I've noticed that when creating a terminal in Jupyterhub, the terminal starts in /tmp by default, rather than the folder I've specified in c.LibcloudSpawner.notebook_dir

I've noticed on the VM spawned, that the systemd file shows the WorkingDirectory is set to /tmp, perhaps this is a starting point for you?

This is using libcloudspawner 1.3.0.0rc1, Jupyterhub 1.0.0

Setting MEM_LIMIT on singleuser instance to avoid instance OOMKiller

Following the libcloudspawner first version, a smart functionality have been introduced, use by singleuser jupyter to limit memory https://jupyterhub.readthedocs.io/en/stable/reference/spawners.html#memory-limits-guarantees

Spelling mistake when creating notebook server

Very minor issue, when selecting the options for a spawner, there is a spelling mistake with the word 'Resources'

Hopefully an easy one to do!

get_version_v3() got an unexpected keyword argument 'credentials'

Hi, I'm investigating using libcloudspawner as a way of making use of an OpenStack cloud infrastructure I have access to. I installed libcloudspawner through Pip and when I select an image through the drop downs, I get the following error message:

[E 2019-07-31 13:56:04.999 JupyterHub user:477] Unhandled error starting xfu59478's server: {"error": {"message": "get_version_v3() got an unexpected keyword argument 'credentials'", "code": 400, "title": "Bad Request"}}
[E 2019-07-31 13:56:05.073 JupyterHub pages:148] Failed to spawn single-user server with form
    Traceback (most recent call last):
      File "/home/xfu59478/jupyterhub/venv3.6/lib64/python3.6/site-packages/jupyterhub/handlers/pages.py", line 146, in post
        await self.spawn_single_user(user, options=options)
      File "/home/xfu59478/jupyterhub/venv3.6/lib64/python3.6/site-packages/jupyterhub/handlers/base.py", line 715, in spawn_single_user
        timedelta(seconds=self.slow_spawn_timeout), finish_spawn_future
      File "/home/xfu59478/jupyterhub/venv3.6/lib64/python3.6/site-packages/jupyterhub/handlers/base.py", line 636, in finish_user_spawn
        await spawn_future
      File "/home/xfu59478/jupyterhub/venv3.6/lib64/python3.6/site-packages/jupyterhub/user.py", line 489, in spawn
        raise e
      File "/home/xfu59478/jupyterhub/venv3.6/lib64/python3.6/site-packages/jupyterhub/user.py", line 409, in spawn
        url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
      File "/home/xfu59478/jupyterhub/venv3.6/lib64/python3.6/site-packages/libcloudspawner/spawner.py", line 236, in start
        self.user_options_from_form)
      File "/home/xfu59478/jupyterhub/venv3.6/lib64/python3.6/site-packages/libcloudspawner/manager/nodemanager.py", line 251, in create_machine
        node_conf['image'] = self._get_image(imagename)
      File "/home/xfu59478/jupyterhub/venv3.6/lib64/python3.6/site-packages/libcloudspawner/manager/nodemanager.py", line 90, in _get_image
        images = self.driver.list_images()
      File "/home/xfu59478/jupyterhub/venv3.6/lib64/python3.6/site-packages/libcloud/compute/drivers/openstack.py", line 364, in list_images
        self.connection.request('/images/detail').object, ex_only_active)
      File "/home/xfu59478/jupyterhub/venv3.6/lib64/python3.6/site-packages/libcloud/common/openstack.py", line 224, in request
        raw=raw)
      File "/home/xfu59478/jupyterhub/venv3.6/lib64/python3.6/site-packages/libcloud/common/base.py", line 534, in request
        action = self.morph_action_hook(action)
      File "/home/xfu59478/jupyterhub/venv3.6/lib64/python3.6/site-packages/libcloud/common/openstack.py", line 291, in morph_action_hook
        self._populate_hosts_and_request_paths()
      File "/home/xfu59478/jupyterhub/venv3.6/lib64/python3.6/site-packages/libcloud/common/openstack.py", line 325, in _populate_hosts_and_request_paths
        osa = osa.authenticate(**kwargs)  # may throw InvalidCreds
      File "/home/xfu59478/jupyterhub/venv3.6/lib64/python3.6/site-packages/libcloud/common/openstack_identity.py", line 808, in authenticate
        method='POST')
      File "/home/xfu59478/jupyterhub/venv3.6/lib64/python3.6/site-packages/libcloud/common/base.py", line 636, in request
        response = responseCls(**kwargs)
      File "/home/xfu59478/jupyterhub/venv3.6/lib64/python3.6/site-packages/libcloud/common/base.py", line 153, in __init__
        headers=self.headers)
    libcloud.common.exceptions.BaseHTTPError: {"error": {"message": "get_version_v3() got an unexpected keyword argument 'credentials'", "code": 400, "title": "Bad Request"}}

Here is the relevant section of my jupyterhub_config.py:

c.JupyterHub.spawner_class = 'libcloudspawner.spawner.LibcloudSpawner'

c.LibcloudSpawner.cloud_url = "https://mydomainname.com:4565/v3"
c.LibcloudSpawner.cloud_user = "myuser"
c.LibcloudSpawner.cloud_userpassword = "mypassword"
c.LibcloudSpawner.cloud_project = "myproject"
c.LibcloudSpawner.libcloudparams = {"arg_user_id": "myuser",
                                    "arg_key": "secret",
                                    "ex_force_auth_url": "https://openstack.stfc.ac.uk:5000/v3",
                                    "ex_tenant_name": "myproject",
                                    "ex_domain_name": "default"
                                   }

c.LibcloudSpawner.machine_sizes = [("1 CPU, 1GB RAM", "m1.tiny"),
                                   ("2 CPUs, 4GB RAM", "c3.small")]
c.LibcloudSpawner.machine_images = [("Centos-7", "Centos-7"),
                                    ("SL7 GUI", "ScientificLinux-7-Gui")]

As you can see, there are a few parameters that I haven't set as I'm not sure what they should be set to.
Have you had anyone else have experiences or any suggestions on how I can move forwards?
Thanks very much!

Version according to Pip is 1.2.1.dev2

JupyterHub can not retrieve instance state after JupyterHub shutdown

Even if JupyterHub is configured for keeping user servers and proxy configuration, when restarting JH, the spawner can't retrieve instance from OpenStack. Resulting, user lost is server and there is an orphaned instance on OS.

Steps to reproduce

Configure JupyterHub with libcloudSpawner. Add :

c.JupyterHub.cleanup_proxy = False
c.JupyterHub.cleanup_servers = False

As user, start a server
Kill or shutdown JupyterHub server
Relaunch JupyterHub

[D 2019-08-22 11:23:40.138 JupyterHub nodemanager:133] Can not retrieve node information             from cloud provider
[W 2019-08-22 11:23:40.139 JupyterHub app:1462] username appears to have stopped while the Hub was down

c.LibcloudSpawner.notebook_dir isn't functional

As requested in the Gitter chat, an issue regarding the notebook_dir config option when configuring Libcloudspawner.
By default, the notebook server will start in /tmp. I would expect to be able to specify a folder in the jupyterhub_config.py like you can do with most spawners, something like the following: c.LibcloudSpawner.notebook_dir = '/folder-name' - this is what I've tried, with no success.
I can see in the jupyterhub-singleuser.service file (stored at /etc/systemd/system) that has a line showing WorkingDirectory=/tmp. I would guess it's a case of setting the WorkingDirectory variable to what the user sets the notebook_dir to be.
Thanks!

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.

jupytercloud-project / jupyterhub-libcloudspawner Goto Github PK