Bug description

Running SystemUserSpawner through ProfileSpawner won't work correctly if c.SystemUserSpawner.remove = False is set in jupyterhub_config.py.

c.SystemUserSpawner.remove = False informs SystemUserSpawner not to automatically remove the Docker container if the user stops the server. This feature is extremly useful because it gives users the ability to install persistent packages.

Error message (web client):

500 : Internal Server Error
The error was:
Failed to check authorization: invalid_client

Cause

This bug occurs because ProfileSpawner always removes the user token from the oauth_clients table if the user stops the notebook server. SystemUserSpawner assumes that the user token remains in oauth_clients if c.SystemUserSpawner.remove = False.

Edit: Actually, this is only a part of the problem... Restoring the token in the database alone does not automatically fix the bug. Maybe someone who knows how the authentication works in more detail, could take a deeper look?

How to reproduce

1. Navigate to hub/home
2. Hit "Start My Server"
3. As soon as the Notebook Server is running, hit "Control Panel"
   • This will redirect you back to hub/home
4. Hit "Stop My Server"
   • This will stop the Docker container
5. Hit "Start My Server"
   • This will restart the Docker container and redirect you to user/foo but instead of the notebook interface the above error message will appear

Hello, thanks for implementing the interface. It looks like it could be extremely useful for our HPC cluster. We have multiple queues and this would seem to be just the thing.

Unfortunately, the jupyterhub-single instance on the compute nodes can't be found when I use wrapspawner but it works fine with batchspawner only. I would be happy to upload parts of my config files if that would be useful. I checked on the compute node and the server is running waiting for a connection.

When wrapspawner is enabled I get:

[snip]
Job 109152.xena.xena.alliance.unm.edu still pending
[D 2020-01-06 16:16:48.588 JupyterHub batchspawner:214] Spawner querying job: sudo -E -u mfricke qstat -x 109152.xena.xena.alliance.unm.edu
[I 2020-01-06 16:16:48.632 JupyterHub batchspawner:330] Notebook server job 109152.xena.xena.alliance.unm.edu started at xena22:58735
[D 2020-01-06 16:16:48.641 JupyterHub spawner:851] Polling subprocess every 30s
[D 2020-01-06 16:16:53.574 JupyterHub batchspawner:214] Spawner querying job: sudo -E -u mfricke qstat -x 109152.xena.xena.alliance.unm.edu
[W 2020-01-06 16:16:53.616 JupyterHub base:744] User mfricke is slow to become responsive (timeout=10)
[D 2020-01-06 16:16:53.617 JupyterHub base:746] Expecting server for mfricke at: http://xena22:58735/user/mfricke/
[I 2020-01-06 16:16:53.631 JupyterHub log:158] 302 POST /hub/spawn -> /user/mfricke/ (mfricke@::ffff:129.24.246.13) 10067.56ms
[I 2020-01-06 16:16:53.646 JupyterHub log:158] 302 GET /user/mfricke/ -> /hub/user/mfricke/ (@::ffff:129.24.246.13) 0.86ms
[D 2020-01-06 16:16:53.663 JupyterHub base:1018] Waiting for mfricke pending spawn
[I 2020-01-06 16:16:58.306 JupyterHub log:158] 200 GET /hub/api (@172.16.1.72) 1.34ms
[D 2020-01-06 16:17:00.941 JupyterHub proxy:686] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
16:17:00.944 [ConfigProxy] info: 200 GET /api/routes
[I 2020-01-06 16:17:00.944 JupyterHub proxy:301] Checking routes
[I 2020-01-06 16:17:03.664 JupyterHub base:1022] Pending spawn for mfricke didn't finish in 10.0 seconds
[I 2020-01-06 16:17:03.664 JupyterHub base:1028] mfricke is pending spawn
[I 2020-01-06 16:17:03.667 JupyterHub log:158] 200 GET /hub/user/mfricke/ (mfricke@::ffff:129.24.246.13) 10010.10ms
[D 2020-01-06 16:17:18.643 JupyterHub batchspawner:214] Spawner querying job: sudo -E -u mfricke qstat -x 109152.xena.xena.alliance.unm.edu
[D 2020-01-06 16:17:48.643 JupyterHub batchspawner:214] Spawner querying job: sudo -E -u mfricke qstat -x 109152.xena.xena.alliance.unm.edu
[D 2020-01-06 16:18:18.643 JupyterHub batchspawner:214] Spawner querying job: sudo -E -u mfricke qstat -x 109152.xena.xena.alliance.unm.edu
[D 2020-01-06 16:18:48.644 JupyterHub batchspawner:214] Spawner querying job: sudo -E -u mfricke qstat -x 109152.xena.xena.alliance.unm.edu
[W 2020-01-06 16:18:49.127 JupyterHub user:510] mfricke's server never showed up at http://xena22:58735/user/mfricke/ after 120 seconds. Giving up
[D 2020-01-06 16:18:49.128 JupyterHub batchspawner:214] Spawner querying job: sudo -E -u mfricke qstat -x 109152.xena.xena.alliance.unm.edu
[I 2020-01-06 16:18:49.169 JupyterHub batchspawner:342] Stopping server job 109152.xena.xena.alliance.unm.edu
[I 2020-01-06 16:18:49.170 JupyterHub batchspawner:233] Cancelling job 109152.xena.xena.alliance.unm.edu: sudo -E -u mfricke qdel 109152.xena.xena.alliance.unm.edu
[snip]

but with batchspawner I get

[snip]
Notebook server job 109153.xena.xena.alliance.unm.edu started at xena22:58732
[D 2020-01-06 16:21:04.345 JupyterHub spawner:851] Polling subprocess every 30s
[D 2020-01-06 16:21:07.914 JupyterHub batchspawner:214] Spawner querying job: sudo -E -u mfricke qstat -x 109153.xena.xena.alliance.unm.edu
[W 2020-01-06 16:21:07.960 JupyterHub base:744] User mfricke is slow to become responsive (timeout=10)
[D 2020-01-06 16:21:07.960 JupyterHub base:746] Expecting server for mfricke at: http://xena22:58732/user/mfricke/
[I 2020-01-06 16:21:07.960 JupyterHub base:1066] mfricke is pending spawn
[I 2020-01-06 16:21:08.072 JupyterHub log:158] 200 GET /hub/user/mfricke/ (mfricke@::ffff:129.24.246.13) 10178.01ms
[D 2020-01-06 16:21:08.327 JupyterHub log:158] 200 GET /favicon.ico (@::ffff:129.24.246.13) 2.35ms
[I 2020-01-06 16:21:16.409 JupyterHub log:158] 200 GET /hub/api (@172.16.1.72) 1.09ms
[D 2020-01-06 16:21:20.957 JupyterHub utils:188] Server at http://xena22:58732/user/mfricke/ responded with 302
[D 2020-01-06 16:21:20.957 JupyterHub _version:53] jupyterhub and jupyterhub-singleuser both on version 0.9.6
[I 2020-01-06 16:21:20.957 JupyterHub base:638] User mfricke took 23.047 seconds to start
[I 2020-01-06 16:21:20.958 JupyterHub proxy:242] Adding user mfricke to proxy /user/mfricke/ => http://xena22:58732
[D 2020-01-06 16:21:20.958 JupyterHub proxy:686] Proxy: Fetching POST http://127.0.0.1:8001/api/routes/user/mfricke
16:21:20.959 [ConfigProxy] info: Adding route /user/mfricke -> http://xena22:58732
16:21:20.960 [ConfigProxy] info: 201 POST /api/routes/user/mfricke
[I 2020-01-06 16:21:20.961 JupyterHub users:533] Server mfricke is ready
[I 2020-01-06 16:21:20.961 JupyterHub log:158] 200 GET /hub/api/users/mfricke/server/progress (mfricke@::ffff:129.24.246.13) 12656.60ms
[D 2020-01-06 16:21:22.143 JupyterHub batchspawner:214] Spawner querying job: sudo -E -u mfricke qstat -x 109153.xena.xena.alliance.unm.edu
[snip]

Thanks for any help or advice you can provide.

All the best,

Matthew

Hi,

I'm not sure if this should be a batchspawner issue or a profilespawner issue. I have filed this batchspawner issue: jupyterhub/batchspawner#127

I have a (Slurm) BatchSpawner wrapped in a ProfileSpawner, for a typical academic compute cluster use case -- users can either start their notebook processes on the head node, or on a compute node via Slurm.

When a batchspawner worker starts up on the remote worker node, it calls an API on the hub, which goes to a batchspawner API handler. The API handler retrieves the user.spawner object, and tries to set its .current_port attribute. It is expecting that spawner object to be a BatchSpawner, but in my setup it is actually a ProfileSpawner, so it sets the .current_port on an object other than the one that the main BatchSpawner class is polling, and it eventually thinks that the remote process timed out.

I added the following to the batchspawner API handler, here https://github.com/jupyterhub/batchspawner/blob/master/batchspawner/api.py#L12 , and it works, but is ugly:

        spawner = user.spawner
	try:
            from wrapspawner import WrapSpawner
            if isinstance(spawner, WrapSpawner):
		spawner = spawner.child_spawner
	except:
            pass
	spawner.current_port = port

Now, it seems like a cleaner solution might be for WrapSpawner to propagate some setattr requests to its child_spawner?

Open to suggestions for a clean fix. Thanks!

I'm working on using wrapspawner and named servers in our deployment. That is, each wrapped spawner's server would get a name. I've subclassed wrapspawner and I find that I need to add this to my construct_child() method after super.construct_child() in order for the server name to get into JUPYTERHUB_OAUTH_CALLBACK_URL

self.child_spawner.orm_spawner = self.orm_spawner

It works, but what's the right way to make this work...?

I have extended the ProfilesSpawner class to create a Spawner class that will add profiles for all docker images on the local machine that have a jupyterhub tag. The biggest issue I currently have is that it will only build the profiles form once, in other words, I have to restart jupyterhub in order to get new docker images to show up. Is there a lifecycle hook that would be a better place for me to dynamically add profiles?

Here is a link to my custom class: https://github.com/verypossible/wrapspawner/blob/feature/docker-profile-spawner/wrapspawner/wrapspawner.py#L218

I'm currently using ProfilesSpawner in combination with the batchspawner to offer our HPC cluster's users different BatchSpawner profiles.

Is there any way how I can make the list of profiles offered to the user dependent on the user's group?

If not, I would appreciate a pointer towards where I would need to implement this.

I am browsing the (nicely written!) documentation of this spawner, and just have a tiny suggestion: adding a screenshot of the User Interface resulting from the given configuration sample.
Thanks!

Hi, I am trying to provide two profiles (ProfilesSpawner and options form). Both forms are displayed but when spawning I get this:

Unhandled error starting  server: maximum recursion depth exceeded
[D 2020-02-10 10:59:09.089 JupyterHub user:739] Stopping 
[E 2020-02-10 10:59:09.114 JupyterHub user:652] Failed to cleanup  server that failed to start
    Traceback (most recent call last):
      File "/py36/lib/python3.6/site-packages/jupyterhub/user.py", line 646, in spawn
        await self.stop(spawner.name)
      File "/py36/lib/python3.6/site-packages/jupyterhub/user.py", line 743, in stop
        status = await spawner.poll()
      File "/site-packages/wrapspawner/wrapspawner.py", line 135, in poll
        return self.child_spawner.poll()
      File "/py36/lib/python3.6/site-packages/wrapspawner/wrapspawner.py", line 135, in poll
        return self.child_spawner.poll()
      File "/py36/lib/python3.6/site-packages/wrapspawner/wrapspawner.py", line 135, in poll
        return self.child_spawner.poll()
      [Previous line repeated 984 more times]
      File "/py36/lib/python3.6/site-packages/wrapspawner/wrapspawner.py", line 134, in poll
        if self.child_spawner:
      File "/py36/lib/python3.6/site-packages/traitlets/traitlets.py", line 556, in __get__
        return self.get(obj, cls)
    RecursionError: maximum recursion depth exceeded
    
[E 2020-02-10 10:59:09.137 JupyterHub pages:248] Failed to spawn single-user server with form
    Traceback (most recent call last):
      File "/py36/lib/python3.6/site-packages/jupyterhub/handlers/pages.py", line 245, in post
        await self.spawn_single_user(user, server_name=server_name, options=options)
      File "/py36/lib/python3.6/site-packages/jupyterhub/handlers/base.py", line 939, in spawn_single_user
        timedelta(seconds=self.slow_spawn_timeout), finish_spawn_future
      File "/py36/lib/python3.6/site-packages/jupyterhub/handlers/base.py", line 852, in finish_user_spawn
        await spawn_future
      File "/py36/lib/python3.6/site-packages/jupyterhub/user.py", line 656, in spawn
        raise e
      File "/py36/lib/python3.6/site-packages/jupyterhub/user.py", line 557, in spawn
        f = maybe_future(spawner.start())
      File "/py36/lib/python3.6/site-packages/wrapspawner/wrapspawner.py", line 125, in start
        return self.child_spawner.start()
      File "/py36/lib/python3.6/site-packages/wrapspawner/wrapspawner.py", line 125, in start
        return self.child_spawner.start()
      File "/py36/lib/python3.6/site-packages/wrapspawner/wrapspawner.py", line 125, in start
        return self.child_spawner.start()
      [Previous line repeated 985 more times]
      File "/py36/lib/python3.6/site-packages/wrapspawner/wrapspawner.py", line 123, in start
        if not self.child_spawner:
      File "/py36/lib/python3.6/site-packages/traitlets/traitlets.py", line 556, in __get__
        return self.get(obj, cls)
    RecursionError: maximum recursion depth exceeded

Any idea?

Hi,
I have tried the configuration as described on
https://github.com/jupyterhub/batchspawner/
but it won't work.

The configurations for the LocalProcessSpawner and the TorqueSpawner works. But when trying to use the wrapspawner, the LocalProcessSpawner works, while the TorqueSpawner cannot connect to the started server on a node.

I add some output of the jupyterhub logfile, the first part is from the direct TorqueSpawner, the second from the wrapspawner - TorqueSpawner try.

TorqueSpawner

[D 2019-12-17 16:35:00.359 JupyterHub user:240] Creating <class 'batchspawner.batchspawner.TorqueSpawner'> for user1:test2
[D 2019-12-17 16:35:00.363 JupyterHub pages:165] Triggering spawn with default options for user1:test2
[D 2019-12-17 16:35:00.364 JupyterHub base:780] Initiating spawn for user1:test2
[D 2019-12-17 16:35:00.364 JupyterHub base:787] 0/100 concurrent spawns
[D 2019-12-17 16:35:00.364 JupyterHub base:792] 1/4 active servers
[D 2019-12-17 16:35:00.397 JupyterHub user:542] Calling Spawner.start for user1:test2
[I 2019-12-17 16:35:00.398 JupyterHub batchspawner:188] Spawner submitting job using sudo -E -u user1 qsub
[I 2019-12-17 16:35:00.398 JupyterHub batchspawner:189] Spawner submitted script:
#!/bin/sh
#PBS -q batch@cluster-head
#PBS -l walltime=168:00:00
#PBS -l nodes=1:ppn=20
#PBS -l mem=120gb
#PBS -N jupyterhub
#PBS -v PATH,LD_LIBRARY_PATH,LANG,MKL_NUM_THREADS,JUPYTERHUB_API_TOKEN,JPY_API_TOKEN,JUPYTERHUB_CLIENT_ID,JUPYTERHUB_HOST,JUPYTERHUB_OAUTH_CALLBACK_URL,JUPYTERHUB_USER,JUPYTERHUB_SERVER_NAME,JUPYTERHUB_API_URL,JUPYTERHUB_ACTIVITY_URL,JUPYTERHUB_BASE_URL,JUPYTERHUB_SERVICE_PREFIX
module load python/3.6.4
jupyterhub-singleuser --ip=0.0.0.0 --port=33595

[I 2019-12-17 16:35:00.452 JupyterHub batchspawner:192] Job submitted. cmd: sudo -E -u user1 qsub output: 669540.cluster-head
[D 2019-12-17 16:35:00.453 JupyterHub batchspawner:214] Spawner querying job: sudo -E -u user1 qstat -x 669540.cluster-head
[D 2019-12-17 16:35:00.615 JupyterHub batchspawner:316] Job 669540.cluster-head still pending
[D 2019-12-17 16:35:01.118 JupyterHub batchspawner:214] Spawner querying job: sudo -E -u user1 qstat -x 669540.cluster-head
[I 2019-12-17 16:35:01.165 JupyterHub batchspawner:330] Notebook server job 669540.cluster-head started at cluster-node3:33595
[D 2019-12-17 16:35:01.173 JupyterHub spawner:1084] Polling subprocess every 30s
[I 2019-12-17 16:35:01.365 JupyterHub log:174] 302 GET /hub/spawn/user1/test2 -> /hub/spawn-pending/user1/test2 (user1@::ffff:10.50.10.37) 1016.25ms
[I 2019-12-17 16:35:01.391 JupyterHub pages:303] user1:test2 is pending spawn
[I 2019-12-17 16:35:01.393 JupyterHub log:174] 200 GET /hub/spawn-pending/user1/test2 (user1@::ffff:10.50.10.37) 11.48ms
[I 2019-12-17 16:35:05.053 JupyterHub log:174] 200 GET /hub/api (@192.168.0.3) 2.06ms
[D 2019-12-17 16:35:05.075 JupyterHub users:708] Activity for user user1: 2019-12-17T15:35:03.431151Z
[D 2019-12-17 16:35:05.076 JupyterHub users:729] Activity on server user1/test2: 2019-12-17T15:35:03.431151Z
[I 2019-12-17 16:35:05.081 JupyterHub log:174] 200 POST /hub/api/users/user1/activity ([email protected]) 19.08ms
[D 2019-12-17 16:35:08.674 JupyterHub utils:218] Server at http://cluster-node3:33595/user/user1/test2/ responded with 302
[D 2019-12-17 16:35:08.675 JupyterHub _version:60] jupyterhub and jupyterhub-singleuser both on version 1.0.0
[I 2019-12-17 16:35:08.675 JupyterHub base:810] User user1:test2 took 8.311 seconds to start
[I 2019-12-17 16:35:08.675 JupyterHub proxy:261] Adding user user1 to proxy /user/user1/test2/ => http://cluster-node3:33595
[D 2019-12-17 16:35:08.676 JupyterHub proxy:765] Proxy: Fetching POST http://127.0.0.1:8001/api/routes/user/user1/test2
[I 2019-12-17 16:35:08.682 JupyterHub users:606] Server user1:test2 is ready
[I 2019-12-17 16:35:08.683 JupyterHub log:174] 200 GET /hub/api/users/user1/servers/test2/progress (user1@::ffff:10.50.10.37) 7202.35ms
[I 2019-12-17 16:35:08.725 JupyterHub log:174] 302 GET /hub/spawn-pending/user1/test2 -> /user/user1/test2 (user1@::ffff:10.50.10.37) 9.94ms
[D 2019-12-17 16:35:08.801 JupyterHub provider:414] Validating client id jupyterhub-user-user1-test2
[D 2019-12-17 16:35:08.804 JupyterHub provider:492] validate_redirect_uri: client_id=jupyterhub-user-user1-test2, redirect_uri=/user/user1/test2/oauth_callback
[D 2019-12-17 16:35:08.808 JupyterHub auth:222] Skipping oauth confirmation for <User(user1 2/3 running)> accessing Server at /user/user1/test2/
[D 2019-12-17 16:35:08.809 JupyterHub provider:414] Validating client id jupyterhub-user-user1-test2
[D 2019-12-17 16:35:08.811 JupyterHub provider:492] validate_redirect_uri: client_id=jupyterhub-user-user1-test2, redirect_uri=/user/user1/test2/oauth_callback
[D 2019-12-17 16:35:08.814 JupyterHub provider:241] Saving authorization code jupyterhub-user-user1-test2, VVr..., (), {}
[I 2019-12-17 16:35:08.824 JupyterHub log:174] 302 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-user1-test2&redirect_uri=%2Fuser%2Fuser1%2Ftest2%2Foauth_callback&response_type=code&state=[secret] -> /user/user1/test2/oauth_callback?code=[secret]&state=[secret] (user1@::ffff:10.50.10.37) 33.45ms
[D 2019-12-17 16:35:08.856 JupyterHub provider:60] authenticate_client <oauthlib.Request SANITIZED>
[D 2019-12-17 16:35:08.869 JupyterHub provider:119] confirm_redirect_uri: client_id=jupyterhub-user-user1-test2, redirect_uri=/user/user1/test2/oauth_callback
[D 2019-12-17 16:35:08.869 JupyterHub provider:339] Saving bearer token {'access_token': 'REDACTED', 'expires_in': 3600, 'token_type': 'Bearer', 'scope': 'identify', 'refresh_token': 'REDACTED'}
[D 2019-12-17 16:35:08.877 JupyterHub provider:194] Deleting oauth code VVr... for jupyterhub-user-user1-test2
[I 2019-12-17 16:35:08.885 JupyterHub log:174] 200 POST /hub/api/oauth2/token ([email protected]) 37.53ms
[I 2019-12-17 16:35:08.913 JupyterHub log:174] 200 GET /hub/api/authorizations/token/[secret] ([email protected]) 22.52ms
[D 2019-12-17 16:35:09.194 JupyterHub log:174] 200 GET /hub/logo (@::ffff:10.50.10.37) 2.20ms
[W 2019-12-17 16:35:10.136 JupyterHub web:1618] 400 POST /hub/api/users/user1/activity (192.168.0.254): No such server 'Head Node' for user user1
[W 2019-12-17 16:35:10.137 JupyterHub log:174] 400 POST /hub/api/users/user1/activity ([email protected]) 15.07ms
[W 2019-12-17 16:35:10.860 JupyterHub web:1618] 400 POST /hub/api/users/user1/activity (192.168.0.254): No such server 'Head Node' for user user1
[W 2019-12-17 16:35:10.861 JupyterHub log:174] 400 POST /hub/api/users/user1/activity ([email protected]) 14.92ms
[W 2019-12-17 16:35:12.832 JupyterHub web:1618] 400 POST /hub/api/users/user1/activity (192.168.0.254): No such server 'Head Node' for user user1
[W 2019-12-17 16:35:12.833 JupyterHub log:174] 400 POST /hub/api/users/user1/activity ([email protected]) 14.32ms
[W 2019-12-17 16:35:15.929 JupyterHub web:1618] 400 POST /hub/api/users/user1/activity (192.168.0.254): No such server 'Head Node' for user user1
[W 2019-12-17 16:35:15.931 JupyterHub log:174] 400 POST /hub/api/users/user1/activity ([email protected]) 14.49ms

Wrapspawner -> TorqueSpawner

[D 2019-12-17 17:32:11.603 JupyterHub user:240] Creating <class 'wrapspawner.wrapspawner.ProfilesSpawner'> for user1:test2
[D 2019-12-17 17:32:11.635 JupyterHub pages:158] Serving options form for user1:test2
[I 2019-12-17 17:32:11.637 JupyterHub log:174] 200 GET /hub/spawn/user1/test2 (user1@::ffff:10.50.10.37) 43.30ms
[D 2019-12-17 17:32:13.678 JupyterHub base:780] Initiating spawn for user1:test2
[D 2019-12-17 17:32:13.679 JupyterHub base:787] 0/100 concurrent spawns
[D 2019-12-17 17:32:13.679 JupyterHub base:792] 0/4 active servers
[D 2019-12-17 17:32:13.715 JupyterHub user:542] Calling Spawner.start for user1:test2
[I 2019-12-17 17:32:13.744 JupyterHub batchspawner:188] Spawner submitting job using sudo -E -u user1 qsub
[I 2019-12-17 17:32:13.744 JupyterHub batchspawner:189] Spawner submitted script:
#!/bin/sh
#PBS -q batch@cluster-head
#PBS -l walltime=168:00:00
#PBS -l nodes=1:ppn=20
#PBS -l mem=120gb
#PBS -N jupyterhub
#PBS -v PATH,LD_LIBRARY_PATH,LANG,MKL_NUM_THREADS,JUPYTERHUB_API_TOKEN,JPY_API_TOKEN,JUPYTERHUB_CLIENT_ID,JUPYTERHUB_HOST,JUPYTERHUB_OAUTH_CALLBACK_URL,JUPYTERHUB_USER,JUPYTERHUB_SERVER_NAME,JUPYTERHUB_API_URL,JUPYTERHUB_ACTIVITY_URL,JUPYTERHUB_BASE_URL,JUPYTERHUB_SERVICE_PREFIX
#PBS -j oe
#PBS -o /dev/null
module load python/3.6.4
jupyterhub-singleuser --port=45951

[I 2019-12-17 17:32:14.126 JupyterHub batchspawner:192] Job submitted. cmd: sudo -E -u user1 qsub output: 669551.cluster-head
[D 2019-12-17 17:32:14.128 JupyterHub batchspawner:214] Spawner querying job: sudo -E -u user1 qstat -x 669551.cluster-head
[D 2019-12-17 17:32:14.266 JupyterHub batchspawner:316] Job 669551.cluster-head still pending
[D 2019-12-17 17:32:14.768 JupyterHub batchspawner:214] Spawner querying job: sudo -E -u user1 qstat -x 669551.cluster-head
[D 2019-12-17 17:32:14.817 JupyterHub batchspawner:316] Job 669551.cluster-head still pending
[D 2019-12-17 17:32:15.319 JupyterHub batchspawner:214] Spawner querying job: sudo -E -u user1 qstat -x 669551.cluster-head
[I 2019-12-17 17:32:15.371 JupyterHub batchspawner:330] Notebook server job 669551.cluster-head started at cluster-node4:45951
[D 2019-12-17 17:32:15.378 JupyterHub spawner:1084] Polling subprocess every 30s
[I 2019-12-17 17:32:18.392 JupyterHub log:174] 200 GET /hub/api (@192.168.0.4) 1.60ms
[D 2019-12-17 17:32:18.415 JupyterHub users:708] Activity for user user1: 2019-12-17T16:32:16.962407Z
[D 2019-12-17 17:32:18.415 JupyterHub users:729] Activity on server user1/test2: 2019-12-17T16:32:16.962407Z
[I 2019-12-17 17:32:18.421 JupyterHub log:174] 200 POST /hub/api/users/user1/activity ([email protected]) 20.17ms
[D 2019-12-17 17:32:23.682 JupyterHub batchspawner:214] Spawner querying job: sudo -E -u user1 qstat -x 669551.cluster-head
[W 2019-12-17 17:32:23.730 JupyterHub base:932] User user1:test2 is slow to become responsive (timeout=10)
[D 2019-12-17 17:32:23.730 JupyterHub base:937] Expecting server for user1:test2 at: http://cluster-node4:45951/user/user1/test2/
[I 2019-12-17 17:32:23.741 JupyterHub log:174] 302 POST /hub/spawn/user1/test2 -> /hub/spawn-pending/user1/test2 (user1@::ffff:10.50.10.37) 10071.92ms
[I 2019-12-17 17:32:23.779 JupyterHub pages:303] user1:test2 is pending spawn
[I 2019-12-17 17:32:23.781 JupyterHub log:174] 200 GET /hub/spawn-pending/user1/test2 (user1@::ffff:10.50.10.37) 9.90ms
[D 2019-12-17 17:32:45.381 JupyterHub batchspawner:214] Spawner querying job: sudo -E -u user1 qstat -x 669551.cluster-head
[D 2019-12-17 17:33:15.383 JupyterHub batchspawner:214] Spawner querying job: sudo -E -u user1 qstat -x 669551.cluster-head

The JupyterHub 3.0.0 beta is out. Access to the db attribute on Authenticators and Spawners triggers a warning there:

[W 2022-08-12 16:09:44.804 JupyterHub spawner:170]
    The shared database session at Spawner.db is deprecated, and will be removed.
    Please manage your own database and connections.
    Contact JupyterHub at https://github.com/jupyterhub/jupyterhub/issues/3700
    if you have questions or ideas about direct database needs for your Spawner.

Here's a link to the issue for more info: jupyterhub/jupyterhub#3700

In wrapspawner the db attribute is accessed when creating a child spawner. If this attribute goes away, it doesn't need to be copied, so I think this is not that big a deal to address here.

Hi,

I'm trying to update my Jupyterhub installation again, and seemingly facing new incarnations of the same bugs I've encountered previously.

Long story short, I have a python3.9 setup with default pip versions of most things, specifically,

jupyterhub                3.1.0
jupyterlab                3.5.2
jupyter_core              5.1.2
traitlets                 5.8.0
batchspawner              1.2.0        (+ manually-applied patch)
wrapspawner               1.0.2.dev0      (git)

and I'm getting the common behavior that batchspawner (Slurm, in my case) works, but once wrapped in ProfilesSpawner, it fails; the user's jupyter server starts up, but never seems to contact the jupyterhub server.

I have read through and tried the things mentioned in #54, #41, #35, and my own ancient #24, but no dice.

I'll give some details on what I'm seeing below, but I guess my question is are there known versions of these packages that work together, and would it be possible to please document them in the README?

Brief Jhub logs:

Jan 06 12:10:12 mn003 start.sh[20715]: [I 2023-01-06 12:10:12.834 JupyterHub log:186] 200 POST /hub/api/batchspawner ([email protected]) 23.27ms
Jan 06 12:10:13 mn003 start.sh[20715]: [I 2023-01-06 12:10:13.022 JupyterHub batchspawner:463] Notebook server job 354117 started at cn002:58213
...
Jan 06 12:10:18 mn003 start.sh[20715]: [W 2023-01-06 12:10:18.810 JupyterHub base:1104] User dlang is slow to become responsive (timeout=10)
Jan 06 12:10:18 mn003 start.sh[20715]: [D 2023-01-06 12:10:18.810 JupyterHub base:1109] Expecting server for dlang at: http://mn003:0/user/dlang/
...
Jan 06 12:12:02 mn003 start.sh[20715]: [W 2023-01-06 12:12:02.744 JupyterHub user:881] dlang's server never showed up at http://mn003:0/user/dlang/ after 120 seconds. Giving up.
Jan 06 12:12:02 mn003 start.sh[20715]:
Jan 06 12:12:02 mn003 start.sh[20715]:     Common causes of this timeout, and debugging tips:
Jan 06 12:12:02 mn003 start.sh[20715]:
Jan 06 12:12:02 mn003 start.sh[20715]:     1. The server didn't finish starting,
Jan 06 12:12:02 mn003 start.sh[20715]:        or it crashed due to a configuration issue.
Jan 06 12:12:02 mn003 start.sh[20715]:        Check the single-user server's logs for hints at what needs fixing.
Jan 06 12:12:02 mn003 start.sh[20715]:     2. The server started, but is not accessible at the specified URL.
Jan 06 12:12:02 mn003 start.sh[20715]:        This may be a configuration issue specific to your chosen Spawner.
Jan 06 12:12:02 mn003 start.sh[20715]:        Check the single-user server logs and resource to make sure the URL
Jan 06 12:12:02 mn003 start.sh[20715]:        is correct and accessible from the Hub.
Jan 06 12:12:02 mn003 start.sh[20715]:     3. (unlikely) Everything is working, but the server took too long to respond.
Jan 06 12:12:02 mn003 start.sh[20715]:        To fix: increase `Spawner.http_timeout` configuration
Jan 06 12:12:02 mn003 start.sh[20715]:        to a number of seconds that is enough for servers to become responsive.
Jan 06 12:12:02 mn003 start.sh[20715]:
Jan 06 12:12:02 mn003 start.sh[20715]: [D 2023-01-06 12:12:02.745 JupyterHub user:930] Stopping dlang

This log message seems to point directly at the issue:

Expecting server for dlang at: http://mn003:0/user/dlang/

where mn003 is the name of my front-end (jhub) server; it looks like the .server attribute isn't getting set, which sounds deeply familiar. On the second line, it looks like batchspawner is reporting the URL of the server (cn002:58213).

Thanks for any suggestions!

jupyterhub/batchspawner#16

Wrapspawner is working great for me but I'm trying to get our hub to run as a non-root user using sudospawner. I've been messing around with the two spawners but am a bit stuck on how to setup the hub configuration to use one alongside the other.

I got this error when I use the ProfilesSpawner with the following config:

c.JupyterHub.spawner_class = 'wrapspawner.ProfilesSpawner'
c.ProfilesSpawner.profiles = [
       ( "Host process", 'local', 'jupyterhub.spawner.LocalProcessSpawner', {'ip':'0.0.0.0'} ),
       ('Docker Python 3', 'singleuser', 'dockerspawner.SystemUserSpawner', dict(image="jupyterhub/singleuser")),
       ('Docker Python 3 Scipy', 'scipy-notebook', 'dockerspawner.SystemUserSpawner', dict(image="jupyter/scipy-notebook")),
 ] 

The error:

 File "/py36/lib/python3.6/site-packages/wrapspawner/wrapspawner.py", line 102, in load_state
        self.construct_child()
      File "/py36/lib/python3.6/site-packages/wrapspawner/wrapspawner.py", line 213, in construct_child
        super().construct_child()
      File "/py36/lib/python3.6/site-packages/wrapspawner/wrapspawner.py", line 79, in construct_child
        **self.child_config
    TypeError: 'str' object is not callable

I do need to go back and verify this again, but recording it as issue so don't forget until I can.

What I observed a while back is that JupyterHub will use start_timeout from spawner instance for a specific timeout. When using ProfilesSpawner this is an issue as it ends up using start_timeout from the Spawner base class, rather than what start_timeout may be defined in the target spawner chosen.

The result of this was that couldn't use start_timeout in the profile or by doing:

c.KubeSpawner.start_timeout = 120

It was necessary to use:

c.Spawner.start_timeout = 120

If observation is right, ProfilesSpawner should perhaps override start_timeout as a property and get start_timeout from the target spawner which was chosen.

The new .progress() should be passed through to the child spawners. Does anyone have a hint on how to do this properly, similar to yield from but also sufficiently backwards compatible? I can make the actual PR and test.

Proper testing would start to be more and more important, but I don't know if I have the ability to set that up easily.

I discussed the multiple profile selection tools (ProfileSpawner, the one in KubeSpawner, etc) with @minrk at the Oslo JupyterHub sprint. His proposal was that profiles could be merged directly into JupyterHub since they have been proven by now. This could directly select the spawner class, and avoid all of the proxying problems we have here. Most of the other issues in this repo could be closed, too.

This would need at least the features:

• Select a spawner
• Support options forms

To me the biggest unknown is "how do options forms work", since the options form is a property of the spawner class but there can be multiple spawner classes and one isn't chosen until after the options form is rendered and an option is selected.

What do you all think about this?

• Who is for and against?
• What functions in wrapspawner would still be needed? For what functionality would wrapspawer still be needed? Could it be removed?
• What configurability would be needed in JupyterHub, e.g. dynamic profile lists, etc.

Bug description

Hi, I would like to use this ProfilesSpawner with https://github.com/jupyterhub/zero-to-jupyterhub-k8s. z2jh comes with an handy configuration to spawn pods in Kubernetes.

Is there a way to forward the configuration to a profile of this spawner ?

How to reproduce

My idea was to forward the config of KubeSpawner directly to the ProfilesSpawner however I get an error. Here is some of the config that I tried and would love to work.

c.JupyterHub.spawner_class = 'wrapspawner.ProfilesSpawner'

c.ProfilesSpawner.profiles = [
    ('Kube', 'singleuser', 'kubespawner.KubeSpawner', c.KubeSpawner),
    # Some other spawner
]

And here is the error :

[I 2024-01-03 13:52:46.538 JupyterHub provider:659] Creating oauth client jupyterhub-user-my_user
                'tolerations': [{'effect': 'NoSchedule',                                                                                                                                                                                                                                                                    
                               'key': 'hub.jupyter.org/dedicated', 
                               'operator': 'Equal', 
                               'toleration_seconds': None,
                               'value': 'user'}, 
                              {'effect': 'NoSchedule', 
                               'key': 'hub.jupyter.org_dedicated', 
                               'operator': 'Equal', 
                               'toleration_seconds': None,
                               'value': 'user'}], 
              'topology_spread_constraints': None,
              'volumes': [{'aws_elastic_block_store': None,
                           'azure_disk': None,
                           'azure_file': None,
                           'cephfs': None,
                           'cinder': None,
                           'config_map': None,
                           'csi': None,
                           'downward_api': None,
                           'empty_dir': None,
                           'ephemeral': None,
                           'fc': None,
                           'flex_volume': None,
                           'flocker': None,
                           'gce_persistent_disk': None,
                           'git_repo': None,
                           'glusterfs': None,
                           'host_path': None,
                           'iscsi': None,
                           'name': 'volume-my-user', 
                           'nfs': None,
                           'persistent_volume_claim': {'claimName': 'claim-my-user'}, 
                           'photon_persistent_disk': None,
                           'portworx_volume': None,
                           'projected': None,
                           'quobyte': None,
                           'rbd': None,
                           'scale_io': None,
                           'secret': None,
                           'storageos': None,
                           'vsphere_volume': None}]},
     'status': None}
    Traceback (most recent call last): 
      File "/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py", line 2482, in _make_create_pod_request 
        await asyncio.wait_for(
      File "/usr/local/lib/python3.11/asyncio/tasks.py", line 479, in wait_for 
        return fut.result() 
               ^^^^^^^^^^^^ 
      File "/usr/local/lib/python3.11/site-packages/kubernetes_asyncio/client/api_client.py", line 192, in __call_api 
        raise e 
      File "/usr/local/lib/python3.11/site-packages/kubernetes_asyncio/client/api_client.py", line 185, in __call_api 
        response_data = await self.request(
                        ^^^^^^^^^^^^^^^^^^^ 
      File "/usr/local/lib/python3.11/site-packages/kubernetes_asyncio/client/rest.py", line 230, in POST 
        return (await self.request("POST", url,                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                                                                           
      File "/usr/local/lib/python3.11/site-packages/kubernetes_asyncio/client/rest.py", line 187, in request 
        raise ApiException(http_resp=r)
    kubernetes_asyncio.client.exceptions.ApiException: (422)
    Reason: error 
    HTTP response headers: <CIMultiDictProxy('Audit-Id': '29fe0728-ee34-4464-abcb-383eb2704e94', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '8d66a213-297f-44ef-9dd7-f701c05a33e7', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'e09e092b-d2b8-4aa1-913d-7f6d6bbda951', 'Date': 'Wed, 03 Jan 2024 13:52:46 GMT', 'Content-Length': '397')>                                                                                                                                                                                                                                                
    HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Pod \"jupyter-my-user\" is invalid: spec.containers[0].ports[0].containerPort: Required value","reason":"Invalid","details":{"name":"jupyter-my-user","kind":"Pod","causes":[{"reason":"FieldValueRequired","message":"Required value","field":"spec.containers[0].ports[0].containerPort"}]},"code":422}     

I am not very familiar with traitlets and how they work. I guess that I am missing something there.
From what I understood, some default configurations are not loaded this way.
Any help would be welcome, it could be great to be able to combine easily z2jh and this project.

\cc @consideRatio

Expected behaviour

I would expect to pass a valid configuration.

Actual behaviour

It currently fails.

Hi, I'd like to allow the user to change some of the batch spawn parameters, e.g. "memory between 1GB and 24GB". Is this possible with wrapspawner?

It looks like I never was made an admin on this repository when we split it from Batchspawner. Only noticing now that I want access to the settings tab to investigate Actions workflows for making releases.

@willingc it looks like you originally created this repo?

I was a bit late to the party to test #27; I have a WrapSpawner implementation that has a custom cmd in the child profiles, and they are now getting overridden by c.Spawner.cmd. I have a profilespawner-like setup and the spawn command is different on different profiles.

Could we modify the directional link logic so that traits set and explicitly passed to the child spawner constructor aren't linked maybe? I can experiment with it next week.

We are testing out wrapspawner.ProfilesSpawner against the upcoming JupyterHub 0.8.0, which I think will be released "in the next week or two." An odd thing happens with ProfilesSpawner, in particular using any particular Spawner like it (even LocalProcessSpawner) fails with

ValueError: oauth_client_id cannot be empty.

In 0.8.0 authentication of services with the hub is handled by OAuth, but my understanding is that there aren't any such services being used here by wrapspawner.

At this gist are a Dockerfile and configuration file that can be used to reproduce the problem and some instructions. I've had 2 colleagues test it out and they can get the failure. Note that we are building this Dockerfile against our fork of wrapspawner for now that has a "fix" for what I mentioned in #10, (I am not convinced that was the right fix, so I haven't done a PR with it).

See the Dockerfile for the test login credentials, but you could also change the configuration so that passwordless/create users happens in LocalProcessSpawner I guess.

Hi, apologies for the double post, I moved this issue from jupyterhub/batchspawner#194.

We currently cannot spawn any workers with ProfilesSpawner enabled.
The worker starts normally but the jupyterhub directly kills it.

Logs of the worker:

+ batchspawner-singleuser jupyterhub-singleuser --ip=0.0.0.0 --NotebookApp.default_url=/lab
[I 2020-11-05 13:22:43.763 SingleUserNotebookApp manager:81] [nb_conda_kernels] enabled, 18 kernels found
[I 2020-11-05 13:22:44.808 SingleUserNotebookApp extension:162] JupyterLab extension loaded from /opt/modules/i12g/anaconda/envs/jupyterhub/lib/python3.7/site-packages/jupyterlab
[I 2020-11-05 13:22:44.808 SingleUserNotebookApp extension:163] JupyterLab application directory is /opt/modules/i12g/anaconda/envs/jupyterhub/share/jupyter/lab
[I 2020-11-05 13:22:44.988 SingleUserNotebookApp __init__:34] [Jupytext Server Extension] Deriving a JupytextContentsManager from LargeFileManager
[I 2020-11-05 13:22:44.989 SingleUserNotebookApp singleuser:561] Starting jupyterhub-singleuser server version 1.1.0
[I 2020-11-05 13:22:44.996 SingleUserNotebookApp notebookapp:2209] Serving notebooks from local directory: /data/nasif12/home_if12/hoelzlwi
[I 2020-11-05 13:22:44.996 SingleUserNotebookApp notebookapp:2209] Jupyter Notebook 6.1.4 is running at:
[I 2020-11-05 13:22:44.996 SingleUserNotebookApp notebookapp:2209] http://[...]:50758/
[I 2020-11-05 13:22:44.996 SingleUserNotebookApp notebookapp:2210] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 2020-11-05 13:22:45.010 SingleUserNotebookApp singleuser:542] Updating Hub with activity every 300 seconds
slurmstepd: error: *** JOB 377371 ON [...] CANCELLED AT 2020-11-05T13:23:39 ***

Logs of jupyterhub:

[I 2020-11-05 13:27:39.649 JupyterHub log:181] 302 POST /jupyter/hub/spawn/<user> -> /jupyter/hub/spawn-pending/<user> (<user>@192.168.16.11) 1013.71ms
[I 2020-11-05 13:27:39.761 JupyterHub pages:398] <user> is pending spawn
[I 2020-11-05 13:27:39.771 JupyterHub log:181] 200 GET /jupyter/hub/spawn-pending/<user> (<user>@192.168.16.11) 29.93ms
[I 2020-11-05 13:27:41.587 JupyterHub log:181] 200 POST /jupyter/hub/api/batchspawner (<user>@192.168.16.13) 24.47ms
[I 2020-11-05 13:27:43.792 JupyterHub log:181] 200 GET /jupyter/hub/api (@192.168.16.13) 2.84ms
[I 2020-11-05 13:27:43.843 JupyterHub log:181] 200 POST /jupyter/hub/api/users/<user>/activity (<user>@192.168.16.13) 36.08ms
[W 2020-11-05 13:27:48.647 JupyterHub base:995] User <user> is slow to start (timeout=10)
[W 2020-11-05 13:28:38.764 JupyterHub user:684] <user>'s server failed to start in 60 seconds, giving up
[I 2020-11-05 13:28:39.153 JupyterHub batchspawner:408] Stopping server job 377372
[I 2020-11-05 13:28:39.155 JupyterHub batchspawner:293] Cancelling job 377372: sudo -E -u <user> scancel 377372
[W 2020-11-05 13:28:51.948 JupyterHub batchspawner:419] Notebook server job 377372 at node03:0 possibly failed to terminate
[E 2020-11-05 13:28:52.010 JupyterHub gen:624] Exception in Future <Task finished coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /opt/modules/i12g/anaconda/envs/jupyterhub/lib/python3.7/site-packages/jupyterhub/handlers/base.py:884> exception=TimeoutError('Timeout')> after timeout
    Traceback (most recent call last):
      File "/opt/modules/i12g/anaconda/envs/jupyterhub/lib/python3.7/site-packages/tornado/gen.py", line 618, in error_callback
        future.result()
      File "/opt/modules/i12g/anaconda/envs/jupyterhub/lib/python3.7/site-packages/jupyterhub/handlers/base.py", line 891, in finish_user_spawn
        await spawn_future
      File "/opt/modules/i12g/anaconda/envs/jupyterhub/lib/python3.7/site-packages/jupyterhub/user.py", line 708, in spawn
        raise e
      File "/opt/modules/i12g/anaconda/envs/jupyterhub/lib/python3.7/site-packages/jupyterhub/user.py", line 607, in spawn
        url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
    tornado.util.TimeoutError: Timeout
    
[I 2020-11-05 13:28:52.019 JupyterHub log:181] 200 GET /jupyter/hub/api/users/<user>/server/progress (<user>@192.168.16.11) 71227.51ms

I am using python 3.7, jupyterhub 1.2, batchspawner 1.0.1 and the current git version of wrapspawner.
When directly applying batchspawner, everything is working fine.

My configuration to reproduce:

c.JupyterHub.allow_named_servers = True
c.JupyterHub.named_server_limit_per_user = 5

c.PAMAuthenticator.open_sessions = False

from jupyterhub.auth import PAMAuthenticator
import pamela
from tornado import gen

class KerberosPAMAuthenticator(PAMAuthenticator):
    @gen.coroutine
    def authenticate(self, handler, data):
        """Authenticate with PAM, and return the username if login is successful.
        Return None otherwise.
        Establish credentials when authenticating instead of reinitializing them
        so that a Kerberos cred cache has the proper UID in it.
        """
        username = data['username']
        try:
            pamela.authenticate(username, data['password'], service=self.service, resetcred=pamela.PAM_ESTABLISH_CRED)
        except pamela.PAMError as e:
            if handler is not None:
                self.log.warning("PAM Authentication failed (%s@%s): %s", username, handler.request.remote_ip, e)
            else:
                self.log.warning("PAM Authentication failed: %s", e)
        else:
            return username

c.JupyterHub.authenticator_class = KerberosPAMAuthenticator


c.JupyterHub.bind_url = 'http://:8686/jupyter/'
c.JupyterHub.default_url = 'home'
c.JupyterHub.hub_connect_ip = 'node01'
c.JupyterHub.hub_ip = '0.0.0.0'
c.JupyterHub.hub_port = 8687

c.Spawner.default_url = '/lab'
c.Spawner.http_timeout = 120

import batchspawner

c.JupyterHub.spawner_class = 'batchspawner.SlurmSpawner'
c.BatchSpawnerBase.req_nprocs = '2'
c.BatchSpawnerBase.req_runtime = '48:00:00'
c.BatchSpawnerBase.req_memory = '12gb'

c.SlurmSpawner.req_partition = 'slurm-jupyter'

c.SlurmSpawner.start_timeout = 240

c.SlurmSpawner.batch_script = """#!/bin/bash -x
{% if partition  %}#SBATCH --partition={{partition}}
{% endif %}{% if runtime    %}#SBATCH --time={{runtime}}
{% endif %}{% if memory     %}#SBATCH --mem={{memory}}
{% endif %}{% if gres       %}#SBATCH --gres={{gres}}
{% endif %}{% if nprocs     %}#SBATCH --cpus-per-task={{nprocs}}
{% endif %}{% if reservation%}#SBATCH --reservation={{reservation}}
{% endif %}{% if options    %}#SBATCH {{options}}{% endif %}

trap 'echo SIGTERM received' TERM
{{prologue}}
which jupyterhub-singleuser
{% if srun %}{{srun}} {% endif %}{{cmd}}
echo "jupyterhub-singleuser ended gracefully"
{{epilogue}}

"""

c.BatchSpawnerBase.req_prologue = '''


export XDG_RUNTIME_DIR=""
export SHELL=/bin/bash
export BASH=/bin/bash

# activate the correct conda environment
source /opt/modules/i12g/anaconda/envs/jupyterhub/bin/activate

env | sort
'''


c.SlurmSpawner.req_srun = ''


###
# comment in the following line to test the ProfilesSpawner
# c.JupyterHub.spawner_class = 'wrapspawner.ProfilesSpawner'

c.ProfilesSpawner.profiles = [
  (
    'SLURM CPU node - 4 cores, 16 GB, 24 hours',
    'juphub-4cpu-16G',
    'batchspawner.SlurmSpawner',
    dict(
      req_nprocs='4',
      req_partition='slurm-jupyter',
      req_runtime='24:00:00',
      req_memory='16000'
    )
  ),
  (
    'SLURM CPU node - 8 cores, 16 GB, 24 hours',
    'juphub-8cpu-16G',
    'batchspawner.SlurmSpawner',
    dict(
      req_nprocs='8',
      req_partition='slurm-jupyter',
      req_runtime='24:00:00',
      req_memory='16000'
    )
  ),
  (
    "Test server",
    'local-test',
    'jupyterhub.spawner.LocalProcessSpawner',
    {
      'ip':'0.0.0.0'
    }
  )
]

from pprint import pprint
pprint(c.ProfilesSpawner.profiles)

Hi all, I’m in the process of setting up JupyterHub (with wrapspawner and batchspawner) to better utilise Jupyter on a Slurm cluster, but I’m running into the following issue.

On a successful spawn of the single user session on a cluster node (as indicated by the Slurm output), the user is redirected to '/hostname/user/username/' as expected, but the page 404s.

The issue seems to be the JUPYTERHUB_SERVICE_PREFIX environment variable that’s passed to the single-user session via the job submission script being unbound, instead of ‘/user/username/’.

Following this problem back through the code, I found myself in the construct_child() function of wrapspawner.

def construct_child(self):
        if self.child_spawner is None:
            self.child_spawner = self.child_class(
                user = self.user,
                db   = self.db,
                hub = self.hub,
                authenticator = self.authenticator,
                oauth_client_id = self.oauth_client_id,
                server = self._server,
                config = self.config,
                **self.child_config
                )

When construct_child() is called, self._server contains the correct value. However, following the snippet of code above, self.child_spawner.server - which is later used to define JUPYTERHUB_SERVICE_PREFIX - is None, rather than equal to self._server as I would expect. All other values passed into self.child_class are maintained in self.child_spawner.

What’s odd is that this self.child_spawner.server variable behaves very strangely. Doing absolutely anything to it in the lines that follow the snippet above will solve the 404 problem. If I manually set self.child_spawner.server to self._server straight after self.child_class is created and applied to self.child_spawner, JUPYTERHUB_SERVICE_PREFIX gets populated correctly and the 404 issue goes away. However, both setting the value to None and simply printing it via a self.log.debug(str(self.child_spawner.server)) statement also fixes it.

I’ve been working through the spawner code to try to make sense of this, but I’m not having much luck.
Can anyone explain this strange behaviour?

Could this be related to the patch I had to implement here: jupyterhub/batchspawner#127 (comment)?

If not, what causes the server value passed to the child_class to not be set correctly, and how can this safely be fixed in a more permanent way?

Conda environment

Jupyterhub config file

Jupyterhub output log for run with debug line patch in place
Slurm log for run with patch in place

Jupyterhub output log for run without debug line patch in place
Slurm log for run without patch in place

Thanks in advance.

In #51 we added a property linking the child spawner's server. About the same time, jupyterhub/jupyterhub#3810 was merged to keep Spawner.server in sync with underlying orm_spawner.server and that PR was referenced in the discussion on #50 and #51. Folks tested #50 and #51 against JupyterHub setups that I don't think included the server sync fix.

While catching up on my deployment this week I noticed I was still running off the branch in #50 (with the strange hack), and when I tried to run off current wrapspawner master it failed to spawn properly. Specifically, during spawn the Hub reports that it expects the spawner on the wrong URL (as reported from here), namely a URL that corresponds to the hub itself.

In the discussion on #50 Min mentioned that the server sync fix may be all that was necessary so I decided to follow-up on that conjecture, and I reverted #51 (equivalent to just running the latest release, 1.0.1), and for me things now seem to be working again. I think #51 somehow conflicts with jupyterhub/jupyterhub#3810 but may be obviated by it anyway.

We probably want to consider reverting #51, but I think others should test. Basically the question is whether #51 breaks anything for anyone running JupyterHub 2.2.0 or later.

Hello,
I want to use CondorSpawner to spawn notebooks on HTCondor.
For this I need to add a pre_spawn_hook to do some work so that the Kerberos
credentials are put into the spawner.environment properly.
When I use CondorSpawner directly the pre_spawn_hook executes and
spawning works.
With wrapspawner.ProfilesSpawner the pre_spawn_hook seems to execute (I added a log message) but spawning doesn't work because the environment in the spawner isn't set properly.
Is adding the pre_spawn_hook via c.Spawner.pre_spawn_hook = get_krb5_var
the right way of doing it or do I need to do it differently so that the hook is added to the child spawners?

I have Jupterhub 0.7.2 with dockerspawner 0.7.0 on Linux VM. This is 4 node docker swarm cluster. I have been using LDAP Auth, but no Oauth.

I have created a new docker image with Jupterhub 0.9.4. Upon running this new container from the VM that has Jupterhub_config.py with Jupyterhub 0.7.2, I am getting the following error.

File "/usr/local/lib/python3.5/dist-packages/jupyterhub/services/auth.py", line 484, in _ensure_not_empty
raise ValueError("%s cannot be empty." % proposal.trait.name)
ValueError: oauth_client_id cannot be empty.

#Below is the configuration of Jupterhub_config.py
[root@jupyterhub]# cat jupyterhub_config.py
from tornado import gen
import os
os.environ["DOCKER_HOST"] = ":4001"
c.DockerSpawner.use_docker_client_env = True

c.JupyterHub.hub_port = 8081
c.JupyterHub.ip = '10.126.207.52'
c.JupyterHub.port = 443
c.JupyterHub.proxy_api_ip = '0.0.0.0'
c.JupyterHub.hub_ip = '10.126.207.52'

c.Authenticator.admin_users = {'devdocadm'}

c.Spawner.default_url = '/user/{username}'

c.ConfigurableHTTPProxy.command = ['configurable-http-proxy', '--ssl-protocol', 'TLSv1_2']
c.JupyterHub.proxy_cmd = ['configurable-http-proxy', '--ssl-protocol=TLSv1_2']

c.JupyterHub.ssl_cert = '/jupyter01/cert/dev-hub-ssl/jupyterhub-dev01.myco.com.chain.pem'
c.JupyterHub.ssl_key = '/jupyter01/cert/dev-hub-ssl/jupyterhub-dev01.myco.com.key'

c.JupyterHub.cookie_secret_file = '/jupyter01/config/jupyterhub/jupyterhub_cookie_secret'

c.JupyterHub.spawner_class = 'dockerspawner.SystemUserSpawner'

c.DockerSpawner.container_image = 'jupyterhub:5000/jhubatf:1'

c.DockerSpawner.read_only_volumes = { '/data01/shared/' : '/home/{username}/shared' }
c.DockerSpawner.volumes = {
'/data01/jupyterhub/custom' : '/opt/conda/lib/python3.5/site-packages/notebook/static/custom',
'/data/{username}' : '/home/{username}/',
'/data01/shared/{username}' : '/home/{username}/myshare',
}
c.SystemUserSpawner.host_homedir_format_string = '/data/{username}'
c.Spawner.ip = '10.126.207.52'
c.Spawner.args = ['--JupyterApp.config_file=/etc/jupyter/jupyter_notebook_config.py']
c.DockerSpawner.container_ip = "0.0.0.0"
c.DockerSpawner.hub_ip_connect = '10.126.207.52'
c.DockerSpawner.remove_containers = True

c.JupyterHub.authenticator_class = 'ldapauthenticator.LDAPAuthenticator'
c.LDAPAuthenticator.server_address = 'ad-nam.myco.com'
c.LDAPAuthenticator.bind_dn_template = 'nam{username}'
c.LDAPAuthenticator.server_port = 636
c.LDAPAuthenticator.lookup_dn = True
c.LDAPAuthenticator.user_attribute = 'sAMAccountName'
c.LDAPAuthenticator.use_ssl = True

c.JupyterHub.tornado_settings = {
'headers': {
'Content-Security-Policy': "frame-ancestors * 'self'"
}
}

I think this exception is harmless, but it shows up a lot in my log and I'm wondering if there might be an alternate approach to what this exception does:

    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/dist-packages/tornado/web.py", line 1704, in _execute
        result = await result
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/apihandlers/users.py", line 737, in get
        async for event in events:
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/utils.py", line 570, in iterate_until
        yield item_future.result()
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/spawner.py", line 1146, in _generate_progress
        async with aclosing(self.progress()) as progress:
      File "/usr/local/lib/python3.8/dist-packages/wrapspawner/wrapspawner.py", line 152, in progress
        raise RuntimeError("No child spawner yet exists - can not get progress yet")
    RuntimeError: No child spawner yet exists - can not get progress yet

This exception was added in #21 to prevent hitting a recursion limit. Is throwing an exception the only way out of that?

So people can install this at with specific version number.

License missing

We do not have any automated testing set up, but we should.

Testing will need to be focused on integration, not unit tests, since the major sources of error are going to be around property or method accesses to the wrapped Spawner class, which means running actual Jupyterhub code and verifying those interface points. We can probably pull the necessary logic from the Jupyterhub test suite.

This is with JuptyterHub 0.8.0b5 --- when I start up without any pre-existing sqlite database I still get the KeyError. Maybe a recent change there causes this to fail now?

A spawner environment dictionary value is allowed to be either a string or a callable that returns a string. But if a wrapspawner child_config contains an environment dictionary where a value is a callable, then get_state() can't return a properly JSON-serializable dictionary. When the state is saved to the database, it seems that a null is recorded. API calls that end up calling get_state() and return spawner state, i.e. /hub/api/users crash because they can't turn a function into a string. It does not record the "realized" value, which makes sense; that is only "realized" when get_env() is called, and not recorded after that.

If environment is allowed to be a dictionary with one or more callable values, then it just cannot be considered properly JSON-serializable, and would break the contract that get_state() return only JSON-serializable dictionaries. That is one problem. Another question though is whether environment qualifies as state that the Hub actually needs to maintain.

If we decided environment could be dropped from child_conf entries and API responses, that's probably not the end of it, since wrapped spawners can have all kinds of arguments that aren't serializable.