jupyterhub / sudospawner Goto Github PK
View Code? Open in Web Editor NEWSpawn JupyterHub single-user servers with sudo
License: BSD 3-Clause "New" or "Revised" License
Spawn JupyterHub single-user servers with sudo
License: BSD 3-Clause "New" or "Revised" License
$USER and $SHELL are not set in user servers launched with sudospawner 0.1.0. Also .bashrc is not processed when a terminal is started.
Variables were set successfully in 0.0.1 up through #8. (Problem does not exist in installation from Jan 28th.)
Hi,
I'm new in this area and trying to install jupyterhub using sudospawner on my local machine to see how it works. Everything works fine for one user (the one I installed everything by that) but for other users I'm getting 500 internal server error. It seems there is a permission issue, but I tried every solution I've found to fix it but nothing has worked.
Ubuntu 16.4
Jupyterhub 0.9.0.b1
Any idea what should I do?
I would be happy to give more information if needed.
Thanks,
The variable c.Spawner.args
in jupyterhub_conf.py
is set as follows:
c.Spawner.args = ['--JupyterApp.config_file=/filePath/jupyter_notebook_config.py']
However, the config_file of notebook
is not getting linked.
JupyterApp.config_file
should get linked!
We've got a hub running with wrapspawner so that users can dynamically choose different spawner class options. I'm currently working on setting it up so it can be run as a non-root user but I'm stuck on how sudospawner could be used with wrapspawner since they both require setting the spawner_class in the configuration file. Has anyone had success setting up a hub this way?
Hi,
We have from times to times empty responses from mediator leading to 500 error for end user. It's hard to get the root cause now. Seems user can not spawn a process.
Anyone having the same issue ?
sudospawner/sudospawner/spawner.py
Line 80 in c291a89
[E 2018-02-22 10:23:41.609 JupyterHub spawner:54] args: '{\'args\': [\'--user="XXXXX"\', \'--cookie-name="jupyter-hub-token-XXXX"\', \'--base-url="/user/XXXXX"\', \'--hub-host=""\', \'--hub-prefix="/hub/"\', \'--hub-api-url="http://127.0.0.1:8081/hub/api"\', \'--ip="127.0.0.1"\', \'--port=12345\', \'--debug\'], \'env\': {\'PATH\': \'/applis/anaconda/4.3/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/applis/jupyterhub/.local/bin:/applis/jupyterhub/bin:/applis/anaconda/4.3/envs/py36_jupyter/bin\', \'LANG\': \'en_US.UTF-8\', \'JUPYTERHUB_API_TOKEN\': \'9fe54affb6c0424ab947d247ffea3d09\', \'JPY_API_TOKEN\': \'9fe54affb6c0424ab947d247ffea3d09\', \'USER\': \'XXXXXX\', \'HOME\': \'/home/XXXXX\', \'SHELL\': \'/bin/bash\'}, \'action\': \'spawn\'}'
[E 2018-02-22 10:23:41.609 JupyterHub spawner:55] user: 'XXXXXX'
[D 2018-02-22 10:23:41.609 JupyterHub spawner:62] Spawning ['sudo', '-u', 'XXXXXX', '-nH', '/applis/anaconda/4.3/envs/py36_jupyter/bin/sudospawner']
[E 2018-02-22 10:23:41.617 JupyterHub spawner:84] Failed to get JSON result from mediator: ''
[E 2018-02-22 10:23:41.620 JupyterHub user:251] Unhandled error starting XXXXXXX's server: substring not found
[E 2018-02-22 10:23:41.632 JupyterHub web:1590] Uncaught exception POST /hub/login?next= (175.128.8.46)
HTTPServerRequest(protocol='https', host='phadlx205.haas.socgen:8788', method='POST', uri='/hub/login?next=', version='HTTP/1.1', remote_ip='175.128.8.46', headers={'X-Forwarded-Host': 'phadlx205.haas.socgen:8788', 'X-Forwarded-Proto': 'https', 'X-Forwarded-Port': '8788', 'X-Forwarded-For': '175.128.8.46', 'Accept-Language': 'fr-FR,fr;q=0.9,en-US;q=0.8,en;q=0.7', 'Accept-Encoding': 'gzip, deflate, br', 'Referer': 'https://phadlx205.haas.socgen:8788/hub/login', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8', 'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.108 Safari/537.36', 'Content-Type': 'application/x-www-form-urlencoded', 'Upgrade-Insecure-Requests': '1', 'Origin': 'https://phadlx205.haas.socgen:8788', 'Cache-Control': 'max-age=0', 'Content-Length': '46', 'Connection': 'close', 'Host': 'phadlx205.haas.socgen:8788'})
Traceback (most recent call last):
File "/applis/anaconda/4.3/envs/py36_jupyter/lib/python3.6/site-packages/tornado/web.py", line 1511, in _execute
result = yield result
File "/applis/anaconda/4.3/envs/py36_jupyter/lib/python3.6/site-packages/jupyterhub/handlers/login.py", line 84, in post
yield self.spawn_single_user(user)
File "/applis/anaconda/4.3/envs/py36_jupyter/lib/python3.6/site-packages/jupyterhub/handlers/base.py", line 328, in spawn_single_user
yield gen.with_timeout(timedelta(seconds=self.slow_spawn_timeout), f)
File "/applis/anaconda/4.3/envs/py36_jupyter/lib/python3.6/site-packages/jupyterhub/user.py", line 261, in spawn
raise e
File "/applis/anaconda/4.3/envs/py36_jupyter/lib/python3.6/site-packages/jupyterhub/user.py", line 229, in spawn
ip_port = yield gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
File "/applis/anaconda/4.3/envs/py36_jupyter/lib/python3.6/site-packages/sudospawner/spawner.py", line 92, in start
yield self.do(action='spawn', args=self.get_args(), env=self.get_env())
File "/applis/anaconda/4.3/envs/py36_jupyter/lib/python3.6/site-packages/sudospawner/spawner.py", line 81, in do
data_str = data_str[data_str.index('{'):data_str.rindex('}')+1]
ValueError: substring not found
[D 2018-02-22 10:23:41.636 JupyterHub base:462] No template for 500
Regards,
Yann
Has anyone had any issues with using sudospawner with version 0.8.0 of jupyterhub?
I can't see any specific errors in the log, other than it mentions that it took too long for the session to spawn. It was all working fine with 0.7.2.
My jupyterhub database had a PID entry for a user notebook server that no longer existed. When the hub started up and tried to health check that PID, it hit the following exception and failed to start. The only workaround was to nuke the database or perform surgery on it to remove the offending state.
Traceback (most recent call last):
File "/opt/jupyterhub/envs/jupyterhub/bin/sudospawner", line 4, in <module>
mediator.main()
File "/opt/jupyterhub/envs/jupyterhub/lib/python3.5/site-packages/sudospawner/mediator.py", line 148, in main
kill(**kwargs)
File "/opt/jupyterhub/envs/jupyterhub/lib/python3.5/site-packages/sudospawner/mediator.py", line 52, in kill
os.kill(pid, signal)
PermissionError: [Errno 1] Operation not permitted
ESC[31m[E 2017-04-17 20:40:54.603 JupyterHub app:1527]ESC[0;10m
Traceback (most recent call last):
File "/opt/jupyterhub/envs/jupyterhub/lib/python3.5/site-packages/jupyterhub/app.py", line 1524, in launch_instance_async
yield self.initialize(argv)
File "/opt/jupyterhub/envs/jupyterhub/lib/python3.5/types.py", line 179, in throw
return self.__wrapped.throw(tp, *rest)
File "/opt/jupyterhub/envs/jupyterhub/lib/python3.5/site-packages/jupyterhub/app.py", line 1315, in initialize
yield self.init_spawners()
File "/opt/jupyterhub/envs/jupyterhub/lib/python3.5/site-packages/jupyterhub/app.py", line 1087, in init_spawners
status = yield spawner.poll()
File "/opt/jupyterhub/envs/jupyterhub/lib/python3.5/site-packages/jupyterhub/spawner.py", line 834, in poll
alive = yield self._signal(0)
File "/opt/jupyterhub/envs/jupyterhub/lib/python3.5/site-packages/sudospawner/spawner.py", line 90, in _signal
reply = yield self.do('kill', pid=self.pid, signal=sig)
File "/opt/jupyterhub/envs/jupyterhub/lib/python3.5/site-packages/sudospawner/spawner.py", line 69, in do
data_str = data_str[data_str.index('{'):data_str.rindex('}')+1]
ValueError: substring not found
I'm not sure how it got into the state that it did. Maybe the notebook server went away while the hub was restarting? Regardless, the hub should be robust against the situation and at least start.
I have read and tried to follow jupyterhub/jupyterhub#314 but I'm guessing I cannot subclass SudoSpawner in the same way.
Ultimately, I want to auto-create the notebook dir in the user's home directory. I understand why jupyterhub does not want to do this for everyone. We have a large user population, with only a smaller subset of jupyter users. Pre-creating for everyone doesn't make sense, the Server Error 500 is a bad experience, and even if it were easy to display an informative error message, it is still not a great user experience. So I'm left to hack it in. :(
Is there a similar way I can subclass SudoSpawner to add a directory creation routine run through sudo?
BTW, the reason we have set notebook dir to something other than the user's homedir is that we want users to be able to set permissions to share notebooks w/o accidentally compromising their homedir permissions. A jupyterhub sharing mechanism would also solve this issue.
I am running jupyterhub as root
user.
I have 2 other users,
I am able to login with alawrence
. But when I try to login with puthiry
, I am getting the following error
[I 2018-05-14 06:37:09.876 JupyterHub log:122] 302 GET /user/puthiry/ โ /hub/user/puthiry/ (@::ffff:10.0.2.2) 18.90ms
sudo: unable to execute /home/alawrence/dev/jupyter/lab/bin/sudospawner: Permission denied
[E 2018-05-14 06:37:10.492 JupyterHub spawner:83] Failed to get JSON result from mediator: ''
[E 2018-05-14 06:37:10.520 JupyterHub user:427] Unhandled error starting puthiry's server: substring not found
[E 2018-05-14 06:37:10.692 JupyterHub web:1621] Uncaught exception GET /hub/user/puthiry/ (::ffff:10.0.2.2)
HTTPServerRequest(protocol='http', host='localhost:8000', method='GET', uri='/hub/user/puthiry/', version='HTTP/1.1', remote_ip='::ffff:10.0.2.2')
Traceback (most recent call last):
File "/home/alawrence/dev/jupyter/lab/lib/python3.6/site-packages/tornado/web.py", line 1543, in _execute
result = yield result
File "/home/alawrence/dev/jupyter/lab/lib/python3.6/site-packages/jupyterhub/handlers/base.py", line 747, in get
yield self.spawn_single_user(current_user)
File "/home/alawrence/dev/jupyter/lab/lib/python3.6/site-packages/jupyterhub/handlers/base.py", line 475, in spawn_single_user
yield gen.with_timeout(timedelta(seconds=self.slow_spawn_timeout), finish_spawn_future)
File "/home/alawrence/dev/jupyter/lab/lib/python3.6/site-packages/jupyterhub/handlers/base.py", line 445, in finish_user_spawn
yield spawn_future
File "/home/alawrence/dev/jupyter/lab/lib/python3.6/site-packages/jupyterhub/user.py", line 439, in spawn
raise e
File "/home/alawrence/dev/jupyter/lab/lib/python3.6/site-packages/jupyterhub/user.py", line 378, in spawn
ip_port = yield gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
File "/home/alawrence/dev/jupyter/lab/lib/python3.6/site-packages/sudospawner/spawner.py", line 92, in start
reply = yield self.do(action='spawn', args=self.get_args(), env=self.get_env())
File "/home/alawrence/dev/jupyter/lab/lib/python3.6/site-packages/sudospawner/spawner.py", line 80, in do
data_str = data_str[data_str.index('{'):data_str.rindex('}')+1]
ValueError: substring not found
[E 2018-05-14 06:37:10.860 JupyterHub log:114] {
"X-Forwarded-Host": "localhost:8000",
"X-Forwarded-Proto": "http",
"X-Forwarded-Port": "8000",
"X-Forwarded-For": "::ffff:10.0.2.2",
"Upgrade-Insecure-Requests": "1",
"Connection": "close",
"Dnt": "1",
"Cookie": "jupyter-hub-token=\"2|1:0|10:1526293512|17:jupyter-hub-token|44:MmIxYjM2YWM4MTJkNDg0NWFmM2NmNzM2MmExZGM4ODM=|e8aeee3929320302dc2d38569c3914a0360b03ac1f224e56188047e14c474979\"",
"Referer": "http://localhost:8000/hub/home",
"Accept-Encoding": "gzip, deflate",
"Accept-Language": "en-US,en;q=0.5",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"User-Agent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:59.0) Gecko/20100101 Firefox/59.0",
"Host": "localhost:8000"
}
[E 2018-05-14 06:37:10.867 JupyterHub log:122] 500 GET /hub/user/puthiry/ (puthiry@::ffff:10.0.2.2) 896.26ms
I've configured Jupyterhub with SudoSpawner (which runs as a non-root user) and I'm able to run it without any trouble. But when I logged in with the user rhea (as provided in the example) with admin permissions set, I'm not able to add users in the system. I'm getting the following error:
[E 2019-07-24 08:26:56.638 JupyterHub users:94] Failed to create user: newuser
Traceback (most recent call last):
File "/opt/conda/lib/python3.6/site-packages/jupyterhub/apihandlers/users.py", line 92, in post
await maybe_future(self.authenticator.add_user(user))
File "/opt/conda/lib/python3.6/site-packages/jupyterhub/auth.py", line 711, in add_user
await maybe_future(self.add_system_user(user))
File "/opt/conda/lib/python3.6/site-packages/jupyterhub/auth.py", line 771, in add_system_user
raise RuntimeError("Failed to create system user %s: %s" % (name, err))
RuntimeError: Failed to create system user newuser: adduser: Only root may add a user or group to the system.
I'm guessing it is caused by running JupyterHub as a non-root user and also using PAM Authentication which eventually uses the system users. The only limitation for my project is I must not run JupyterHub as a root user. Is there a way to add users using sudospawner with a different authenticator, else Is there a way to give restricted sudo permission for the user rhea to be able to add users to the system?? @minrk @willingc @yuvipanda
Hi guys!
I try to use the SudoSpawner for JupyterHub. I followed these instructions: https://github.com/jupyter/jupyterhub/wiki/Using-sudo-to-run-JupyterHub-without-root-privileges all the tests was fine, but I've some weird error.
I tried to run:
sudo -u jhub_runner jupyterhub --JupyterHub.spawner_class=sudospawner.SudoSpawner --debug
but when I try to login with my dedicated user I get:
Unhandled error starting pierluigi_jupyter's server: No JSON object could be decoded
I investigated a little in spawner.py and I figured out that error is at:
return json.loads(data.decode('utf8'))
and it is due to the fact that data is an empty string.
cmd = ['sudo', '-u', 'pierluigi_jupyter', '-nH', 'sudospawner']
kwargs = {'action': 'spawn', 'args': ['--user=pierluigi_jupyter', '--port=53251', '--cookie-name=jupyter-hub-token-pierluigi_jupyter', '--base-url=/user/pierluigi_jupyter', '--hub-prefix=/hub/', '--hub-api-url=http://localhost:8081/hub/api', '--ip=localhost'], 'env': {'USER': 'pierluigi_jupyter', 'HOME': '/home/pierluigi_jupyter', 'JPY_API_TOKEN': 'f32f975c2cf14bbe90362f9aa0f9f595', 'SHELL': '/bin/sh', 'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'LANG': 'en_US.UTF-8'}}
json.dumps(kwargs).encode('utf8') = b'{"action": "spawn", "args": ["--user=pierluigi_jupyter", "--port=53251", "--cookie-name=jupyter-hub-token-pierluigi_jupyter", "--base-url=/user/pierluigi_jupyter", "--hub-prefix=/hub/", "--hub-api-url=http://localhost:8081/hub/api", "--ip=localhost"], "env": {"USER": "pierluigi_jupyter", "HOME": "/home/pierluigi_jupyter", "JPY_API_TOKEN": "f32f975c2cf14bbe90362f9aa0f9f595", "SHELL": "/bin/sh", "PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "LANG": "en_US.UTF-8"}}'
data.decode('utf8') = ''
do you have any hint?
PS JupyterHub itself works fine without any issue.
Per the JupyterHub documentation
c.Spawner.mem_limit: A limit specifies the maximum amount of memory that may be allocated, though there is no promise that the maximum amount will be available. In supported spawners, you can set c.Spawner.mem_limit to limit the total amount of memory that a single-user notebook server can allocate. Attempting to use more memory than this limit will cause errors. The single-user notebook server can discover its own memory limit by looking at the environment variable MEM_LIMIT, which is specified in absolute bytes.
In the configuration file, there is a caveat though:
This is a configuration setting. Your spawner must implement support for the limit to work. The default spawner,
LocalProcessSpawner
, does not implement this support.
So does SudoSpawner implement this? Can it?
Hi. I get 500 error when creating notebook as a user with "." in username. We are using ldap to authenticate users and almost everyone has name in form of "firstletterofthename.surname".
I would have a notebook.
It goes like this:
(look for the "ValueError: substring not found", it might be it)
mar 13 10:44:27 some-host-name jupyterhub[636247]: [E 2024-03-13 10:44:27.605 JupyterHub pages:313] Error starting server some.username: substring not found
mar 13 10:44:27 some-host-name jupyterhub[636247]: Traceback (most recent call last):
mar 13 10:44:27 some-host-name jupyterhub[636247]: None: None
mar 13 10:44:27 some-host-name jupyterhub[636247]:
mar 13 10:44:27 some-host-name jupyterhub[636247]: [W 2024-03-13 10:44:27.606 JupyterHub web:1873] 500 GET /hub/spawn (127.0.0.1): Unhandled error starting server some.username
mar 13 10:44:27 some-host-name jupyterhub[636247]: [E 2024-03-13 10:44:27.612 JupyterHub log:183] {
mar 13 10:44:27 some-host-name jupyterhub[636247]: "X-Forwarded-Proto": "http",
mar 13 10:44:27 some-host-name jupyterhub[636247]: "X-Forwarded-Port": "80",
mar 13 10:44:27 some-host-name jupyterhub[636247]: "Connection": "Keep-Alive",
mar 13 10:44:27 some-host-name jupyterhub[636247]: "X-Forwarded-Server": "some-address.local",
mar 13 10:44:27 some-host-name jupyterhub[636247]: "X-Forwarded-Host": "some-host-name.com",
mar 13 10:44:27 some-host-name jupyterhub[636247]: "X-Forwarded-For": "ip.ip.ip.ip,127.0.0.1",
mar 13 10:44:27 some-host-name jupyterhub[636247]: "Sec-Fetch-User": "?1",
mar 13 10:44:27 some-host-name jupyterhub[636247]: "Sec-Fetch-Site": "same-origin",
mar 13 10:44:27 some-host-name jupyterhub[636247]: "Sec-Fetch-Mode": "navigate",
mar 13 10:44:27 some-host-name jupyterhub[636247]: "Sec-Fetch-Dest": "document",
mar 13 10:44:27 some-host-name jupyterhub[636247]: "Upgrade-Insecure-Requests": "1",
mar 13 10:44:27 some-host-name jupyterhub[636247]: "Cookie": "_xsrf=[secret]; jupyterhub-hub-login=[secret]; _ga_PWDD85JKVC=[secret]; _ga=[secret]; jupyterhub-session-id=[secret]",
mar 13 10:44:27 some-host-name jupyterhub[636247]: "Referer": "https://some-host-name.com/hub/login",
mar 13 10:44:27 some-host-name jupyterhub[636247]: "Accept-Encoding": "gzip, deflate, br",
mar 13 10:44:27 some-host-name jupyterhub[636247]: "Accept-Language": "pl,en-US;q=0.7,en;q=0.3",
mar 13 10:44:27 some-host-name jupyterhub[636247]: "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
mar 13 10:44:27 some-host-name jupyterhub[636247]: "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:123.0) Gecko/20100101 Firefox/123.0",
mar 13 10:44:27 some-host-name jupyterhub[636247]: "Host": "some-host-name.com"
mar 13 10:44:27 some-host-name jupyterhub[636247]: }
mar 13 10:44:27 some-host-name jupyterhub[636247]: [E 2024-03-13 10:44:27.612 JupyterHub log:191] 500 GET /hub/spawn ([email protected]) 264.13ms
mar 13 10:44:30 some-host-name jupyterhub[636247]: [I 2024-03-13 10:44:30.079 JupyterHub log:191] 200 GET /hub/home ([email protected]) 9.39ms
mar 13 10:44:31 some-host-name jupyterhub[636247]: [I 2024-03-13 10:44:31.472 JupyterHub provider:659] Creating oauth client jupyterhub-user-some.username
mar 13 10:44:31 some-host-name sudo[640181]: jupyterhub : PWD=/opt/jupyterhub/app/hub ; USER=some.username ; COMMAND=/opt/jupyterhub/app/hub/bin/sudospawner
mar 13 10:44:31 some-host-name jupyterhub[636247]: /opt/jupyterhub/app/hub/bin/python3.9: error while loading shared libraries: libcrypt.so.1: failed to map segment from shared object
mar 13 10:44:31 some-host-name jupyterhub[636247]: [E 2024-03-13 10:44:31.653 JupyterHub spawner:83] Failed to get JSON result from mediator: ''
mar 13 10:44:31 some-host-name jupyterhub[636247]: [E 2024-03-13 10:44:31.654 JupyterHub user:884] Unhandled error starting some.username's server: substring not found
mar 13 10:44:31 some-host-name jupyterhub[636247]: Traceback (most recent call last):
mar 13 10:44:31 some-host-name jupyterhub[636247]: File "/opt/jupyterhub/app/hub/lib64/python3.9/site-packages/jupyterhub/user.py", line 798, in spawn
mar 13 10:44:31 some-host-name jupyterhub[636247]: url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
mar 13 10:44:31 some-host-name jupyterhub[636247]: File "/opt/jupyterhub/app/hub/lib64/python3.9/site-packages/sudospawner/spawner.py", line 92, in start
mar 13 10:44:31 some-host-name jupyterhub[636247]: reply = yield self.do(action='spawn', args=self.get_args(), env=self.get_env())
mar 13 10:44:31 some-host-name jupyterhub[636247]: File "/opt/jupyterhub/app/hub/lib64/python3.9/site-packages/sudospawner/spawner.py", line 80, in do
mar 13 10:44:31 some-host-name jupyterhub[636247]: data_str = data_str[data_str.index('{'):data_str.rindex('}')+1]
mar 13 10:44:31 some-host-name jupyterhub[636247]: ValueError: substring not found
mar 13 10:44:31 some-host-name jupyterhub[636247]:
mar 13 10:44:31 some-host-name jupyterhub[636247]: [E 2024-03-13 10:44:31.702 JupyterHub pages:313] Error starting server some.username: substring not found
mar 13 10:44:31 some-host-name jupyterhub[636247]: Traceback (most recent call last):
mar 13 10:44:31 some-host-name jupyterhub[636247]: None: None
mar 13 10:44:31 some-host-name jupyterhub[636247]:
mar 13 10:44:31 some-host-name jupyterhub[636247]: [W 2024-03-13 10:44:31.703 JupyterHub web:1873] 500 GET /hub/spawn/some.username (127.0.0.1): Unhandled error starting server some.username
mar 13 10:44:31 some-host-name jupyterhub[636247]: [E 2024-03-13 10:44:31.708 JupyterHub log:183] {
mar 13 10:44:31 some-host-name jupyterhub[636247]: "X-Forwarded-Proto": "http",
mar 13 10:44:31 some-host-name jupyterhub[636247]: "X-Forwarded-Port": "80",
mar 13 10:44:31 some-host-name jupyterhub[636247]: "Connection": "Keep-Alive",
mar 13 10:44:31 some-host-name jupyterhub[636247]: "X-Forwarded-Server": "some-address.local",
mar 13 10:44:31 some-host-name jupyterhub[636247]: "X-Forwarded-Host": "some-host-name.com",
mar 13 10:44:31 some-host-name jupyterhub[636247]: "X-Forwarded-For": "ip.ip.ip.ip,127.0.0.1",
mar 13 10:44:31 some-host-name jupyterhub[636247]: "Sec-Fetch-User": "?1",
mar 13 10:44:31 some-host-name jupyterhub[636247]: "Sec-Fetch-Site": "same-origin",
mar 13 10:44:31 some-host-name jupyterhub[636247]: "Sec-Fetch-Mode": "navigate",
mar 13 10:44:31 some-host-name jupyterhub[636247]: "Sec-Fetch-Dest": "document",
mar 13 10:44:31 some-host-name jupyterhub[636247]: "Upgrade-Insecure-Requests": "1",
mar 13 10:44:31 some-host-name jupyterhub[636247]: "Cookie": "_xsrf=[secret]; jupyterhub-hub-login=[secret]; _ga_PWDD85JKVC=[secret]; _ga=[secret]; jupyterhub-session-id=[secret]",
mar 13 10:44:31 some-host-name jupyterhub[636247]: "Referer": "https://some-host-name.com/hub/home",
mar 13 10:44:31 some-host-name jupyterhub[636247]: "Accept-Encoding": "gzip, deflate, br",
mar 13 10:44:31 some-host-name jupyterhub[636247]: "Accept-Language": "pl,en-US;q=0.7,en;q=0.3",
mar 13 10:44:31 some-host-name jupyterhub[636247]: "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
mar 13 10:44:31 some-host-name jupyterhub[636247]: "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:123.0) Gecko/20100101 Firefox/123.0",
mar 13 10:44:31 some-host-name jupyterhub[636247]: "Host": "some-host-name.com"
mar 13 10:44:31 some-host-name jupyterhub[636247]: }
mar 13 10:44:31 some-host-name jupyterhub[636247]: [E 2024-03-13 10:44:31.708 JupyterHub log:191] 500 GET /hub/spawn/some.username ([email protected]) 269.19ms
mar 13 10:44:33 some-host-name jupyterhub[636247]: [I 2024-03-13 10:44:33.468 JupyterHub log:191] 200 GET /hub/home ([email protected]) 9.18ms
mar 13 10:44:35 some-host-name jupyterhub[636247]: [I 2024-03-13 10:44:35.175 JupyterHub login:44] User logged out: some.username
mar 13 10:44:35 some-host-name jupyterhub[636247]: [I 2024-03-13 10:44:35.181 JupyterHub log:191] 302 GET /hub/logout -> /hub/login (@127.0.0.1) 10.03ms
mar 13 10:44:35 some-host-name jupyterhub[636247]: [I 2024-03-13 10:44:35.203 JupyterHub log:191] 200 GET /hub/login (@127.0.0.1) 5.73ms
mar 13 10:45:12 some-host-name jupyterhub[636250]: 10:45:12.946 [ConfigProxy] info: 200 GET /api/routes
mar 13 10:45:39 some-host-name python3.9[636247]: pam_sss(login:auth): authentication success; logname= uid=2001 euid=2001 tty= ruser= rhost= user=some.username
mar 13 10:45:39 some-host-name jupyterhub[636247]: [I 2024-03-13 10:45:39.330 JupyterHub base:837] User logged in: some.username
mar 13 10:45:39 some-host-name jupyterhub[636247]: [I 2024-03-13 10:45:39.332 JupyterHub log:191] 302 POST /hub/login?next= -> /hub/spawn ([email protected]) 497.85ms
mar 13 10:45:39 some-host-name jupyterhub[636247]: [I 2024-03-13 10:45:39.382 JupyterHub provider:659] Creating oauth client jupyterhub-user-some.username
mar 13 10:45:39 some-host-name sudo[640565]: jupyterhub : PWD=/opt/jupyterhub/app/hub ; USER=some.username ; COMMAND=/opt/jupyterhub/app/hub/bin/sudospawner
mar 13 10:45:39 some-host-name jupyterhub[636247]: /opt/jupyterhub/app/hub/bin/python3.9: error while loading shared libraries: libcrypt.so.1: failed to map segment from shared object
mar 13 10:45:39 some-host-name jupyterhub[636247]: [E 2024-03-13 10:45:39.561 JupyterHub spawner:83] Failed to get JSON result from mediator: ''
mar 13 10:45:39 some-host-name jupyterhub[636247]: [E 2024-03-13 10:45:39.563 JupyterHub user:884] Unhandled error starting some.username's server: substring not found
mar 13 10:45:39 some-host-name jupyterhub[636247]: Traceback (most recent call last):
mar 13 10:45:39 some-host-name jupyterhub[636247]: File "/opt/jupyterhub/app/hub/lib64/python3.9/site-packages/jupyterhub/user.py", line 798, in spawn
mar 13 10:45:39 some-host-name jupyterhub[636247]: url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
mar 13 10:45:39 some-host-name jupyterhub[636247]: File "/opt/jupyterhub/app/hub/lib64/python3.9/site-packages/sudospawner/spawner.py", line 92, in start
mar 13 10:45:39 some-host-name jupyterhub[636247]: reply = yield self.do(action='spawn', args=self.get_args(), env=self.get_env())
mar 13 10:45:39 some-host-name jupyterhub[636247]: File "/opt/jupyterhub/app/hub/lib64/python3.9/site-packages/sudospawner/spawner.py", line 80, in do
mar 13 10:45:39 some-host-name jupyterhub[636247]: data_str = data_str[data_str.index('{'):data_str.rindex('}')+1]
mar 13 10:45:39 some-host-name jupyterhub[636247]: ValueError: substring not found
mar 13 10:45:39 some-host-name jupyterhub[636247]:
mar 13 10:45:39 some-host-name jupyterhub[636247]: [E 2024-03-13 10:45:39.609 JupyterHub pages:313] Error starting server some.username: substring not found
mar 13 10:45:39 some-host-name jupyterhub[636247]: Traceback (most recent call last):
mar 13 10:45:39 some-host-name jupyterhub[636247]: None: None
mar 13 10:45:39 some-host-name jupyterhub[636247]:
mar 13 10:45:39 some-host-name jupyterhub[636247]: [W 2024-03-13 10:45:39.610 JupyterHub web:1873] 500 GET /hub/spawn (127.0.0.1): Unhandled error starting server some.username
mar 13 10:45:39 some-host-name jupyterhub[636247]: [E 2024-03-13 10:45:39.614 JupyterHub log:183] {
mar 13 10:45:39 some-host-name jupyterhub[636247]: "X-Forwarded-Proto": "http",
mar 13 10:45:39 some-host-name jupyterhub[636247]: "X-Forwarded-Port": "80",
mar 13 10:45:39 some-host-name jupyterhub[636247]: "Connection": "Keep-Alive",
mar 13 10:45:39 some-host-name jupyterhub[636247]: "X-Forwarded-Server": "some-address.local",
mar 13 10:45:39 some-host-name jupyterhub[636247]: "X-Forwarded-Host": "some-host-name.com",
mar 13 10:45:39 some-host-name jupyterhub[636247]: "X-Forwarded-For": "ip.ip.ip.ip,127.0.0.1",
mar 13 10:45:39 some-host-name jupyterhub[636247]: "Sec-Fetch-User": "?1",
mar 13 10:45:39 some-host-name jupyterhub[636247]: "Sec-Fetch-Site": "same-origin",
mar 13 10:45:39 some-host-name jupyterhub[636247]: "Sec-Fetch-Mode": "navigate",
mar 13 10:45:39 some-host-name jupyterhub[636247]: "Sec-Fetch-Dest": "document",
mar 13 10:45:39 some-host-name jupyterhub[636247]: "Upgrade-Insecure-Requests": "1",
mar 13 10:45:39 some-host-name jupyterhub[636247]: "Cookie": "_xsrf=[secret]; jupyterhub-hub-login=[secret]; _ga_PWDD85JKVC=[secret]; _ga=[secret]; jupyterhub-session-id=[secret]",
mar 13 10:45:39 some-host-name jupyterhub[636247]: "Referer": "https://some-host-name.com/hub/login",
mar 13 10:45:39 some-host-name jupyterhub[636247]: "Accept-Encoding": "gzip, deflate, br",
mar 13 10:45:39 some-host-name jupyterhub[636247]: "Accept-Language": "pl,en-US;q=0.7,en;q=0.3",
mar 13 10:45:39 some-host-name jupyterhub[636247]: "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
mar 13 10:45:39 some-host-name jupyterhub[636247]: "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:123.0) Gecko/20100101 Firefox/123.0",
mar 13 10:45:39 some-host-name jupyterhub[636247]: "Host": "some-host-name.com"
mar 13 10:45:39 some-host-name jupyterhub[636247]: }
mar 13 10:45:39 some-host-name jupyterhub[636247]: [E 2024-03-13 10:45:39.614 JupyterHub log:191] 500 GET /hub/spawn ([email protected]) 267.85ms
mar 13 10:45:41 some-host-name jupyterhub[636247]: [I 2024-03-13 10:45:41.706 JupyterHub log:191] 200 GET /hub/home ([email protected]) 8.40ms
We are using RHEL 8.9 with Python3.9.
$ grep -Ev '^$|^#' jupyterhub_config.py
c = get_config() #noqa
c.JupyterHub.spawner_class='sudospawner.SudoSpawner'
c.Spawner.cmd = '/opt/jupyterhub/app/hub/bin/sudospawner'
c.Spawner.notebook_dir = '/opt/jupyterhub/app/hub/notatniki'
c.Spawner.http_timeout = 180
$ grep -Ev '^$|^#' /etc/systemd/system/jupyterhub.service
[Unit]
Description=JupyterHub
[Service]
Name=jupyterhub
User=jupyterhub
Environment=PYTHONUNBUFFERED=1
Environment=VIRTUALENV=/opt/jupyterhub/app/hub/notatniki
Environment=PYTHONPATH=$VIRTUALENV/lib:$PYTHONPATH
Environment=PATH=/opt/jupyterhub/app/hub/bin:/opt/jupyterhub/node_modules/.bin/:/opt/jupyterjub/.local/bin:/usr/local/sbin:/usr/sbin:/usr/bin:/sbin:/bin
WorkingDirectory=/opt/jupyterhub/app/hub
ExecStart=/opt/jupyterhub/app/hub/bin/jupyterhub --JupyterHub.confirm_no_ssl=True --port=8888 --ip=127.0.0.1 --SudoSpawner.sudospawner_path='/opt/jupyterhub/app/hub/bin/sudospawner' -f /opt/jupyterhub/app/hub/etc/jupyter/jupyterhub_config.py
Restart=always
RestartSec=40
Type=simple
PIDFile=/var/run/jupyterhub.pid
[Install]
WantedBy=default.target
$PATH:
/opt/jupyterhub/app/hub/bin
/opt/jupyterhub/.local/bin
/opt/jupyterhub/bin
/usr/local/bin
/usr/bin
/usr/local/sbin
/usr/sbin
/opt/jupyterhub/node_modules/.bin
sys.path:
/opt/jupyterhub/app/hub/bin
/usr/lib64/python39.zip
/usr/lib64/python3.9
/usr/lib64/python3.9/lib-dynload
/opt/jupyterhub/app/hub/lib64/python3.9/site-packages
/opt/jupyterhub/app/hub/lib/python3.9/site-packages
sys.executable:
/opt/jupyterhub/app/hub/bin/python3.9
sys.version:
3.9.18 (main, Sep 22 2023, 17:58:34)
[GCC 8.5.0 20210514 (Red Hat 8.5.0-20)]
platform.platform():
Linux-4.18.0-513.18.1.el8_9.x86_64-x86_64-with-glibc2.28
which -a jupyter:
/opt/jupyterhub/app/hub/bin/jupyter
pip list:
Package Version
------------------------- ---------------
alembic 1.13.1
anyio 4.3.0
argon2-cffi 23.1.0
argon2-cffi-bindings 21.2.0
arrow 1.3.0
asttokens 2.4.1
async-generator 1.10
async-lru 2.0.4
attrs 23.2.0
Babel 2.14.0
beautifulsoup4 4.12.3
bleach 6.1.0
certifi 2024.2.2
certipy 0.1.3
cffi 1.16.0
charset-normalizer 3.3.2
comm 0.2.1
cryptography 42.0.5
debugpy 1.8.1
decorator 5.1.1
defusedxml 0.7.1
exceptiongroup 1.2.0
executing 2.0.1
fastjsonschema 2.19.1
fqdn 1.5.1
greenlet 3.0.3
h11 0.14.0
httpcore 1.0.4
httpx 0.27.0
idna 3.6
importlib_metadata 7.0.2
ipykernel 6.29.3
ipython 8.18.1
isoduration 20.11.0
jedi 0.19.1
Jinja2 3.1.3
json5 0.9.22
jsonpointer 2.4
jsonschema 4.21.1
jsonschema-specifications 2023.12.1
jupyter_client 8.6.0
jupyter_core 5.7.1
jupyter-events 0.9.0
jupyter-lsp 2.2.4
jupyter_server 2.13.0
jupyter_server_terminals 0.5.2
jupyter-telemetry 0.1.0
jupyterhub 4.0.2
jupyterlab 4.1.4
jupyterlab_pygments 0.3.0
jupyterlab_server 2.25.3
Mako 1.3.2
MarkupSafe 2.1.5
matplotlib-inline 0.1.6
mistune 3.0.2
nbclient 0.7.4
nbconvert 7.16.2
nbformat 5.9.2
nest-asyncio 1.6.0
notebook 7.1.1
notebook_shim 0.2.4
oauthlib 3.2.2
overrides 7.7.0
packaging 23.2
pamela 1.1.0
pandocfilters 1.5.1
parso 0.8.3
pexpect 4.9.0
pip 24.0
platformdirs 4.2.0
prometheus_client 0.20.0
prompt-toolkit 3.0.43
psutil 5.9.8
ptyprocess 0.7.0
pure-eval 0.2.2
pycparser 2.21
Pygments 2.17.2
pyOpenSSL 24.0.0
python-dateutil 2.9.0.post0
python-json-logger 2.0.7
PyYAML 6.0.1
pyzmq 25.1.2
referencing 0.33.0
requests 2.31.0
rfc3339-validator 0.1.4
rfc3986-validator 0.1.1
rpds-py 0.18.0
ruamel.yaml 0.18.6
ruamel.yaml.clib 0.2.8
Send2Trash 1.8.2
setuptools 50.3.2
six 1.16.0
sniffio 1.3.1
soupsieve 2.5
SQLAlchemy 2.0.28
stack-data 0.6.3
sudospawner 0.5.2
terminado 0.18.0
tinycss2 1.2.1
tomli 2.0.1
tornado 6.4
traitlets 5.14.1
types-python-dateutil 2.8.19.20240106
typing_extensions 4.10.0
uri-template 1.3.0
urllib3 2.2.1
wcwidth 0.2.13
webcolors 1.13
webencodings 0.5.1
websocket-client 1.7.0
websockets 12.0
zipp 3.17.0
If a user attempts to start a server when their (NFS-mounted) home directory is unavailable, the resulting error message and log message are very unhelpful, to both the user and to the administrator.
Error message to user, and log message for admins, saying something like 'I can't find that home directory', or at least something which suggests it's an account or filesystem problem rather than a jupyterhub problem.
A user attempts to log in and start a server, but their home directory is unavailable. The user sees an error message:
Spawn failed: pid
What appears in the log is:
[E 2020-09-07 18:06:22.657 JupyterHub user:638] Unhandled error starting 2130509g's server: 'pid'
[E 2020-09-07 18:06:22.674 JupyterHub gen:598] Exception in Future <Task finished name='Task-441179' coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, d
Traceback (most recent call last):
File "/opt/rh/rh-python38/root/usr/local/lib64/python3.8/site-packages/tornado/gen.py", line 593, in error_callback
future.result()
File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/jupyterhub/handlers/base.py", line 852, in finish_user_spawn
await spawn_future
File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/jupyterhub/user.py", line 656, in spawn
raise e
File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/jupyterhub/user.py", line 560, in spawn
url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/sudospawner/spawner.py", line 101, in start
self.pid = reply['pid']
KeyError: 'pid'
As you can see, this is an unconventional spelling of 'I can't find the home directory'.
Perhaps that KeyError
could be caught, and translated into the underlying cause, or something pointing in the right direction.
I haven't verified the following steps, but based on my observations, and what I did to fix this locally, the following should exhibit this behaviour.
This user had an NFS-mounted home directory, but the automount map was wrong, so the home directory couldn't be mounted, and failed after a 10s (or thereabouts) timeout.
When I fixed the automount map, and verified that ls /home/<user>
now worked, I was able to restart the server as admin, and the user was able to log in as normal.
Server OS: CentOS7
Client OS: don't know
JupyterHub version: 1.1.0
@minrk When were you thinking that we should release 0.5?
I want activate user's own conda env when jupyterlab start up. So I put conda activate ENVNAME
command to .bashrc, but not working.
let's say activate py38_user2 for user2:
user2's bashrc
conda activate py38_user2
jupyter core : 4.7.1
jupyter-notebook : 6.2.0
qtconsole : not installed
ipython : 7.24.1
ipykernel : 5.5.5
jupyter client : 6.1.11
jupyter lab : 3.0.16
nbconvert : 6.0.7
ipywidgets : 7.6.3
nbformat : 5.0.8
traitlets : 5.0.5
conda env list:
# conda environments:
#
py38_user2 * /home/user2/.conda/envs/py38_user2
base /u01/miniconda3
jupyterhub_config.py
c.Spawner.notebook_dir = '~/notebooks'
c.Spawner.default_url = '/lab'
c.LocalProcessSpawner.shell_cmd = ["bash", "-l", "-c"]
c.Spawner.args = ["--KernelSpecManager.ensure_native_kernel=False"]
c.JupyterHub.spawner_class = 'sudospawner.SudoSpawner'
c.PAMAuthenticator.open_sessions = False
c.LocalAuthenticator.create_system_users = True
c.PAMAuthenticator.add_user_cmd = ['adduser', '-m']
Is there any way to switch/activate the conda environment when the lab startup?
Hi,
When using Jupyterhub + sudospawner, $SHELL var looks ignored. I have users set with bash and shell launched is sh.
Vanilla install on CentOS 7. Running with sudospawner fails with:
[E 2015-06-30 13:46:28.826 JupyterHub web:1496] Uncaught exception POST /hub/login?next= (172.20.128.2)
HTTPServerRequest(protocol='http', host='172.20.132.114', method='POST', uri='/hub/login?next=', version='HTTP/1.1', remote_ip='172.20.128.2', headers={'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:31.0) Gecko/20100101 Firefox/31.0', 'Content-Length': '37', 'X-Forwarded-Proto': 'http', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'Referer': 'http://172.20.132.114/hub/', 'Cache-Control': 'max-age=0', 'Connection': 'close', 'Host': '172.20.132.114', 'X-Forwarded-For': '172.20.128.2', 'Accept-Language': 'en-US,en;q=0.5', 'Content-Type': 'application/x-www-form-urlencoded', 'X-Forwarded-Port': '80', 'Accept-Encoding': 'gzip, deflate'})
Traceback (most recent call last):
File "/usr/local/lib/python3.3/site-packages/tornado/web.py", line 1415, in _execute
result = yield result
File "/usr/local/lib/python3.3/site-packages/jupyterhub/handlers/login.py", line 66, in post
yield self.spawn_single_user(user)
File "/usr/local/lib/python3.3/site-packages/jupyterhub/handlers/base.py", line 260, in spawn_single_user
yield gen.with_timeout(timedelta(seconds=self.slow_spawn_timeout), f)
File "/usr/local/lib/python3.3/site-packages/jupyterhub/orm.py", line 386, in spawn
raise e
File "/usr/local/lib/python3.3/site-packages/jupyterhub/orm.py", line 367, in spawn
yield gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
File "/usr/local/lib/python3.3/site-packages/sudospawner/spawner.py", line 56, in start
reply = yield self.do(action='spawn', args=self.get_args(), env=self.env)
File "/usr/local/lib/python3.3/site-packages/sudospawner/spawner.py", line 37, in do
cmd.extend(self.sudo_args)
File "/usr/local/lib/python3.3/site-packages/IPython/utils/traitlets.py", line 424, in __get__
raise TraitError('Unexpected error in TraitType: '
IPython.utils.traitlets.TraitError: Unexpected error in TraitType: default value not set properly
Fixed by modifying /usr/local/lib/python3.3/site-packages/sudospawner/spawner.py
:
15c15
< from IPython.utils.traitlets import List, Unicode, Bool
---
> from traitlets import List, Unicode, Bool
That is all.
In
The jupyterhub-singleuser
command is explicity used there. Is there a way to configure it so that an absolute path to this can be provided.
I don't see an obvious place to change in order to make this configurable
I want to set the log level for JupyterHub.
There doesn't seem to be a way to set the log level of the sudo spawner mediator.
In my jupyterhub_config.py
I have
c.JupyterHub.log_level = 'WARN'
c.ConfigurableHTTPProxy.command = ['configurable-http-proxy', '--log-level=WARN']
c.Spawner.cmd = ['jupyterhub-singleuser', '--log-level=WARN']
I have stdout
and stderr
going to syslog. I still see debug messages from jupyterhub mediator.
Feb 13 15:57:59 box kern.debug jupyterhub[1220]: [I 180213 15:57:22 mediator:89] Spawning /boot/tp/python3/bin/sudospawner-singleuser --port=54907 '--notebook-dir="/boot/dd/jupyterhub/notebooks/cody2"' [I 2018-02-13 15:57:22.748 SingleUserNotebookApp singleuser:365] Starting jupyterhub-singleuser server version 0.8.1 [I 2018-02-13 15:57:22.752 SingleUserNotebookApp notebookapp:1585] Serving notebooks from local directory: /boot/dd/jupyterhub/notebooks/cody2 [I 2018-02-13 15:57:22.752 SingleUserNotebookApp notebookapp:
Sometimes it's not possible for users to login into hub and they get 500 error. Digging deep, we see
yield gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
File "..../site-packages/sudospawner/spawner.py", line 92, in start
reply = yield self.do(action='spawn', args=self.get_args(), env=self.get_env())
File "..../site-packages/sudospawner/spawner.py", line 59, in do
p = Subprocess(cmd, stdin=Subprocess.STREAM, stdout=Subprocess.STREAM, stderr=Subprocess.STREAM)
File ".../site-packages/tornado/process.py", line 244, in __init__
self.proc = subprocess.Popen(*args, **kwargs)
...
File ".../subprocess.py", line 1490, in _execute_child
restore_signals, start_new_session, preexec_fn)
BlockingIOError: [Errno 11] Resource temporarily unavailable
more than that, db gets corrupt, as servers
table in database holds the entry with port that is not running. Maybe it's another issue of Jupyterhub not to validate if entries in that table are actually a valid ones, but it's really annoying.
examining https://github.com/jupyterhub/jupyterhub/blob/master/jupyterhub/user.py we see lots of .commit()
calls, but not a single .rollback()
.
Trying to run jupyterhub as user jupyterhub following instructions from https://jupyterhub.readthedocs.io/en/stable/reference/config-sudo.html
Jupyterhub starts and then terminates:
sudo -u jupyterhub -g jupyterhub /opt/conda/bin/jupyterhub --Spawner.default_url='/lab' --config=/opt/conda/etc/jupyterhub/jupyterhub_config.py --debug
[D 2020-06-17 08:40:49.839 JupyterHub application:556] Looking for /opt/conda/etc/jupyterhub/jupyterhub_config in /opt/conda/etc/jupyterhub
[D 2020-06-17 08:40:49.842 JupyterHub application:578] Loaded config file: /opt/conda/etc/jupyterhub/jupyterhub_config.py
[D 2020-06-17 08:40:49.847 JupyterHub app:2079] Writing PID 691 to /opt/conda/etc/jupyterhub/jupyterhub-proxy.pid
[I 2020-06-17 08:40:49.849 JupyterHub app:2120] Using Authenticator: jupyterhub.auth.PAMAuthenticator-1.0.0
[I 2020-06-17 08:40:49.849 JupyterHub app:2120] Using Spawner: jupyterhub.spawner.LocalProcessSpawner-1.0.0
[I 2020-06-17 08:40:49.867 JupyterHub app:1257] Loading cookie_secret from /opt/conda/etc/jupyterhub/jupyterhub_cookie_secret
[D 2020-06-17 08:40:49.869 JupyterHub app:1424] Connecting to db: postgresql://jupyterhub:[redacted]@database.lan:5432/jupyterhub
[D 2020-06-17 08:40:50.028 JupyterHub orm:749] database schema version found: 4dc2d5a8c53c
[I 2020-06-17 08:40:50.075 JupyterHub proxy:460] Generating new CONFIGPROXY_AUTH_TOKEN
[I 2020-06-17 08:40:50.235 JupyterHub app:1563] Not using whitelist. Any authenticated user will be allowed.
[D 2020-06-17 08:40:50.304 JupyterHub app:1910] Loading state for l from db
[D 2020-06-17 08:40:50.308 JupyterHub app:1910] Loading state for a from db
[D 2020-06-17 08:40:50.312 JupyterHub app:1910] Loading state for x from db
[D 2020-06-17 08:40:50.316 JupyterHub app:1910] Loading state for e from db
[D 2020-06-17 08:40:50.319 JupyterHub app:1910] Loading state for alex from db
[D 2020-06-17 08:40:50.322 JupyterHub app:1926] Loaded users:
l admin
a admin
x admin
e admin
alex admin
[I 2020-06-17 08:40:50.370 JupyterHub app:2337] Hub API listening on http://127.0.0.1:8081/jupyter/hub/
[W 2020-06-17 08:40:50.371 JupyterHub proxy:516] Found proxy pid file: /opt/conda/etc/jupyterhub/jupyterhub-proxy.pid
[W 2020-06-17 08:40:50.372 JupyterHub proxy:533] Proxy still running at pid=691
Terminated
Jupyterhub does not terminate immediately after restart
Jupyterhub terminates after start
Followed instructions from
https://jupyterhub.readthedocs.io/en/stable/reference/config-sudo.html
c.Spawner.default_url = '/lab'
c.JupyterHub.bind_url = 'http://:8000/jupyter'
c.JupyterHub.cookie_secret_file = '/opt/conda/etc/jupyterhub/jupyterhub_cookie_secret'
c.PAMAuthenticator.admin_groups = {'sudo'}
c.Authenticator.admin_users = {'alex'}
c.LocalAuthenticator.create_system_users = True
pg_pass = '...'
pg_host = 'database.lan'
c.JupyterHub.db_url = 'postgresql://jupyterhub:{}@{}:5432/jupyterhub'.format(
pg_pass, pg_host
)
## File to write PID Useful for daemonizing JupyterHub.
c.JupyterHub.pid_file = '/opt/conda/etc/jupyterhub/jupyterhub-proxy.pid'
Needs to reference BSD 3 and Jupyter Contributors.
The link to instructions about creating a jupyter user seems to have disappeared. It links to this blank wiki page: https://github.com/jupyter/jupyterhub/wiki/Using-sudo-to-run-the-server-as-non-root
We use an alternative authentication and authorization package on our Linux hosts called "Centrify", which replaces the sudo command with a new command called "dzdo". Would it be possible to support the sudo command to use as a configuration parameter so that sudospawner could support using dzdo as the command to start the spawned servers? Or, would you suggest creating an alternate installation of sudospawner ("dzdospawner"?) instead? It appears that the only line that would need to actually change would be line 57 in spawner.py ("cmd = ['sudo', '-u', self.user.name]"), but I'm worried about potential future updates being missed if we do that. A config parameter, with a default of "sudo" would keep from having to duplicate the codebase in order to change the one line.
When trying to launch a jupyterhub server using sudospawner, I am attempting to pass a -E argument to sudo in order to preserve the environment
When I do this it only returns that --SudoSpawner.sudo_args argument is of type 'str' when it needs to be a list
No matter how I edit this argument, it always evaluates to a string, making this option useless
the passed argument using --SudoSpawner.sudo_args argument for jupyterhub should result in the passed string being appended to the sudo args list, instead of parsed as a list itself
Passed value is used as a list, causing an error
jupyterhub --JupyterHub.spawner_class=sudospawner.SudoSpawner --SudoSpawner.sudo_args=-E
jupyterhub --JupyterHub.spawner_class=sudospawner.SudoSpawner --SudoSpawner.sudo_args=[-E]
jupyterhub --JupyterHub.spawner_class=sudospawner.SudoSpawner --SudoSpawner.sudo_args="[-E]"
tested on
CentOS 7.9
Ubuntu 18.4
python 3.6
# paste output of `pip freeze` or `conda list` here
# jupyterhub_config.py
# paste relevant logs here, if any
I'm using the sudospawner with great effect! One issue I'm having is that logs from the spawned processes are not making it back to the jupyterhub process. I tried piping stderr back to the parent process, but that doesn't appear to help.
Hi,
after having setup sudospawner jupyterhub starts fine. But when the first user tries to login the session fails because it cannot create the etc/migrated file:
Feb 16 12:41:29 euler start_jhub[30082]: Traceback (most recent call last):
Feb 16 12:41:29 euler start_jhub[30082]: File "/opt/jhub/bin/jupyterhub-singleuser", line 8, in <module>
Feb 16 12:41:29 euler start_jhub[30082]: sys.exit(main())
Feb 16 12:41:29 euler start_jhub[30082]: File "/opt/jhub/lib/python3.10/site-packages/jupyterhub/singleuser/app.py", line 66, in main
Feb 16 12:41:29 euler start_jhub[30082]: return SingleUserLabApp.launch_instance()
Feb 16 12:41:29 euler start_jhub[30082]: File "/opt/jhub/lib/python3.10/site-packages/jupyter_core/application.py", line 277, in launch_instance
Feb 16 12:41:29 euler start_jhub[30082]: return super().launch_instance(argv=argv, **kwargs)
Feb 16 12:41:29 euler start_jhub[30082]: File "/opt/jhub/lib/python3.10/site-packages/traitlets/config/application.py", line 1042, in launch_instance
Feb 16 12:41:29 euler start_jhub[30082]: app.initialize(argv)
Feb 16 12:41:29 euler start_jhub[30082]: File "/opt/jhub/lib/python3.10/site-packages/jupyterhub/singleuser/mixins.py", line 971, in initialize
Feb 16 12:41:29 euler start_jhub[30082]: result = super().initialize(*args, **kwargs)
Feb 16 12:41:29 euler start_jhub[30082]: File "/opt/jhub/lib/python3.10/site-packages/jupyterhub/singleuser/mixins.py", line 642, in initialize
Feb 16 12:41:29 euler start_jhub[30082]: super().initialize(argv)
Feb 16 12:41:29 euler start_jhub[30082]: File "/opt/jhub/lib/python3.10/site-packages/traitlets/config/application.py", line 113, in inner
Feb 16 12:41:29 euler start_jhub[30082]: return method(app, *args, **kwargs)
Feb 16 12:41:29 euler start_jhub[30082]: File "/opt/jhub/lib/python3.10/site-packages/jupyter_server/serverapp.py", line 2507, in initialize
Feb 16 12:41:29 euler start_jhub[30082]: super().initialize(argv=argv)
Feb 16 12:41:29 euler start_jhub[30082]: File "/opt/jhub/lib/python3.10/site-packages/traitlets/config/application.py", line 113, in inner
Feb 16 12:41:29 euler start_jhub[30082]: return method(app, *args, **kwargs)
Feb 16 12:41:29 euler start_jhub[30082]: File "/opt/jhub/lib/python3.10/site-packages/jupyter_core/application.py", line 252, in initialize
Feb 16 12:41:29 euler start_jhub[30082]: self.migrate_config()
Feb 16 12:41:29 euler start_jhub[30082]: File "/opt/jhub/lib/python3.10/site-packages/jupyterhub/singleuser/mixins.py", line 434, in migrate_config
Feb 16 12:41:29 euler start_jhub[30082]: super().migrate_config()
Feb 16 12:41:29 euler start_jhub[30082]: File "/opt/jhub/lib/python3.10/site-packages/jupyter_core/application.py", line 181, in migrate_config
Feb 16 12:41:29 euler start_jhub[30082]: migrate()
Feb 16 12:41:29 euler start_jhub[30082]: File "/opt/jhub/lib/python3.10/site-packages/jupyter_core/migrate.py", line 241, in migrate
Feb 16 12:41:29 euler start_jhub[30082]: with open(os.path.join(env["jupyter_config"], "migrated"), "w", encoding="utf-8") as f:
Feb 16 12:41:29 euler start_jhub[30082]: PermissionError: [Errno 13] Permission denied: '/opt/jhub/etc/migrated'
Feb 16 12:41:30 euler start_jhub[30082]: [E 230216 12:41:30 mediator:43] mediator result: {'ok': False, 'error': 'Exited with status: 1'}
Feb 16 12:41:30 euler start_jhub[30082]: [E 2023-02-16 12:41:30.249 JupyterHub user:833] Unhandled error starting fst's server: 'pid'
When I set /opt/jhub/etc to permissions 1777 the startup works find and /opt/jhub/etc/migrated is created with user fst as owner.
That doesn't make sense, I guess, because etc/ shouldn't be writeable for everyone.
Is there a way to force the migrated file to be created in some tmp dir? Or is this just a bug?
Once the file has been created, other users can login and start a session, too. I can also touch the migrated file as root (being empty), then the login works, too. But if doesn't exist yet, it should be created as the user running jupyterhub and not as the user who first logs in.
This happens with jupyterhub 3.1.1 and pthon 3.10.8 on SuSE Linux Enterprise 15 SP4
# jupyterhub_config.py
c.Spawner.cmd = ['jupyter-labhub']
c.Spawner.default_url = '/lab'
c.JupyterHub.cleanup_servers = False
c.ConfigurableHTTPProxy.should_start = False
c.JupyterHub.cookie_secret_file = '/etc/jhub/jupyterhub_cookie_secret'
c.ConfigurableHTTPProxy.api_url = 'http://localhost:8791'
c.JupyterHub.pid_file = '/run/jhub/jhub.pid'
c.JupyterHub.db_url = '/var/lib/jhub/jupyterhub.sqlite'
c.JupyterHub.spawner_class = 'sudospawner.SudoSpawner'
c.SudoSpawner.sudospawner_path = '/opt/jhub/bin/sudospawner'
There is no publish workflow setup for this project yet.
This is what we should add I think: https://github.com/jupyterhub/jupyterhub-idle-culler/blob/master/.github/workflows/publish.yml
@minrk @yuvipanda could you add jupyterhub-bot to https://pypi.org/project/sudospawner/? Then I'll setup a deployment token for this project and a GitHub secret named PYPI_PASSWORD
like used in the workflow referenced above.
I'm running the JupyterHub with SudoSpawner and PAMAuthentication in my organization K8s environment, where I'm mounting my pod to PVC. PVC does not have root access, so the command used by the SudoSpawner ['sudo', '-u', 'user_1', '-nH', '/opt/conda/bin/sudospawner']
fails as it starts with sudo command.
Following is my Jupyter logs (the logs are from where the user inputs the credentials):
[D 2019-08-29 13:05:15.663 JupyterHub base:482] Setting cookie for user_1: jupyterhub-hub-login
[D 2019-08-29 13:05:15.664 JupyterHub base:478] Setting cookie jupyterhub-hub-login: {'httponly': True, 'path': '/hub/'}
[I 2019-08-29 13:05:15.664 JupyterHub base:663] User logged in: user_1
[D 2019-08-29 13:05:15.664 JupyterHub user:240] Creating <class 'sudospawner.spawner.SudoSpawner'> for user_1:
[W 2019-08-29 13:05:15.669 JupyterHub configurable:168] Config option `mediator_log_level` not recognized by `SudoSpawner`.
[I 2019-08-29 13:05:15.670 JupyterHub log:174] 302 POST /hub/login?next= -> /hub/spawn (user_1@::ffff:10.192.126.0) 75.50ms
[D 2019-08-29 13:05:16.208 JupyterHub pages:165] Triggering spawn with default options for user_1
[D 2019-08-29 13:05:16.209 JupyterHub base:780] Initiating spawn for user_1
[D 2019-08-29 13:05:16.209 JupyterHub base:787] 0/100 concurrent spawns
[D 2019-08-29 13:05:16.209 JupyterHub base:792] 0 active servers
sh: 1: cannot create /run/motd.dynamic.new: Permission denied
[W 2019-08-29 13:05:16.245 JupyterHub auth:941] Failed to open PAM session for user_1: [PAM Error 14] Cannot make/remove an entry for the specified session
[W 2019-08-29 13:05:16.245 JupyterHub auth:942] Disabling PAM sessions from now on.
[D 2019-08-29 13:05:16.253 JupyterHub user:542] Calling Spawner.start for user_1
[D 2019-08-29 13:05:16.254 JupyterHub spawner:63] Spawning ['sudo', '-u', 'user_1', '-nH', '/opt/conda/bin/sudospawner']
sudo: effective uid is not 0, is /usr/bin/sudo on a file system with the 'nosuid' option set or an NFS file system without root privileges?
[E 2019-08-29 13:05:16.264 JupyterHub spawner:83] Failed to get JSON result from mediator: ''
[E 2019-08-29 13:05:16.264 JupyterHub user:626] Unhandled error starting user_1's server: substring not found
[D 2019-08-29 13:05:16.264 JupyterHub user:724] Stopping user_1
[D 2019-08-29 13:05:16.273 JupyterHub user:752] Deleting oauth client jupyterhub-user-user_1
[D 2019-08-29 13:05:16.281 JupyterHub user:755] Finished stopping user_1
[I 2019-08-29 13:05:16.287 JupyterHub log:174] 302 GET /hub/spawn -> /hub/spawn-pending/user_1 (user_1@::ffff:10.193.236.0) 85.62ms
ERROR:asyncio:Task exception was never retrieved
future: <Task finished coro=<BaseHandler.spawn_single_user() done, defined at /opt/conda/lib/python3.7/site-packages/jupyterhub/handlers/base.py:697> exception=ValueError('substring not found')>
Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/jupyterhub/handlers/base.py", line 889, in spawn_single_user
timedelta(seconds=self.slow_spawn_timeout), finish_spawn_future
File "/opt/conda/lib/python3.7/site-packages/jupyterhub/handlers/base.py", line 807, in finish_user_spawn
await spawn_future
File "/opt/conda/lib/python3.7/site-packages/jupyterhub/user.py", line 642, in spawn
raise e
File "/opt/conda/lib/python3.7/site-packages/jupyterhub/user.py", line 546, in spawn
url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
File "/opt/conda/lib/python3.7/site-packages/tornado/gen.py", line 742, in run
yielded = self.gen.throw(*exc_info) # type: ignore
File "/opt/conda/lib/python3.7/site-packages/sudospawner/spawner.py", line 92, in start
reply = yield self.do(action='spawn', args=self.get_args(), env=self.get_env())
File "/opt/conda/lib/python3.7/site-packages/tornado/gen.py", line 735, in run
value = future.result()
File "/opt/conda/lib/python3.7/site-packages/tornado/gen.py", line 748, in run
yielded = self.gen.send(value)
File "/opt/conda/lib/python3.7/site-packages/sudospawner/spawner.py", line 80, in do
data_str = data_str[data_str.index('{'):data_str.rindex('}')+1]
ValueError: substring not found
[E 2019-08-29 13:05:16.801 JupyterHub pages:284] Previous spawn for user_1 failed: substring not found
[E 2019-08-29 13:05:16.807 JupyterHub log:166] {
"X-Forwarded-Host": "127.0.0.1:30235",
"X-Forwarded-Proto": "http",
"X-Forwarded-Port": "30235",
"X-Forwarded-For": "::ffff:10.193.236.0",
"Upgrade-Insecure-Requests": "1",
"Connection": "close",
"Dnt": "1",
"Cookie": "jupyterhub-hub-login=[secret]; jupyterhub-session-id=[secret]",
"Referer": "http://127.0.0.1:30235/hub/login",
"Accept-Encoding": "gzip, deflate",
"Accept-Language": "en-US,en;q=0.5",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0",
"Host": "127.0.0.1:30235"
}
[E 2019-08-29 13:05:16.807 JupyterHub log:174] 500 GET /hub/spawn-pending/user_1 (user_1@::ffff:10.193.236.0) 13.06ms
Please suggest me a fix for this issue or if it's not being able to fix, a possible alternative for my approach. Thanks in advance.
Hi,
I am running JupyterHub as separate non-root user with systemd without sudospawner. Instead I leverage the systemd AmbientCapabilities option. My jupyterhub.service file looks like this:
[Unit]
Description=JupyterHub
After=syslog.target network.target podman-chp.service
[Service]
User=jupyterhub
AmbientCapabilities=CAP_SETUID CAP_SETGID
Environment="PATH=/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/jupyterhub/bin"
Environment="CONFIGPROXY_AUTH_TOKEN=<secret>"
ExecStart=/opt/jupyterhub/bin/jupyterhub -f /opt/jupyterhub/etc/jupyterhub/jupyterhub_config.py
[Install]
WantedBy=multi-user.target
With this I can use the PAMAuthenticator for logging in and use the standard spawner for creating the single user notebooks. My question now is: Am I missing some feature of the sudospawner configuration? Or do I get the same effects with my configuration?
NOTE: I am running on CentOS 8. Maybe this makes a difference?
I set up a sudospawner-singleuser
script alongside the sudospawner
and jupyterhub-singleuser
scripts in /usr/local/bin
, as documented in Custom Singleuser Launch Command, but while the sudospawner
script does get launched, the sudospawner-singleuser
script is not.
I added this to /usr/local/bin/sudospawner
:
import os
os.system('echo "Starting sudospawner" | wall')
and set /usr/local/bin/sudospawner-singleuser
to:
#!/bin/bash -l
echo "Starting sudospawner-singleuser" | wall
Then after restarting jupyterhub
and logging in, I see this in my terminal:
Broadcast Message from philipsd6@myhost
(somewhere) at 13:23 ...
Starting sudospawner
But I don't see "Starting sudospawner-singleuser", so it must not be executed. Both scripts have the same ownership/permissions.
Even if c.Spawner.env_keep
is set as expected, jupyterhub does not pass these variables to sudospawner and thus sudospawner fails with
...
[W 2021-09-06 05:20:18.191 JupyterHub auth:1079] Failed to open PAM session for foobar: [PAM Error 14] Cannot make/remove an entry for the specified session
[W 2021-09-06 05:20:18.192 JupyterHub auth:1080] Disabling PAM sessions from now on.
Traceback (most recent call last):
File "/usr/local/jupyter/bin/sudospawner", line 8, in <module>
from sudospawner import mediator
ModuleNotFoundError: No module named 'sudospawner'
[E 2021-09-06 05:20:18.275 JupyterHub spawner:95] Failed to get JSON result from mediator: ''
[E 2021-09-06 05:20:18.276 JupyterHub user:720] Unhandled error starting foobar's server: substring not found
[W 2021-09-06 05:20:18.327 JupyterHub web:1787] 500 GET /jupyter/n00-00/hub/spawn/foobar (::ffff:10.2.0.10): Error in Authenticator.pre_spawn_start: ValueError substring not found
...
sudospawner should run.
sudospawner invocation fails.
Use any vanilla ubuntu 18.04 install and execute as unprivileged user (example uses user admin
which belongs to group staff
) - copy and paste should work:
bash
sudo apt-get install npm nodejs \
python3-pip python3-setuptools python3-wheel python3-cffi libffi-dev \
python3-dev python3-jinja2 python3-six python3-dbg python3-venv
cat >/var/tmp/jupyter.env<<EOF
export JUPYTER_PATH=/usr/local/jupyter
export PYTHONUSERBASE=\${JUPYTER_PATH}
PATH=\${JUPYTER_PATH}/bin:\${PATH}
export PYTHONPATH=\${JUPYTER_PATH}/lib/python3.6/site-packages
export NPM_CONFIG_GLOBALCONFIG=\${JUPYTER_PATH}/npmrc
EOF
source /var/tmp/jupyter.env
sudo mkdir -p ${JUPYTER_PATH}
sudo rm -rf ${JUPYTER_PATH}/*
sudo chown admin:staff ${JUPYTER_PATH}
printf "prefix=${JUPYTER_PATH}\n" >${JUPYTER_PATH}/npmrc
cp /var/tmp/jupyter.env ${JUPYTER_PATH}/env
# just to make sure everything is clean
rm -rf ~/{.local,.cache/pip,node_modules}
python3 -m pip install --upgrade pip
# install jupyter
pip3 install jupyter
python3 -m ipykernel install --prefix=${JUPYTER_PATH}
sudo mkdir -p /usr/share/jupyter
sudo ln -sf ${JUPYTER_PATH}/share/jupyter/kernels /usr/share/jupyter/kernels
# install jupyterhub
pip3 install jupyterhub
pip3 install git+https://github.com/jupyter/sudospawner
cat >${JUPYTER_PATH}/jupyterhub_config.py<<EOF
c.JupyterHub.spawner_class = 'sudospawner.SudoSpawner'
c.SudoSpawner.sudospawner_path = '/usr/local/jupyter/bin/sudospawner'
c.Spawner.env_keep = [ 'PATH', 'PYTHONPATH', 'LD_LIBRARY_PATH', 'JUPITER_PATH' ]
import os
import sys
for var in os.environ:
print (var, os.environ[var], file=sys.stderr)
c.Spawner.env_keep.append(var)
print ('cfg DONE.', file=sys.stderr)
c.Spawner.notebook_dir = '~/'
c.Spawner.args = ['--NotebookApp.allow_origin=*']
c.NotebookApp.allow_origin = '*'
c.JupyterHub.base_url = '/jupyter/n00-00'
c.Authenticator.admin_users = set(['admin'])
EOF
# install the proxy
npm install --only=production --no-optional --global \
[email protected]
# let user juppy run the service
sudo useradd -u 123 -g 4 -c 'JupiterHub Service' -m -r juppy
printf "juppy ALL=(%jupyter) NOPASSWD: ${JUPYTER_PATH}/bin/sudospawner\n" >/tmp/x
sudo mv /tmp/x /etc/sudoers.d/jupyterHub
# make sure /etc/nsswitch.conf contians a something like this:
# `sudoers: files`
cat >/tmp/x<<EOF
#!/bin/bash
cd ~ && pwd || exit 1
source ${JUPYTER_PATH}/env
${JUPYTER_PATH}/bin/jupyterhub -f ${JUPYTER_PATH}/jupyterhub_config.py >jupyterhub.log 2>&1 &
EOF
chmod 755 /tmp/x
sudo mv /tmp/x /etc/init.d/jupyterhub
# that's it wrt. the setup
# optionally for debug one may overwrite sudospwaner, e.g.
cat >>${JUPYTER_PATH}/bin/sudospawner<<EOF
#!/usr/bin/python3
import os
import sys
for var in os.environ:
print (var, os.environ[var],file=sys.stderr)
from sudospawner import mediator
mediator.main()
EOF
# to start the service manually, run e.g.:
sudo /bin/su -c /etc/init.d/jupyterhub juppy
# and go to the following URL and try to login
firefox http://${yourbox}:8000//jupyter/n00-00/
# and finally have a look at the log:
cat ~juppy/jupyterhub.log
As we had problems with the jupyterlab processes spawned from jupyterhub, we added
c.Spawner.args = ["--debug"]
to jupyterhub_config.py to get debugging messages from the singleuser processes in the systemd journal.
This does work when we set this option, then restart jupyterhub, then stop and start the servers. Then we see /usr/bin/python3.10 /opt/jhub/bin/jupyterhub-singleuser --debug
in the linux process list and get all debugging messages in the journal.
But if we restart jupyterhub again afterwards, the debugging output gets lost. The jupyterhub-singleuser process keeps running and can be used in the browser, but its debugging messages no longer are caught by the jupyterhub process and thus no longer sent to the systemd journal.
Not sure if this is a bug or if it's just not possible to reestablish the connection to the output channel of the singleuser processes.
To workaround, I changed our sudospawner-singleuser script so that it calls
exec "$(dirname "$0")/jupyterhub-singleuser" $@ 2>&1 | /usr/bin/logger -t jupyterlab
This way the singleuser processes do the logging to the journal themselves and that keeps working after restarting jupyterhub.
So I've two questions:
a) Is the described behaviour a bug or is just impossible to further collect output of the singleuser processes after restarting jupyterhub (because of broken pipes or sth...)?
b) Could my workaround cause problems because I redirect stdout/stderr on my own?
Output from pam_oddjob_mkhomedir causes an exception and 500 error the first time users log in.
[E 2016-02-02 17:16:20.149 JupyterHub orm:382] Unhandled error starting {user}'s server: Extra data: line 1 column 14 (char 13)
Various functions in the LocalProcessSpawner
class assume privileged access by using the builtin functions, e.g.: shutil.rmtree
and os.makedirs
.
Being able to use sudospawner with SSL enabled for all internal communication
pre_spawn_start
fails with permission denied when calling move_cert
.
ubuntu 20.04
$ jupyterhub --version
1.4.2
$ python3 --version
Python 3.8.10
acme==1.1.0
alembic==1.7.4
anyio==3.3.4
appdirs==1.4.3
argon2-cffi==21.1.0
async-generator==1.10
attrs==19.3.0
Automat==0.8.0
Babel==2.9.1
backcall==0.2.0
bleach==4.1.0
blinker==1.4
bottle==0.12.15
certbot==0.40.0
certbot-apache==0.39.0
certifi==2019.11.28
certipy==0.1.3
cffi==1.15.0
chardet==3.0.4
Click==7.0
colorama==0.4.3
command-not-found==0.3
ConfigArgParse==0.13.0
configobj==5.0.6
constantly==15.1.0
cryptography==2.8
cycler==0.10.0
dbus-python==1.2.16
debugpy==1.5.1
decorator==5.1.0
defusedxml==0.7.1
devscripts===2.20.2ubuntu2
distlib==0.3.0
distro==1.4.0
distro-info===0.23ubuntu1
docker==4.1.0
entrypoints==0.3
filelock==3.0.12
future==0.18.2
Glances==3.1.3
gpg===1.13.1-unknown
greenlet==1.1.2
httplib2==0.14.0
hyperlink==19.0.0
idna==2.8
importlib-metadata==1.5.0
importlib-resources==5.3.0
incremental==16.10.1
influxdb==5.2.0
ipykernel==6.4.1
ipython==7.28.0
ipython-genutils==0.2.0
jedi==0.18.0
Jinja2==3.0.2
josepy==1.2.0
json5==0.9.6
jsonschema==4.1.0
jupyter-client==7.0.6
jupyter-core==4.8.1
jupyter-server==1.11.1
jupyter-telemetry==0.1.0
jupyterhub==1.4.2
jupyterlab==3.2.0
jupyterlab-pygments==0.1.2
jupyterlab-server==2.8.2
keyring==18.0.1
kiwisolver==1.0.1
language-selector==0.1
launchpadlib==1.10.13
lazr.restfulclient==0.14.2
lazr.uri==1.0.3
Mako==1.1.5
MarkupSafe==2.0.1
matplotlib==3.1.2
matplotlib-inline==0.1.3
meld==3.20.2
mistune==0.8.4
mock==3.0.5
more-itertools==4.2.0
nbclassic==0.3.2
nbclient==0.5.4
nbconvert==6.2.0
nbformat==5.1.3
nest-asyncio==1.5.1
netifaces==0.10.4
notebook==6.4.4
numpy==1.17.4
oauthenticator==14.2.0
oauthlib==3.1.0
olefile==0.46
packaging==21.0
pamela==1.0.0
pandocfilters==1.5.0
parsedatetime==2.4
parso==0.8.2
pbr==5.4.5
pexpect==4.8.0
pickleshare==0.7.5
Pillow==7.0.0
ply==3.11
prometheus-client==0.11.0
prompt-toolkit==3.0.20
psutil==5.5.1
ptyprocess==0.7.0
pyasn1==0.4.2
pyasn1-modules==0.2.1
pycairo==1.16.2
pycparser==2.20
pycryptodomex==3.6.1
pycurl==7.43.0.2
pyflakes==2.1.1
Pygments==2.10.0
PyGObject==3.36.0
PyHamcrest==1.9.0
PyICU==2.4.2
PyJWT==1.7.1
pyOpenSSL==19.0.0
pyparsing==2.4.6
pyRFC3339==1.1
pyrsistent==0.18.0
pysmi==0.3.2
pysnmp==4.4.6
pystache==0.5.4
python-apt==2.0.0+ubuntu0.20.4.5
python-augeas==0.5.0
python-dateutil==2.7.3
python-debian===0.1.36ubuntu1
python-json-logger==2.0.2
python-magic==0.4.16
pytz==2019.3
pyxdg==0.26
PyYAML==5.3.1
pyzmq==22.3.0
requests==2.22.0
requests-toolbelt==0.8.0
requests-unixsocket==0.2.0
ruamel.yaml==0.17.16
ruamel.yaml.clib==0.2.6
SecretStorage==2.3.1
Send2Trash==1.8.0
service-identity==18.1.0
simplejson==3.16.0
six==1.14.0
sniffio==1.2.0
SQLAlchemy==1.4.25
ssh-import-id==5.10
sudospawner==0.5.2
supervisor==4.1.0
systemd-python==234
terminado==0.12.1
testpath==0.5.0
tornado==6.1
traitlets==5.1.0
Twisted==18.9.0
ubuntu-advantage-tools==20.3
ufw==0.36
unattended-upgrades==0.1
unidiff==0.5.5
urllib3==1.25.8
virtualenv==20.0.17
wadllib==1.3.3
wcwidth==0.2.5
webencodings==0.5.1
websocket-client==0.53.0
zipp==3.6.0
zope.component==4.3.0
zope.event==4.4
zope.hookable==5.0.0
zope.interface==4.7.1
c.JupyterHub.internal_ssl = True
c.JupyterHub.spawner_class = 'sudospawner.SudoSpawner'
c.Spawner.debug = True
23:58:52.770 [ConfigProxy] info: Adding route / -> https://127.0.0.1:8081
23:58:52.771 [ConfigProxy] info: Route added / -> https://127.0.0.1:8081
23:58:52.772 [ConfigProxy] info: 201 POST /api/routes/
[I 2021-10-18 23:58:52.773 JupyterHub app:2849] JupyterHub is now running at https://:8000
[I 2021-10-18 23:58:53.634 JupyterHub log:189] 302 GET / -> /hub/ (@::ffff:130.44.171.90) 1.47ms
[I 2021-10-18 23:58:53.737 JupyterHub log:189] 302 GET /hub/ -> /hub/spawn (edgarcosta@::ffff:130.44.171.90) 16.68ms
[I 2021-10-18 23:58:53.995 JupyterHub provider:574] Creating oauth client jupyterhub-user-edgarcosta
[I 2021-10-18 23:58:54.115 JupyterHub spawner:950] Creating certs for edgarcosta: DNS:localhost;IP:127.0.0.1
[E 2021-10-18 23:58:54.305 JupyterHub user:718] Unhandled error starting edgarcosta's server: [Errno 13] Permission denied: '/home/edgarcosta/.jupyterhub'
[W 2021-10-18 23:58:54.476 JupyterHub web:1787] 500 GET /hub/spawn (::ffff:130.44.171.90): Error in Authenticator.pre_spawn_start: PermissionError [Errno 13] Permission denied: '/home/edgarcosta/.jupyterhub'
jupyterlab-hub no longer exists in 2.0.0 and creating a sudospawner-singleuser is no longer necessary. sudospawner will work by adding --Spawner.default_url='/lab' to the jupyterlab executable
In the case where I want to keep my notebook server environment separate from my jupyterhub environment, it would be helpful to point to a script that activates the desired environment and then delegates the command line params to jupyterhub-singleuser. As it stands, I have to generate a script with that name, rename the actual juputerhub-singleuser script to something else, and then delegate to it.
I notice that the code comments say the jupyterhub-singleuser script is hardcoded "for security". Is there a known risk to respecting the Spawner.cmd traitlet (https://github.com/jupyterhub/jupyterhub/blob/2dd3463ea89e8be435fabe245e17f8e127252c3f/jupyterhub/spawner.py#L140) in the sudospawner? On first glance, it appears that only admin users with write access to the jupyterhub config would have to ability to set that executable command.
Dear all,
I try to setup jupyterhub using a sudospawer and the default PAM authenticator in the context of multiple conda environment (using nb_conda_kernels
).
To do so:
jupyter-admin
--> /home/jupyter-admin
/home/jupyter-admin
I create and activate a new conda environment notebook-env
using the following conda envionment file:name: notebook-env
channels:
- conda-forge
- defaults
dependencies:
- notebook
- jupyterlab
- jupyterhub
- configurable-http-proxy
- sudospawner
- nb_conda_kernels
/etc/sudoers
fileCmnd_Alias JUPYTER_COMMAND = /home/jupyter-admin/miniconda3/envs/notebook-env/bin/sudospawner
jupyter-admin ALL=(ALL:ALL) NOPASSWD:JUPYTER_COMMAND
import os
import sys
import tempfile
bin_dir = os.path.split(sys.executable)[0]
spawner = os.path.join(bin_dir, 'sudospawner')
database = os.path.join(tempfile.gettempdir(), "jupyterhub", "jupyterhub.sqlite")
if not os.path.isdir(os.path.dirname(database)):
os.makedirs(os.path.dirname(database))
# Keep some environment variables if needed:
for var in ("PYTHONHOME", "PYTHONPATH"):
if var in os.environ:
c.SudoSpawner.env_keep.append(var)
# Use the sudo spawner for launching the server under a user name different than root
c.JupyterHub.spawner_class = 'sudospawner.SudoSpawner'
c.JupyterHub.ip = '0.0.0.0'
c.JupyterHub.db_url = database
c.SudoSpawner.sudospawner_path = spawner
c.Authenticator.admin_users = {"pellegrini"}
c.Spawner.default_url = "/lab"
c.Spawner.cmd = ['jupyter-labhub']
when starting my jupyterhub server, I get the following error when logging under pellegrini
sh: 1: cannot create /run/motd.dynamic.new: Permission denied
The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.
The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.
[W 2019-05-06 16:37:03.423 JupyterHub auth:642] Failed to open PAM session for pellegrini: [PAM Error 14] Cannot make/remove an entry for the specified session
[W 2019-05-06 16:37:03.423 JupyterHub auth:643] Disabling PAM sessions from now on.
[D 2019-05-06 16:37:03.442 JupyterHub spawner:63] Spawning ['sudo', '-u', 'pellegrini', '-nH', '/home/jupyter-admin/miniconda3/envs/visa-jupyter/bin/sudospawner']
Strictly speaking this is not a true error in the sense that I can eventually login but I would like to be sure that I did not set something wrong.
I read some related issue (e.g. #51) and tried the proposed recipe but it did not help.
Would you have any ideas ?
thanks in advance
Eric
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.