ideonate / jhsingle-native-proxy Goto Github PK

I am having trouble with dash apps to respond with the Error report from ContainDS Dashboards. I get an intermittent 500 Internal server error instead. I have tried using the --ready-timeout flag hoping that it would help, but no luck. I think that the child process is getting killed before the error could propagate up.

`Traceback (most recent call last):
File "/usr/local/lib64/python3.11/site-packages/tornado/web.py", line 1713, in _execute
result = await result
^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/jhsingle_native_proxy/websocket.py", line 102, in get
return await self.http_get(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/jhsingle_native_proxy/proxyhandlers.py", line 849, in http_get
return await self.proxy(self.port, path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/jhsingle_native_proxy/proxyhandlers.py", line 843, in proxy
return await self.oauth_proxy(port, path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/jhsingle_native_proxy/proxyhandlers.py", line 798, in oauth_proxy
return await self.core_proxy(port, path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/jhsingle_native_proxy/proxyhandlers.py", line 829, in core_proxy
if not await self.ensure_process():
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/jhsingle_native_proxy/proxyhandlers.py", line 762, in ensure_process
await proc.kill()
File "/usr/local/lib/python3.11/site-packages/simpervisor/process.py", line 167, in kill
return await self._signal_and_wait(signal.SIGKILL)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/simpervisor/process.py", line 140, in _signal_and_wait
self.proc.send_signal(signum)
File "/usr/lib64/python3.11/asyncio/subprocess.py", line 140, in send_signal
self._transport.send_signal(signal)
File "/usr/lib64/python3.11/asyncio/base_subprocess.py", line 145, in send_signal
self._check_proc()
File "/usr/lib64/python3.11/asyncio/base_subprocess.py", line 142, in _check_proc
raise ProcessLookupError()```

Expected behavior:
The ensure_process method should capture error details, and allow the exception to propagate up for higher-level exception handling so that the error is appropriately reported instead of terminating the process prematurely, leading to generic 500 Internal Server Errors.

Expected response:

Just wanted to see if we can adjust the behavior within the if not is_ready block to let the error handled by the higher level handler? Is there any other solution we could try?

Have you thought about depending on https://github.com/jupyterhub/jupyter-server-proxy as an alternative to having to copy and paste the code from it?

Was wondering if something needs to change in jupyter-server-rpoxy to make that possible and other thoughts on this.

One advantage would be that jhsingle-native-proxy gets all the improvements and bug fixes "automatically".

Recent update of simpervisor from 0.4 to 1.0 breaks apps wrapped in jhsingle-native-proxy.

In the immediate term, we need to pin dependency for the current version to simpervisor=0.4 or simpervisor<1 to stop new builds from breaking, but in the longer term updating to reflect changes in simpervisor 1.0 will be required.

I'll be happy to open PR(s)!

I'm using this with jupyterhub and cdsdashboards (the combo is perfect for my use case) but I was having issues with connect timeouts for notebooks that were a bit more computationally expensive. It was hitting the tornado 20 sec default request timeout. My quick and dirty solution was to change the proxy_request_options function to return dict(follow_redirects=False, request_timeout=300) I someone mention issues with timeouts and thought posting this might help others. Hardwiring it here fixes my problem, but maybe a more elegant solution is to have jupyterhub's c.Spawner.http_timeout passed along, though I just started playing with all of this last week and don't know the pros and cons of that approach.

Thanks for the awesome work

Hey thanks for the great tool.

Using cdsdashboard, I got the following error when starting a dashboard:

[I 2020-06-26 09:57:43.527 JupyterHub proxy:262] Adding user fcollonval to proxy /user/fcollonval/dash-sin-and-cos-graph/ => http://127.0.0.1:42893
ERROR:tornado.application:Uncaught exception GET /user/fcollonval/dash-sin-and-cos-graph/ (127.0.0.1)
HTTPServerRequest(protocol='http', host='127.0.0.1:42893', method='GET', uri='/user/fcollonval/dash-sin-and-cos-graph/', version='HTTP/1.1', remote_ip='127.0.0.1')
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/tornado/web.py", line 1703, in _execute
    result = await result
  File "/opt/conda/lib/python3.7/site-packages/jhsingle_native_proxy/websocket.py", line 94, in get
    return await self.http_get(*args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/jhsingle_native_proxy/proxyhandlers.py", line 592, in http_get
    return await self.proxy(self.port, path)
  File "/opt/conda/lib/python3.7/site-packages/jhsingle_native_proxy/proxyhandlers.py", line 586, in proxy
    return await self.oauth_proxy(port, path)
TypeError: object NoneType can't be used in 'await' expression

Installation:

In Docker with base docker = debian:buster-slim

Python package of interest:

python 3.7.0
cdsdashboards          0.0.19             
jhsingle-native-proxy  0.3.1              
jupyter-archive        0.6.2              
jupyter-client         6.1.3              
jupyter-core           4.6.3              
jupyter-server         0.1.1              
jupyter-telemetry      0.0.5              
jupyterhub             1.1.0              
jupyterlab             2.1.5              
jupyterlab-git         0.20.0             
jupyterlab-pygments    0.1.1              
jupyterlab-server      1.1.5

Getting the following error "Error: No such option: --SingleUserNotebookApp.default_url" when using this package in a single user container image with Jupyterhub. The image is loaded as a profile list of single user environments.

Hey,

I've been using the jhsingle-native-proxy to expose voila to an authenticated JupyterHub and its working great. However one thing that's been noticed is that we no longer get the spinning loader and instead we have a blank page until the notebook has been fully executed. For some of our more complex applications this can be 15-20 seconds of load time with no visible indicator to users that work is progressing,

After digging into the details, it looks like voila flushes progressively so that the user is presented with an initial response then progressively sees progress until the notebook execution has completed.

However when sat behind the proxy, we're waiting for the response to be completed before writing back out. This means that nothing is rendered to the user until execution has been completed.

Is this something that has already been witnessed with the proxy and is there a known way to work around it? If not, any suggestion on an approach to look to support it?

Hi Dan,

I'm trying to use the jhsingle-native-proxy on top of voila. The authentication works well, until I noticed that when voila is used without the proxy, the behavior is a little different.

A template is being used that shows the progress of the running notebook while voila is processing it, a template was created that is passed to voilà using the --template flag, this work well along with jupyterhub. However, with jhsingle-native-proxy the notebook is not shown until voilà finishes its execution, therefore, we are not capable to see the progress of the notebook itself. This is very important due to some notebooks taking a while to run, the template shows the progress, while with jhsingle-native-proxy shows a blank page and it only shows the result of converted notebooks when it finishes. For some notebooks, the user will have to wait until the notebooks completes, the user won't know if the notebook is running or not. Is there a a solution to this problem? Any ideas or suggestions?

Best regards.

I am using Jhsingle-native-proxy locally and with Docker but despite using the --debug option it seems that the DEBUG and INFO log messages are not being displayed in my console. Currently Jhsingle-native-proxy does not seem to be logging/outputting any log messages seen in proxyhandlers.py to my console.

I have also noticed that Jhsingle-native-proxy is using the app_log logger from tornado.log, whose level is set to either DEBUG or INFO depending on whether the --debug option is used.

Is there a way to get the log messages to display in the console and fix this issue?

Hi @danlester , thanks for this and cdsdashboards! Deployed cdsdashboards to a python 3.9 environment and got the following stack trace when trying to deploy a dashboard

Traceback (most recent call last):
  File "/opt/envs/system/jupyterhub/lib/python3.9/site-packages/tornado/web.py", line 1704, in _execute
    result = await result
  File "/opt/envs/system/jupyterhub/lib/python3.9/site-packages/jhsingle_native_proxy/websocket.py", line 103, in get
    return await self.http_get(*args, **kwargs)
  File "/opt/envs/system/jupyterhub/lib/python3.9/site-packages/jhsingle_native_proxy/proxyhandlers.py", line 724, in http_get
    return await self.proxy(self.port, path)
  File "/opt/envs/system/jupyterhub/lib/python3.9/site-packages/jhsingle_native_proxy/proxyhandlers.py", line 718, in proxy
    return await self.oauth_proxy(port, path)
  File "/opt/envs/system/jupyterhub/lib/python3.9/site-packages/jhsingle_native_proxy/proxyhandlers.py", line 673, in oauth_proxy
    return await self.core_proxy(port, path)
  File "/opt/envs/system/jupyterhub/lib/python3.9/site-packages/jhsingle_native_proxy/proxyhandlers.py", line 704, in core_proxy
    if not await self.ensure_process():
  File "/opt/envs/system/jupyterhub/lib/python3.9/site-packages/jhsingle_native_proxy/proxyhandlers.py", line 618, in ensure_process
    await proc.start()
  File "/opt/envs/system/jupyterhub/lib/python3.9/site-packages/simpervisor/process.py", line 83, in start
    with (await self._proc_lock):
TypeError: object Lock can't be used in 'await' expression

It's resolved by reverting back to python 3.8. Mostly wanted to let you know about future incompatibilities. Regrettably, I don't have any bandwidth to submit a PR at this time.

Hi,
I would like to use your dockerfile with voila-reveal (and with voila-gridstacks) it requires to pass an extra keyword
--template=reveal (or gridstack).

I've tried to add {--}template=reveal it but it does seem to work.
Could you tell how to proceed to be able to make it work?

Best

Thanks for this package, it seems to be exactly what we're looking for. We have a web app developed using Voila, and a running TLJH instance on AWS. We'd like to display the web app when users login to the hub (as opposed to a lab dashboard or notebook tree). However, we're not totally clear on how to make this function.

We've been testing locally with a Docker install of TLJH, and from the github page, it seems that we can just have python -m jhsingle_native_proxy.main --destport 0 voila ./WebApp.ipynb {--}port={port} {--}no-browser {--}Voila.server_url=/ {--}Voila.base_url={base_url}/ {--}debug running and be forwarded to the rendered webapp. But it doesn't seem that anything in particular happens.

Is it possible to expand on what exactly needs to be done to get to the rendered Voila web app?

Thanks again!

Having launch with binder button for examples would be greatly appreciated.

Thank you.

Hi, thanks for writing this very useful utility! I had a question/feature request, hopefully this is not too bad of a place to submit it.

Right now, I've successfully used jhsingle-native-proxy to serve a notebook via Voila behind JupyterHub, and this is really nice as it allows me to take advantage of JupyterHub's authentication mechanism before allowing access to the Hub-managed service.

I was wondering if there's a good way to expose some of the logs of the process being proxied. Right now, in jhsingle-native-proxy, I see the --debug flag, which will set both the tornado app_log level to DEBUG, and pass debug=True to the tornado Application.

Would it be possible to get INFO level logs though? Right now, if I don't set the --debug argument, then all of my Voila logs seems to get swallowed by the proxy, but if I use the --debug flag, then it logs too much information.

I'm having an issue integrating cdsdashboards to our ML platform http://iko.ai, my dashboard server is timing out during readiness and there is no way to configure the time out value because it's hardcoded here: https://github.com/ideonate/jhsingle-native-proxy/blob/master/jhsingle_native_proxy/proxyhandlers.py#L700

I'd like to be able to configure the readiness timeout value through command-line argument and/or environment variables?

Debug logs:

INFO:tornado.application:SuperviseAndProxyHandler http_get 55527
DEBUG:tornado.application:Allowing Hub user USER_NAME
DEBUG:tornado.application:Storing origin host SERVER_NAME
INFO:tornado.application:['streamlit', 'run', '/home/USER_NAME/./vosk_app.py', '--server.port=55527', '--server.headless=True', '--browser.             serverAddress=SERVER_NAME', '--browser.gatherUsageStats=false']
DEBUG:tornado.application:Trying to start mainprocess
DEBUG:tornado.application:Started mainprocess
DEBUG:tornado.application:Connection to http://localhost:55527/ refused
DEBUG:tornado.application:Readyness: False after 0.00495457649230957 seconds, next check in 0.01s
DEBUG:tornado.application:Connection to http://localhost:55527/ refused
DEBUG:tornado.application:Readyness: False after 0.017713308334350586 seconds, next check in 0.02s
DEBUG:tornado.application:Connection to http://localhost:55527/ refused
DEBUG:tornado.application:Readyness: False after 0.04067730903625488 seconds, next check in 0.04s
DEBUG:tornado.application:Connection to http://localhost:55527/ refused
DEBUG:tornado.application:Readyness: False after 0.08370685577392578 seconds, next check in 0.08s
DEBUG:tornado.application:Connection to http://localhost:55527/ refused
DEBUG:tornado.application:Readyness: False after 0.1665360927581787 seconds, next check in 0.16s
DEBUG:tornado.application:Connection to http://localhost:55527/ refused
DEBUG:tornado.application:Readyness: False after 0.329953670501709 seconds, next check in 0.32s
DEBUG:tornado.application:Connection to http://localhost:55527/ refused
DEBUG:tornado.application:Readyness: False after 0.6533911228179932 seconds, next check in 0.64s
DEBUG:tornado.application:Connection to http://localhost:55527/ refused
DEBUG:tornado.application:Readyness: False after 1.2969000339508057 seconds, next check in 1.28s
DEBUG:tornado.application:Connection to http://localhost:55527/ refused
DEBUG:tornado.application:Readyness: False after 2.5811028480529785 seconds, next check in 2.56s
DEBUG:tornado.application:Connection to http://localhost:55527/ refused
DEBUG:tornado.application:Readyness: False after 5.144717454910278 seconds, next check in 4.847672929763794s
DEBUG:tornado.application:Connection to http://localhost:55527/ refused
DEBUG:tornado.application:Readyness: False after 9.998777389526367 seconds, next check in -0.0035436248779296877s
DEBUG:tornado.application:Connection to http://localhost:55527/ refused
DEBUG:tornado.application:Readyness: False after 10.007751226425171 seconds, next check in -0.007087249755859375s

I'm trying to launch a Streamlit dashboard on Jupyterhub-K8s (EKS) but facing 404 : Not Found You are requesting a page that does not exist! error. I've looked in to the dashboard pod container log, I see below error, and there are no other log statements in the log file.

Whereas same setup is working fine in Minikube, though I see the above error in the log file, Streamlit dashboard launched successfully.

@danlester Could you please help me find the root cause of this issue? TIA.

Hello everyone,

Running web applications on multi-user systems (even behind a JupyterHub) requires that the web application supports some kind of authentication (this can be a token that has to be passed in the URL, as we know it from Jupyter/JupyterLab).

However, some applications lack this basic security and are accessible to anyone who can log into the multi-user system and access their port.

If these web applications would not use a port but a UNIX socket for internal communication with jhsingle-native-proxy, this security hole could be closed in a quite elegant way.
For jupyter-server-proxy this great feature was just recently added - jupyterhub/jupyter-server-proxy#321 and might be a starting point.

Hi, believe it or not, but the 10 MB websocket max_message_size is too low for my use case :). I would be grateful if you could increase it (let's say to 1 GB) or make it configurable.