aseigneurin / spark-ui-proxy Goto Github PK

Lightweight proxy to expose the UI of an Apache Spark cluster that is behind a firewall

License: MIT License

Python 100.00%

spark-ui-proxy's Introduction

Spark UI Proxy

If you are running a Spark Standalone cluster behind a firewall (let's say it is running on Amazon AWS), you might have issues accessing the UI of your cluster, especially because each worker has its own UI, making it difficult if not impossible to reroute all the ports using only SSH tunnels.

                          Firewall
                             |
                             |      ------------------------------
                             |      |        Spark Master        |
                             |      |  e.g. http://10.0.0.1:8080 |
                             |      ------------------------------
                             |
----------------------       |      ------------------------------
|    Your computer   | ----->X      |        Spark Worker        |
| e.g. 192.168.0.10  |       |      |  e.g. http://10.0.0.2:8080 |
----------------------       |      ------------------------------
                             |
                             |      ------------------------------
                             |      |        Spark Worker        |
                             |      |  e.g. http://10.0.0.3:8080 |
                             |      ------------------------------
                             |

This Python script creates a lightweight HTTP server that proxies all the requests to your Spark Master and Spark Workers. All you have to do is create a single SSH tunnel to this proxy, and the proxy will forward all the requests for you. All the links between the nodes will be functional.

                          Firewall
                             |
                             |                                     ------------------------------
                             |                                     |        Spark Master        |
                             |                                  -> |  e.g. http://10.0.0.1:8080 |
                             |                                 /   ------------------------------
                             |                                /
----------------------    tunnel    ------------------------ /     ------------------------------
|    Your computer   | -----------> |    spark-ui-proxy    | ----> |        Spark Worker        |
| e.g. 192.168.0.10  | :9999   :9999| http://10.0.0.1:9999 | \     |  e.g. http://10.0.0.2:8080 |
----------------------       |      ------------------------  \    ------------------------------
                             |                                 \
                             |                                  \  ------------------------------
                             |                                   ->|        Spark Worker        |
                             |                                     |  e.g. http://10.0.0.3:8080 |
                             |                                     ------------------------------
                             |

How to use it

Let's say the Spark Master has its UI running on localhost:8080 (localhost refers to the Spark Master node), and we want to access that UI on localhost:9999 (localhost here refers to your computer).

Start by creating an SSH tunnel from your computer to the Spark Master (but it could be to any of the nodes):

$ ssh -L 9999:localhost:9999 <public IP/name of the node>

On this node, run the Python proxy:

$ python spark-ui-proxy.py localhost:8080 9999

You can stop the proxy at any time by hitting Ctrl+C.

Alternatively, you may run the proxy in background:

$ nohup python spark-ui-proxy.py localhost:8080 9999 &

You can also run it with docker:

$ docker build -t spark-ui-proxy .
$ docker run -d --net host spark-ui-proxy localhost:8080 9999

Now, on your computer, open http://localhost:9999 and you should see the UI of your Spark cluster!

spark-ui-proxy's People

Contributors

Stargazers

Watchers

spark-ui-proxy's Issues

LICENSE?

Can you add a license file so that the community knows the rules of engagement?
Thanks, Andrew

Exceptions are thrown causing the UI to be slow

Hi, this is exactly what I was looking for, but I found a small issue:

Current setup
I'm running the following on Kubernetes:

1 spark master: 10.70.19.3, DNS: spark-master
3 spark workers: 10.70.23.3, 10.70.20.3, 10.70.19.4
1 spark-proxy-ui: 10.70.23.4

Test Case

Access the spark proxy UI --> The Spark Webui load successfully.
Click on a link for a worker --> The Worker page loads
Click on the "Back to Master" --> This is slow because exceptions are thrown, but after a while it redirects back
Click on a different worker link --> The worker page loads (sometimes, exceptions are thrown here also)
Click on "Back to Master" -- Same behaviour, with exceptions

Sometime, exceptions are thrown when I click on the worker links, so "Back to Master" it's not necessarily the only issue.

Logs for the test case

You can find the logs from the spark-proxy-ui for my test case below:

$ kubectl logs -f spark-proxy-ui-controller-09ju8
Starting server on http://0.0.0.0:80
10.70.28.0 - - [22/Sep/2016 13:31:26] "GET /proxy:10.70.23.3:8081 HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:31:27] "GET /proxy:10.70.23.3:8081/static/bootstrap.min.css HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:31:27] "GET /proxy:10.70.23.3:8081/static/webui.css HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:31:27] "GET /proxy:10.70.23.3:8081/static/vis.min.css HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:31:27] "GET /proxy:10.70.23.3:8081/static/timeline-view.css HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:31:27] "GET /proxy:10.70.23.3:8081/static/jquery-1.11.1.min.js HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:31:27] "GET /proxy:10.70.23.3:8081/static/sorttable.js HTTP/1.1" 200 -
get: /proxy:10.70.23.3:8081
host: 10.70.23.3:8081
path:
target: http://10.70.23.3:8081
get: /proxy:10.70.23.3:8081/static/bootstrap.min.css
host: 10.70.23.3:8081
path: /static/bootstrap.min.css
target: http://10.70.23.3:8081/static/bootstrap.min.css
get: /proxy:10.70.23.3:8081/static/webui.css
host: 10.70.23.3:8081
path: /static/webui.css
target: http://10.70.23.3:8081/static/webui.css
get: /proxy:10.70.23.3:8081/static/vis.min.css
host: 10.70.23.3:8081
path: /static/vis.min.css
target: http://10.70.23.3:8081/static/vis.min.css
get: /proxy:10.70.23.3:8081/static/timeline-view.css
host: 10.70.23.3:8081
path: /static/timeline-view.css
target: http://10.70.23.3:8081/static/timeline-view.css
get: /proxy:10.70.23.3:8081/static/jquery-1.11.1.min.js
host: 10.70.23.3:8081
path: /static/jquery-1.11.1.min.js
target: http://10.70.23.3:8081/static/jquery-1.11.1.min.js
get: /proxy:10.70.23.3:8081/static/sorttable.js
host: 10.70.23.3:8081
path: /static/sorttable.js
target: http://10.70.23.3:8081/static/sorttable.js
10.70.28.0 - - [22/Sep/2016 13:31:27] "GET /proxy:10.70.23.3:8081/static/vis.min.js HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:31:27] "GET /proxy:10.70.23.3:8081/static/bootstrap-tooltip.js HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:31:27] "GET /proxy:10.70.23.3:8081/static/initialize-tooltips.js HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:31:27] "GET /proxy:10.70.23.3:8081/static/table.js HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:31:27] "GET /proxy:10.70.23.3:8081/static/additional-metrics.js HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:31:27] "GET /proxy:10.70.23.3:8081/static/timeline-view.js HTTP/1.1" 200 -
get: /proxy:10.70.23.3:8081/static/vis.min.js
host: 10.70.23.3:8081
path: /static/vis.min.js
target: http://10.70.23.3:8081/static/vis.min.js
get: /proxy:10.70.23.3:8081/static/bootstrap-tooltip.js
host: 10.70.23.3:8081
path: /static/bootstrap-tooltip.js
target: http://10.70.23.3:8081/static/bootstrap-tooltip.js
get: /proxy:10.70.23.3:8081/static/initialize-tooltips.js
host: 10.70.23.3:8081
path: /static/initialize-tooltips.js
target: http://10.70.23.3:8081/static/initialize-tooltips.js
get: /proxy:10.70.23.3:8081/static/table.js
host: 10.70.23.3:8081
path: /static/table.js
target: http://10.70.23.3:8081/static/table.js
get: /proxy:10.70.23.3:8081/static/additional-metrics.js
host: 10.70.23.3:8081
path: /static/additional-metrics.js
target: http://10.70.23.3:8081/static/additional-metrics.js
get: /proxy:10.70.23.3:8081/static/timeline-view.js
host: 10.70.23.3:8081
path: /static/timeline-view.js
target: http://10.70.23.3:8081/static/timeline-view.js
get: /proxy:10.70.23.3:8081/static/spark-logo-77x50px-hd.png
host: 10.70.23.3:8081
10.70.28.0 - - [22/Sep/2016 13:31:28] "GET /proxy:10.70.23.3:8081/static/spark-logo-77x50px-hd.png HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:31:28] "GET /favicon.ico HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:31:35] "GET /proxy:spark-master:8080 HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:31:36] "GET /proxy:spark-master:8080/static/bootstrap.min.css HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:31:39] "GET /proxy:spark-master:8080/static/vis.min.css HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:31:39] "GET /proxy:spark-master:8080/static/webui.css HTTP/1.1" 200 -
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/SocketServer.py", line 290, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/usr/local/lib/python2.7/SocketServer.py", line 318, in process_request
    self.finish_request(request, client_address)
  File "/usr/local/lib/python2.7/SocketServer.py", line 331, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/local/lib/python2.7/SocketServer.py", line 652, in __init__
    self.handle()
  File "/usr/local/lib/python2.7/BaseHTTPServer.py", line 340, in handle
    self.handle_one_request()
  File "/usr/local/lib/python2.7/BaseHTTPServer.py", line 328, in handle_one_request
    method()
  File "/spark-ui-proxy.py", line 19, in do_GET
    self.proxyRequest(None)
  File "/spark-ui-proxy.py", line 35, in proxyRequest
    proxiedRequest = urllib2.urlopen(targetUrl, data)
  File "/usr/local/lib/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/local/lib/python2.7/urllib2.py", line 429, in open
    response = self._open(req, data)
  File "/usr/local/lib/python2.7/urllib2.py", line 447, in _open
    '_open', req)
  File "/usr/local/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/local/lib/python2.7/urllib2.py", line 1228, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/local/lib/python2.7/urllib2.py", line 1198, in do_open
    raise URLError(err)
URLError: <urlopen error [Errno -3] Try again>
path: /static/spark-logo-77x50px-hd.png
target: http://10.70.23.3:8081/static/spark-logo-77x50px-hd.png
get: /favicon.ico
host: spark-master:8080
path: /favicon.ico
target: http://spark-master:8080/favicon.ico
get: /proxy:spark-master:8080
host: spark-master:8080
path:
target: http://spark-master:8080
get: /proxy:spark-master:8080/static/bootstrap.min.css
host: spark-master:8080
path: /static/bootstrap.min.css
target: http://spark-master:8080/static/bootstrap.min.css
get: /proxy:spark-master:8080/static/vis.min.css
host: spark-master:8080
path: /static/vis.min.css
target: http://spark-master:8080/static/vis.min.css
get: /proxy:spark-master:8080/static/webui.css
host: spark-master:8080
path: /static/webui.css
target: http://spark-master:8080/static/webui.css
get: /proxy:spark-master:8080/static/timeline-view.css
host: spark-master:8080
path: /static/timeline-view.css
target: http://spark-master:8080/static/timeline-view.css
----------------------------------------
Exception happened during processing of request from ('10.70.28.0', 53652)
10.70.28.0 - - [22/Sep/2016 13:31:44] "GET /proxy:spark-master:8080/static/sorttable.js HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:31:44] "GET /proxy:spark-master:8080/static/jquery-1.11.1.min.js HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:31:45] "GET /proxy:spark-master:8080/static/vis.min.js HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:31:45] "GET /proxy:spark-master:8080/static/bootstrap-tooltip.js HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:31:45] "GET /proxy:spark-master:8080/static/table.js HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:31:49] "GET /proxy:spark-master:8080/static/initialize-tooltips.js HTTP/1.1" 200 -
----------------------------------------
get: /proxy:spark-master:8080/static/sorttable.js
host: spark-master:8080
path: /static/sorttable.js
target: http://spark-master:8080/static/sorttable.js
get: /proxy:spark-master:8080/static/jquery-1.11.1.min.js
host: spark-master:8080
path: /static/jquery-1.11.1.min.js
target: http://spark-master:8080/static/jquery-1.11.1.min.js
get: /proxy:spark-master:8080/static/vis.min.js
host: spark-master:8080
path: /static/vis.min.js
target: http://spark-master:8080/static/vis.min.js
get: /proxy:spark-master:8080/static/bootstrap-tooltip.js
host: spark-master:8080
path: /static/bootstrap-tooltip.js
target: http://spark-master:8080/static/bootstrap-tooltip.js
get: /proxy:spark-master:8080/static/table.js
host: spark-master:8080
path: /static/table.js
target: http://spark-master:8080/static/table.js
get: /proxy:spark-master:8080/static/initialize-tooltips.js
host: spark-master:8080
path: /static/initialize-tooltips.js
target: http://spark-master:8080/static/initialize-tooltips.js
10.70.28.0 - - [22/Sep/2016 13:31:49] "GET /proxy:spark-master:8080/static/additional-metrics.js HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:31:49] "GET /proxy:spark-master:8080/static/timeline-view.js HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:31:50] "GET /proxy:spark-master:8080/static/spark-logo-77x50px-hd.png HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:31:50] "GET /favicon.ico HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:32:36] "GET /proxy:10.70.20.3:8081 HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:32:36] "GET /proxy:10.70.20.3:8081/static/bootstrap.min.css HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:32:36] "GET /proxy:10.70.20.3:8081/static/vis.min.css HTTP/1.1" 200 -
get: /proxy:spark-master:8080/static/additional-metrics.js
host: spark-master:8080
path: /static/additional-metrics.js
target: http://spark-master:8080/static/additional-metrics.js
get: /proxy:spark-master:8080/static/timeline-view.js
host: spark-master:8080
path: /static/timeline-view.js
target: http://spark-master:8080/static/timeline-view.js
get: /proxy:spark-master:8080/static/spark-logo-77x50px-hd.png
host: spark-master:8080
path: /static/spark-logo-77x50px-hd.png
target: http://spark-master:8080/static/spark-logo-77x50px-hd.png
get: /favicon.ico
host: spark-master:8080
path: /favicon.ico
target: http://spark-master:8080/favicon.ico
get: /proxy:10.70.20.3:8081
host: 10.70.20.3:8081
path:
target: http://10.70.20.3:8081
get: /proxy:10.70.20.3:8081/static/bootstrap.min.css
host: 10.70.20.3:8081
path: /static/bootstrap.min.css
target: http://10.70.20.3:8081/static/bootstrap.min.css
get: /proxy:10.70.20.3:8081/static/vis.min.css
host: 10.70.20.3:8081
path: /static/vis.min.css
target: http://10.70.20.3:8081/static/vis.min.css
10.70.28.0 - - [22/Sep/2016 13:32:36] "GET /proxy:10.70.20.3:8081/static/webui.css HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:32:36] "GET /proxy:10.70.20.3:8081/static/timeline-view.css HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:32:36] "GET /proxy:10.70.20.3:8081/static/sorttable.js HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:32:36] "GET /proxy:10.70.20.3:8081/static/jquery-1.11.1.min.js HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:32:36] "GET /proxy:10.70.20.3:8081/static/vis.min.js HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:32:37] "GET /proxy:10.70.20.3:8081/static/bootstrap-tooltip.js HTTP/1.1" 200 -
get: /proxy:10.70.20.3:8081/static/webui.css
host: 10.70.20.3:8081
path: /static/webui.css
target: http://10.70.20.3:8081/static/webui.css
get: /proxy:10.70.20.3:8081/static/timeline-view.css
host: 10.70.20.3:8081
path: /static/timeline-view.css
target: http://10.70.20.3:8081/static/timeline-view.css
get: /proxy:10.70.20.3:8081/static/sorttable.js
host: 10.70.20.3:8081
path: /static/sorttable.js
target: http://10.70.20.3:8081/static/sorttable.js
get: /proxy:10.70.20.3:8081/static/jquery-1.11.1.min.js
host: 10.70.20.3:8081
path: /static/jquery-1.11.1.min.js
target: http://10.70.20.3:8081/static/jquery-1.11.1.min.js
get: /proxy:10.70.20.3:8081/static/vis.min.js
host: 10.70.20.3:8081
path: /static/vis.min.js
target: http://10.70.20.3:8081/static/vis.min.js
get: /proxy:10.70.20.3:8081/static/bootstrap-tooltip.js
host: 10.70.20.3:8081
path: /static/bootstrap-tooltip.js
target: http://10.70.20.3:8081/static/bootstrap-tooltip.js
get: /proxy:10.70.20.3:8081/static/initialize-tooltips.js
host: 10.70.20.3:8081
path: /static/initialize-tooltips.js
10.70.28.0 - - [22/Sep/2016 13:32:37] "GET /proxy:10.70.20.3:8081/static/initialize-tooltips.js HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:32:37] "GET /proxy:10.70.20.3:8081/static/table.js HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:32:37] "GET /proxy:10.70.20.3:8081/static/additional-metrics.js HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:32:37] "GET /proxy:10.70.20.3:8081/static/timeline-view.js HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:32:37] "GET /proxy:10.70.20.3:8081/static/spark-logo-77x50px-hd.png HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:32:38] "GET /favicon.ico HTTP/1.1" 200 -
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/SocketServer.py", line 290, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/usr/local/lib/python2.7/SocketServer.py", line 318, in process_request
    self.finish_request(request, client_address)
  File "/usr/local/lib/python2.7/SocketServer.py", line 331, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/local/lib/python2.7/SocketServer.py", line 652, in __init__
    self.handle()
  File "/usr/local/lib/python2.7/BaseHTTPServer.py", line 340, in handle
    self.handle_one_request()
  File "/usr/local/lib/python2.7/BaseHTTPServer.py", line 328, in handle_one_request
    method()
  File "/spark-ui-proxy.py", line 19, in do_GET
    self.proxyRequest(None)
  File "/spark-ui-proxy.py", line 35, in proxyRequest
    proxiedRequest = urllib2.urlopen(targetUrl, data)
  File "/usr/local/lib/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/local/lib/python2.7/urllib2.py", line 429, in open
    response = self._open(req, data)
  File "/usr/local/lib/python2.7/urllib2.py", line 447, in _open
    '_open', req)
  File "/usr/local/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/local/lib/python2.7/urllib2.py", line 1228, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/local/lib/python2.7/urllib2.py", line 1198, in do_open
    raise URLError(err)
URLError: <urlopen error [Errno -3] Try again>
target: http://10.70.20.3:8081/static/initialize-tooltips.js
get: /proxy:10.70.20.3:8081/static/table.js
host: 10.70.20.3:8081
path: /static/table.js
target: http://10.70.20.3:8081/static/table.js
get: /proxy:10.70.20.3:8081/static/additional-metrics.js
host: 10.70.20.3:8081
path: /static/additional-metrics.js
target: http://10.70.20.3:8081/static/additional-metrics.js
get: /proxy:10.70.20.3:8081/static/timeline-view.js
host: 10.70.20.3:8081
path: /static/timeline-view.js
target: http://10.70.20.3:8081/static/timeline-view.js
get: /proxy:10.70.20.3:8081/static/spark-logo-77x50px-hd.png
host: 10.70.20.3:8081
path: /static/spark-logo-77x50px-hd.png
target: http://10.70.20.3:8081/static/spark-logo-77x50px-hd.png
get: /favicon.ico
host: spark-master:8080
path: /favicon.ico
target: http://spark-master:8080/favicon.ico
get: /proxy:spark-master:8080
host: spark-master:8080
path:
target: http://spark-master:8080
----------------------------------------
Exception happened during processing of request from ('10.70.28.0', 53708)
----------------------------------------
10.70.28.0 - - [22/Sep/2016 13:33:29] "GET /proxy:spark-master:8080 HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:33:29] "GET /proxy:spark-master:8080/static/bootstrap.min.css HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:33:29] "GET /proxy:spark-master:8080/static/vis.min.css HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:33:29] "GET /proxy:spark-master:8080/static/webui.css HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:33:29] "GET /proxy:spark-master:8080/static/timeline-view.css HTTP/1.1" 200 -
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/SocketServer.py", line 290, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/usr/local/lib/python2.7/SocketServer.py", line 318, in process_request
    self.finish_request(request, client_address)
  File "/usr/local/lib/python2.7/SocketServer.py", line 331, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/local/lib/python2.7/SocketServer.py", line 652, in __init__
    self.handle()
  File "/usr/local/lib/python2.7/BaseHTTPServer.py", line 340, in handle
    self.handle_one_request()
  File "/usr/local/lib/python2.7/BaseHTTPServer.py", line 328, in handle_one_request
    method()
  File "/spark-ui-proxy.py", line 19, in do_GET
    self.proxyRequest(None)
  File "/spark-ui-proxy.py", line 35, in proxyRequest
    proxiedRequest = urllib2.urlopen(targetUrl, data)
  File "/usr/local/lib/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/local/lib/python2.7/urllib2.py", line 429, in open
    response = self._open(req, data)
  File "/usr/local/lib/python2.7/urllib2.py", line 447, in _open
    '_open', req)
  File "/usr/local/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/local/lib/python2.7/urllib2.py", line 1228, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/local/lib/python2.7/urllib2.py", line 1198, in do_open
    raise URLError(err)
URLError: <urlopen error [Errno -3] Try again>
get: /proxy:spark-master:8080
host: spark-master:8080
path:
target: http://spark-master:8080
get: /proxy:spark-master:8080/static/bootstrap.min.css
host: spark-master:8080
path: /static/bootstrap.min.css
target: http://spark-master:8080/static/bootstrap.min.css
get: /proxy:spark-master:8080/static/vis.min.css
host: spark-master:8080
path: /static/vis.min.css
target: http://spark-master:8080/static/vis.min.css
get: /proxy:spark-master:8080/static/webui.css
host: spark-master:8080
path: /static/webui.css
target: http://spark-master:8080/static/webui.css
get: /proxy:spark-master:8080/static/timeline-view.css
host: spark-master:8080
path: /static/timeline-view.css
target: http://spark-master:8080/static/timeline-view.css
get: /proxy:spark-master:8080/static/sorttable.js
host: spark-master:8080
path: /static/sorttable.js
target: http://spark-master:8080/static/sorttable.js
----------------------------------------
Exception happened during processing of request from ('10.70.28.0', 53731)
----------------------------------------
10.70.28.0 - - [22/Sep/2016 13:33:34] "GET /proxy:spark-master:8080/static/jquery-1.11.1.min.js HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:33:39] "GET /proxy:spark-master:8080/static/vis.min.js HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:33:39] "GET /proxy:spark-master:8080/static/bootstrap-tooltip.js HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:33:39] "GET /proxy:spark-master:8080/static/table.js HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:33:39] "GET /proxy:spark-master:8080/static/additional-metrics.js HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:33:39] "GET /proxy:spark-master:8080/static/initialize-tooltips.js HTTP/1.1" 200 -
get: /proxy:spark-master:8080/static/jquery-1.11.1.min.js
host: spark-master:8080
path: /static/jquery-1.11.1.min.js
target: http://spark-master:8080/static/jquery-1.11.1.min.js
get: /proxy:spark-master:8080/static/vis.min.js
host: spark-master:8080
path: /static/vis.min.js
target: http://spark-master:8080/static/vis.min.js
get: /proxy:spark-master:8080/static/bootstrap-tooltip.js
host: spark-master:8080
path: /static/bootstrap-tooltip.js
target: http://spark-master:8080/static/bootstrap-tooltip.js
get: /proxy:spark-master:8080/static/table.js
host: spark-master:8080
path: /static/table.js
target: http://spark-master:8080/static/table.js
get: /proxy:spark-master:8080/static/additional-metrics.js
host: spark-master:8080
path: /static/additional-metrics.js
target: http://spark-master:8080/static/additional-metrics.js
get: /proxy:spark-master:8080/static/initialize-tooltips.js
host: spark-master:8080
path: /static/initialize-tooltips.js
target: http://spark-master:8080/static/initialize-tooltips.js
get: /proxy:spark-master:8080/static/timeline-view.js
10.70.28.0 - - [22/Sep/2016 13:33:39] "GET /proxy:spark-master:8080/static/timeline-view.js HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:33:40] "GET /proxy:spark-master:8080/static/spark-logo-77x50px-hd.png HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:33:40] "GET /favicon.ico HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:39:52] "GET /proxy:10.70.20.3:8081 HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:39:52] "GET /proxy:10.70.20.3:8081/static/bootstrap.min.css HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:39:52] "GET /proxy:10.70.20.3:8081/static/vis.min.css HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:39:52] "GET /proxy:10.70.20.3:8081/static/webui.css HTTP/1.1" 200 -
path: /static/spark-logo-77x50px-hd.png
target: http://spark-master:8080/static/spark-logo-77x50px-hd.png
get: /favicon.ico
host: spark-master:8080
path: /favicon.ico
target: http://spark-master:8080/favicon.ico
----------------------------------------
Exception happened during processing of request from ('10.70.28.0', 53987)
----------------------------------------
get: /proxy:10.70.20.3:8081
host: 10.70.20.3:8081
path:
target: http://10.70.20.3:8081
get: /proxy:10.70.20.3:8081/static/bootstrap.min.css
host: 10.70.20.3:8081
path: /static/bootstrap.min.css
target: http://10.70.20.3:8081/static/bootstrap.min.css
get: /proxy:10.70.20.3:8081/static/vis.min.css
host: 10.70.20.3:8081
path: /static/vis.min.css
target: http://10.70.20.3:8081/static/vis.min.css
get: /proxy:10.70.20.3:8081/static/webui.css
host: 10.70.20.3:8081
path: /static/webui.css
target: http://10.70.20.3:8081/static/webui.css
get: /proxy:10.70.20.3:8081/static/timeline-view.css
host: 10.70.20.3:8081
path: /static/timeline-view.css
10.70.28.0 - - [22/Sep/2016 13:39:52] "GET /proxy:10.70.20.3:8081/static/timeline-view.css HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:39:52] "GET /proxy:10.70.20.3:8081/static/sorttable.js HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:39:52] "GET /proxy:10.70.20.3:8081/static/jquery-1.11.1.min.js HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:39:53] "GET /proxy:10.70.20.3:8081/static/vis.min.js HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:39:53] "GET /proxy:10.70.20.3:8081/static/bootstrap-tooltip.js HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:39:53] "GET /proxy:10.70.20.3:8081/static/initialize-tooltips.js HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:39:53] "GET /proxy:10.70.20.3:8081/static/table.js HTTP/1.1" 200 -
target: http://10.70.20.3:8081/static/timeline-view.css
get: /proxy:10.70.20.3:8081/static/sorttable.js
host: 10.70.20.3:8081
path: /static/sorttable.js
target: http://10.70.20.3:8081/static/sorttable.js
get: /proxy:10.70.20.3:8081/static/jquery-1.11.1.min.js
host: 10.70.20.3:8081
path: /static/jquery-1.11.1.min.js
target: http://10.70.20.3:8081/static/jquery-1.11.1.min.js
get: /proxy:10.70.20.3:8081/static/vis.min.js
host: 10.70.20.3:8081
path: /static/vis.min.js
target: http://10.70.20.3:8081/static/vis.min.js
get: /proxy:10.70.20.3:8081/static/bootstrap-tooltip.js
host: 10.70.20.3:8081
path: /static/bootstrap-tooltip.js
target: http://10.70.20.3:8081/static/bootstrap-tooltip.js
get: /proxy:10.70.20.3:8081/static/initialize-tooltips.js
host: 10.70.20.3:8081
path: /static/initialize-tooltips.js
target: http://10.70.20.3:8081/static/initialize-tooltips.js
get: /proxy:10.70.20.3:8081/static/table.js
host: 10.70.20.3:8081
path: /static/table.js
target: http://10.70.20.3:8081/static/table.js
get: /proxy:10.70.20.3:8081/static/additional-metrics.js
10.70.28.0 - - [22/Sep/2016 13:39:53] "GET /proxy:10.70.20.3:8081/static/additional-metrics.js HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:39:53] "GET /proxy:10.70.20.3:8081/static/timeline-view.js HTTP/1.1" 200 -
10.70.28.0 - - [22/Sep/2016 13:39:53] "GET /proxy:10.70.20.3:8081/static/spark-logo-77x50px-hd.png HTTP/1.1" 200 -
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/SocketServer.py", line 290, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/usr/local/lib/python2.7/SocketServer.py", line 318, in process_request
    self.finish_request(request, client_address)
  File "/usr/local/lib/python2.7/SocketServer.py", line 331, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/local/lib/python2.7/SocketServer.py", line 652, in __init__
    self.handle()
  File "/usr/local/lib/python2.7/BaseHTTPServer.py", line 340, in handle
    self.handle_one_request()
  File "/usr/local/lib/python2.7/BaseHTTPServer.py", line 328, in handle_one_request
    method()
  File "/spark-ui-proxy.py", line 19, in do_GET
    self.proxyRequest(None)
  File "/spark-ui-proxy.py", line 35, in proxyRequest
    proxiedRequest = urllib2.urlopen(targetUrl, data)
  File "/usr/local/lib/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/local/lib/python2.7/urllib2.py", line 429, in open
    response = self._open(req, data)
  File "/usr/local/lib/python2.7/urllib2.py", line 447, in _open
    '_open', req)
  File "/usr/local/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/local/lib/python2.7/urllib2.py", line 1228, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/local/lib/python2.7/urllib2.py", line 1198, in do_open
    raise URLError(err)
URLError: <urlopen error [Errno -3] Try again>

There should be 404 page for unrecognized paths

As it is now the proxy returns the Spark master homepage for unrecognized path. E.g.: localhost:9999/yadayada returns the Spark master home page. Why not to return 404 in that case? If you think it is a good idea I can send you a PR as soon as I have some time.

Slower than Spark UI in Kubernetes

Hi - Thank you for the spark-ui-proxy. It made my life much easier. I no longer have to port-forward to each of the worker nodes.

I used this proxy with spark version 1.6.3 and deployed as pods (master, worker, ui-proxy each in separate pods) in kubernetes cluster. Here is the deployment file. The proxy is slower than I expected. It takes around 2 - 3 seconds.

Direct port-forwarding to the master/worker nodes are super fast.

Is there any specific resource requirements for this proxy? CPU, RAM etc?. Anything we could do to speed up?

---
apiVersion: v1
kind: Service
metadata:
  name: spark-ui-proxy
  labels:
    component: spark-ui-proxy
spec:
  ports:
    - port: 80
      targetPort: 80
  clusterIP: None
  selector:
    component: spark-ui-proxy
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: spark-ui-proxy
spec:
  replicas: 1
  template:
    metadata:
      labels:
        component: spark-ui-proxy
    spec:
      containers:
        - name: spark-ui-proxy
          image: networkaispark/spark-ui-proxy:1.0.0
          ports:
            - containerPort: 80
          args:
            - spark-master-v1:8080
          livenessProbe:
              httpGet:
                path: /
                port: 80
              initialDelaySeconds: 120
              timeoutSeconds: 5

Mimetype prediction

spark-ui-proxy returns incorrect mime types for stylesheets. If the proxy is running behind a loadbalancer that upgrades the connections to https, Chrome and Firefox refuse to process the css leading to a broken looking UI.

When directly accessing the stylesheets from the spark master you get:
Content-Type: text/css
When accessing stylesheets through the proxy it returns:
Content-Type: text/plain; charset=utf-8

I've sent a Pull request which just adds a few lines to predict mimetypes which should resolve this. Otherwise there might be a way to just forward the Content-Type through the proxy.

bug - executor page not loading

it seems like the new executors page is not loading.
using spark 2.1 - we have a new executor page...

the problem there that it uses js files to generate the content

The proxy only returns OK

Hi! I have followed the instructions in the README, and deployed the UI proxy with Docker. However if I try to open http://localhost:9999 I just get an OK message. Any idea on what I may be doing wrong?

spark-ui-proxy for spark standalone cluster with spark master configured in HA

Thanks for creating this spark-ui-proxy. In my case, I have a spark standalone cluster with spark master configured with zookeeper to achieve spark master high availability. There are couple of nodes running spark master service in ALIVE (Active) and STANDBY mode.
I had to run this spark-ui-proxy on both the master instances. When I use a apache httpd virtual hosting for load balancing this, requests are not sticking to ALIVE spark master, instead it is going in round robin fashion as both the spark-ui-proxy servers are listen on a port and responding with a status!

To overcome this, I am trying to depend on mod_proxy_hcheck of apache httpd which will disable the balancer member based on the GET calls response. Unfortunately , with this spark-ui-proxy I am getting OK status on both the spark master which again takes me back to round-robin issue between ALIVE and STANDBY master nodes.

Within your code , I see below for health checking, which is returning OK in both the cases:

Add an health checking endpoint.

    if self.path in ["/healthz"]:
        self.send_response(code=200)
        self.send_header("Content-type", "text/plain")
        self.end_headers()
        self.wfile.write("OK")
        return

Is it possible to return ACTIVE or STANDBY responses which will really help in achiving HA with spark-master using this spark-ui-proxy ?

Appreciate any help.

Connection Refused

I get the following error message when I run the script:

(NOTE: I don't have root access on the server.)

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.