Comments (7)
Spark On Yarn mode, you can modify UIUtils.uiRoot, get AppId from current SparkContext Object, and then generate a new uiRoot. Just need to modify UIUtils and SparkUI.
from kyuubi.
Hi @kevin00chen Thanks for your advice. All help are welcome and feel free to create a pull request:)
from kyuubi.
@yaooqinn 方便留个联系方式么,最近在尝试Kyuubi
from kyuubi.
wechat id is as same as github id
from kyuubi.
I tried another approach, which may be not so elegant but may be enough for within company.
I found that the url from Yarn is current, so if we suppose the first applicationId in url is correct, then we can use this to get applicationId and overwrite the uiRoot from System props.
The reason not getting this from current spark context is that UIUtils is an object, we should use global env to get current spark context. which I'm not sure if there will be some problem in high parallelism
from kyuubi.
@sqlwindspeaker in which part we can reset system props?
from kyuubi.
Sorry i made a mistake yesterday, I was thinking that we can get the applicationId from the URL in the HttpServletRequest, but actually not. applicationId is passed to the proxy, and the final URL does not contains this message. So I finally use HttpServletRequest.setAttribute to approach.
// write appId
def attachPage(page: WebUIPage): Unit = {
val pagePath = "/" + page.prefix
val renderHandler = createServletHandler(pagePath,
(request: HttpServletRequest) => {
request.setAttribute("spark.app.id", conf.get("spark.app.id"))
page.render(request)
}, securityManager, conf, basePath)
val renderJsonHandler = createServletHandler(pagePath.stripSuffix("/") + "/json",
(request: HttpServletRequest) => page.renderJson(request), securityManager, conf, basePath)
attachHandler(renderHandler)
attachHandler(renderJsonHandler)
val handlers = pageToHandlers.getOrElseUpdate(page, ArrayBuffer[ServletContextHandler]())
handlers += renderHandler
}
// read appId
def uiRoot(request: HttpServletRequest): String = {
// Knox uses X-Forwarded-Context to notify the application the base path
val knoxBasePath = Option(request.getHeader("X-Forwarded-Context"))
// SPARK-11484 - Use the proxyBase set by the AM, if not found then use env.
val yarnAppId = request.getAttribute("spark.app.id").asInstanceOf[String]
logInfo(s"yarnAppId is: $yarnAppId")
if (yarnAppId.startsWith("application_")) {
s"/proxy/$yarnAppId"
} else {
sys.props.get("spark.ui.proxyBase")
.orElse(sys.env.get("APPLICATION_WEB_PROXY_BASE"))
.orElse(knoxBasePath)
.getOrElse("")
}
}
from kyuubi.
Related Issues (20)
- [Bug] Memory Overhead of OrcMapreduceRecordReader Initialization results in driver oom when fetch big result using FetchOrcStatement
- [Improvement] Kyuubi simulates Hive's proxy submission mode HOT 2
- [Bug] Get only one column using KyuubiStatement.executeAsync function HOT 1
- [Bug] spark historyserver Show incomplete applications kyuubi query engine ui error HOT 7
- [TASK][EASY] DeRegisterWatcher should re-watch when any event is received HOT 1
- [Bug] Error operating ExecuteStatement: java.lang.NullPointerException when use paimon procedures HOT 3
- [Improvement] off-heap memory usage of the driver is too high HOT 8
- [Bug] kyuubi-spark-auth cannot retrieve the database information for paimon in the TableTableExtractor class when extracting resource information. HOT 1
- [Bug] Presto TABLE_NOT_FOUND returns lower case table name causing has_table regex to fail HOT 1
- [Bug] kyuubi authentication enabling both KERBEROS and LDAP functions simultaneously is not available
- [Bug] kyuubi authentication enabling both KERBEROS and LDAP functions simultaneously is not available
- [Bug] function 'get_table_names' in the python client returns incorrect results HOT 5
- kyuubi execute sql,sometime get SocketTimeoutException : read time out HOT 8
- [Bug] Web UI Basic authN ON KERBEROS HOT 5
- Port ZOOKEEPER-4377 to Kyuubi shaded ZooKeeper 3.4 and 3.6 HOT 3
- Add Apache Superset for BI tools page.
- [Bug] Kyuubi 1.19.1 Spark engine small file not merged issue HOT 2
- [Bug] spark hive connector failed to read tpcds parquet table HOT 3
- [FEATURE] `kyuubi-beeline` supports `--conf` HOT 1
- [Bug] The kyuubi.env.SPARK_HOME parameter does not take effect in spark batch jobs
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kyuubi.