Giter Club home page Giter Club logo

Comments (7)

kevin00chen avatar kevin00chen commented on August 28, 2024

Spark On Yarn mode, you can modify UIUtils.uiRoot, get AppId from current SparkContext Object, and then generate a new uiRoot. Just need to modify UIUtils and SparkUI.

from kyuubi.

yaooqinn avatar yaooqinn commented on August 28, 2024

Hi @kevin00chen Thanks for your advice. All help are welcome and feel free to create a pull request:)

from kyuubi.

kevin00chen avatar kevin00chen commented on August 28, 2024

@yaooqinn 方便留个联系方式么,最近在尝试Kyuubi

from kyuubi.

yaooqinn avatar yaooqinn commented on August 28, 2024

wechat id is as same as github id

from kyuubi.

sqlwindspeaker avatar sqlwindspeaker commented on August 28, 2024

I tried another approach, which may be not so elegant but may be enough for within company.

I found that the url from Yarn is current, so if we suppose the first applicationId in url is correct, then we can use this to get applicationId and overwrite the uiRoot from System props.

The reason not getting this from current spark context is that UIUtils is an object, we should use global env to get current spark context. which I'm not sure if there will be some problem in high parallelism

from kyuubi.

yaooqinn avatar yaooqinn commented on August 28, 2024

@sqlwindspeaker in which part we can reset system props?

from kyuubi.

sqlwindspeaker avatar sqlwindspeaker commented on August 28, 2024

Sorry i made a mistake yesterday, I was thinking that we can get the applicationId from the URL in the HttpServletRequest, but actually not. applicationId is passed to the proxy, and the final URL does not contains this message. So I finally use HttpServletRequest.setAttribute to approach.

// write appId
 def attachPage(page: WebUIPage): Unit = {
    val pagePath = "/" + page.prefix
    val renderHandler = createServletHandler(pagePath,
      (request: HttpServletRequest) => {
        request.setAttribute("spark.app.id", conf.get("spark.app.id"))
        page.render(request)
      }, securityManager, conf, basePath)
    val renderJsonHandler = createServletHandler(pagePath.stripSuffix("/") + "/json",
      (request: HttpServletRequest) => page.renderJson(request), securityManager, conf, basePath)
    attachHandler(renderHandler)
    attachHandler(renderJsonHandler)
    val handlers = pageToHandlers.getOrElseUpdate(page, ArrayBuffer[ServletContextHandler]())
    handlers += renderHandler
  }

// read appId
  def uiRoot(request: HttpServletRequest): String = {
    // Knox uses X-Forwarded-Context to notify the application the base path
    val knoxBasePath = Option(request.getHeader("X-Forwarded-Context"))
    // SPARK-11484 - Use the proxyBase set by the AM, if not found then use env.

    val yarnAppId = request.getAttribute("spark.app.id").asInstanceOf[String]

    logInfo(s"yarnAppId is: $yarnAppId")

    if (yarnAppId.startsWith("application_")) {
      s"/proxy/$yarnAppId"
    } else {
      sys.props.get("spark.ui.proxyBase")
        .orElse(sys.env.get("APPLICATION_WEB_PROXY_BASE"))
        .orElse(knoxBasePath)
        .getOrElse("")
    }
  }

from kyuubi.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.