MetricsSystem

最新推荐文章于 2022-07-12 10:39:14 发布

宝哥大数据

最新推荐文章于 2022-07-12 10:39:14 发布

阅读量313

点赞数

分类专栏： # spark

本文链接：https://blog.csdn.net/wuxintdrh/article/details/89165337

版权

spark 专栏收录该内容

145 篇文章 14 订阅

订阅专栏

一、MetricsSystem初始化

在SparkEnv中初始化， Driver端的MetricsSystem不会立刻启动，需要task scheduler提供一个app ID，
executor端的立刻启动

          val metricsSystem = if (isDriver) {
            //driver端立刻启动
            // Don't start metrics system right now for Driver.
            // We need to wait for the task scheduler to give us an app ID.
            // Then we can start the metrics system.
            MetricsSystem.createMetricsSystem("driver", conf, securityManager)
        } else { //executor 
            // We need to set the executor ID before the MetricsSystem is created because sources and
            // sinks specified in the metrics configuration file will want to incorporate this executor's
            // ID into the metrics they report.
            conf.set("spark.executor.id", executorId)
            val ms = MetricsSystem.createMetricsSystem("executor", conf, securityManager)
            ms.start()  //立刻启动
            ms
        }

二、MetricsSystem启动

在SparkContext中启动， driver的MetricsSystem 需要设置app Id ,然后启动

        _applicationId = _taskScheduler.applicationId()
        _applicationAttemptId = taskScheduler.applicationAttemptId()
        _conf.set("spark.app.id", _applicationId)
        if (_conf.getBoolean("spark.ui.reverseProxy", false)) {
            System.setProperty("spark.ui.proxyBase", "/proxy/" + _applicationId)
        }
        _ui.foreach(_.setAppId(_applicationId))
        _env.blockManager.initialize(_applicationId)

        // The metrics system for Driver need to be set spark.app.id to app ID.
        // So it should start after we get app ID from the task scheduler and set spark.app.id.
        _env.metricsSystem.start()
        // Attach the driver metrics servlet handler to the web ui after the metrics system is started.
        _env.metricsSystem.getServletHandlers.foreach(handler => ui.foreach(_.attachHandler(handler)))

三、start() 中具体做了什么


        def start() {
            require(!running, "Attempting to start a MetricsSystem that is already running")
            running = true
            StaticSources.allSources.foreach(registerSource)
            //注册 sources
            registerSources()
            //注册sink
            registerSinks()
			//启动sink
            sinks.foreach(_.start)
        }