kylin cube集成kafka源码阅读

最新推荐文章于 2024-08-05 17:44:19 发布

AI干IT

最新推荐文章于 2024-08-05 17:44:19 发布

阅读量642

点赞数

本文链接：https://blog.csdn.net/mpren/article/details/81564098

版权

本文深入探讨了Kylin如何集成Kafka进行cube构建，从curl命令开始，详细解读源码中的JobService、FetcherRunner和JobRunner等关键组件。重点在于KafkaFlatTableJob如何作为Hadoop Job运行，利用KafkaInputFormat将Kafka数据适配到Hadoop HDFS。

摘要由CSDN通过智能技术生成

首先看一条kafka cube构建语句

curl -X PUT –user ADMIN:KYLIN -H “Content-Type: application/json;charset=utf-8” -d ‘{ “sourceOffsetStart”: 0,”sourceOffsetEnd”: 9223372036854775807,”buildType”: “BUILD”}’ http://99.48.2.1:7070/kylin/api/cubes/cubename/build2

源码入口CubeController

/**
 * Build/Rebuild a cube segment by source offset
 */
@RequestMapping(value = "/{cubeName}/build2", method = { RequestMethod.PUT }, produces = { "application/json" })
@ResponseBody
public JobInstance build2(@PathVariable String cubeName, @RequestBody JobBuildRequest2 req) {
    boolean existKafkaClient = false;
    ...
    return rebuild2(cubeName, req);
}

private JobInstance buildInternal(String cubeName, TSRange tsRange, SegmentRange segRange, //
        Map<Integer, Long> sourcePartitionOffsetStart, Map<Integer, Long> sourcePartitionOffsetEnd,
        String buildType, boolean force) {
        return jobService.submitJob(cube, tsRange, segRange, sourcePartitionOffsetStart, sourcePartitionOffsetEnd,
                CubeBuildTypeEnum.valueOf(buildType), force, submitter);

}

这里出现了jobService，我们进入看看

public class JobService extends BasicService implements InitializingBean {

看到InitializingBean自然想到了afterPropertiesSet（），方法里面做了什么呢？

@SuppressWarnings("unchecked")
@Override
public void afterPropertiesSet() throws Exception {
    ...
    new Thread(new Runnable() {
        @Override
        public void run() {
            try {
                scheduler.init(new JobEngineConfig(kylinConfig), new ZookeeperJobLock());
                if (!scheduler.hasStarted()) {
                    logger.info("scheduler has not been started");
                }
            } catch (Exception e) {
                throw new RuntimeException(e);
            }
        }
    }).start();
    ...
}

这里启动了一个调度器线程，跟进去看init方法,这里面很重要。

@Override
public synchronized void init(JobEngineConfig jobEngineConfig, JobLock jobLock) throws SchedulerException {
    String serverMode = jobEngineConfig.getConfig().getServerMode();
    if (!("job".equals(serverMode.toLowerCase()) || "all".equals(serverMode.toLowerCase()))) {
        logger.info("server mode: " + serverMode + ", no need to run job scheduler");
        return;
    }
    logger.info("Initializing Job Engine ....");

    if