kylin2.1.0_CDH5.9.3实现streaming CUBE

cloudera manage官方文档说明了CDH集成kafka的版本

Product Compatibility Matrix for CDK Powered By Apache Kafka

CDK Powered By Apache Kafka is currently distributed as a package and in a parcel that is independent of the CDH parcel. The parcel integrates with Cloudera Manager using a Custom Service Descriptor (CSD).

For the latest documentation, see Kafka Documentation.

Product	Feature	Lowest Supported Cloudera Manager Version	Supported CDH Versions	Integrated into CDH
CDK 3.0.0 Powered By Apache Kafka		Cloudera Manager 5.13.x	CDH 5.13.x and higher	No
CDK 2.2.x Powered By Apache Kafka		Cloudera Manager 5.9.x	CDH 5.9.x and higher if Sentry integration is required, CDH 5.4.x and higher otherwise	No
CDK 2.1.x Powered By Apache Kafka	Sentry authorization	Cloudera Manager 5.9.x	CDH 5.9.x and higher if Sentry integration is required, CDH 5.4.x and higher otherwise	No
CDK 2.0.x Powered By Apache Kafka	Enhanced security	Cloudera Manager 5.5.3	CDH 5.4.x and higher	No
CDK 1.4.x Powered By Apache Kafka	Distributed both as package and parcel	Cloudera Manager 5.2.x	CDH 5.4.x, 5.5.x, 5.6.x	No
CDK 1.3.x Powered By Apache Kafka	Includes Kafka Monitoring	Cloudera Manager 5.2.x	CDH 5.4.x, 5.5.x, 5.6.x	No
CDK 1.2.x Powered By Apache Kafka		Cloudera Manager 5.2.x	CDH 5.4.x, 5.5.x, 5.6.x	No

按照kylin流式建模的官方文档，支持kafka0.10.0以上的版本，对应CDK2.2.0以上版本。

在kylin2.1.0_CDH5.9.3的版本上安装集成了 CDK 3.0.0 。

修改kylin的配置文件 kylin_job_conf_inmem.xml


 <property>
        <name>mapreduce.map.memory.mb</name>
        <value>512</value>
        <description></description>
    </property>


    <property>
        <name>mapreduce.map.java.opts</name>
        <value>-Xmx1000m -XX:OnOutOfMemoryError='kill -9 %p'</value>
        <description></description>
    </property>


    <property>
        <name>mapreduce.task.io.sort.mb</name>
        <value>50</value>
        <description></description>
    </property>

如果缺少响应依赖的jar包可以执行kylin安装目录bin下的环境文件即可解决。

按照官方文档一步一步可以实现kylin的流式建模。