cmak(kafka Manager) 编译教程

【kafka Manager 编译教程】【编译环境：Centos7】

KafkaManager下载

【2.0以下版本存在 JQuery漏洞，3.0.0.1+依赖zk3.5+，所以选用3.0.0.0，适配现网zk3.4.10】

【官方地址】
 https://github.com/yahoo/CMAK

 【版本】【3.0.0.2开始已提供编译版本】
  CMAK-3.0.0.0.zip

Sbt下载

【编译环境按照kafka manager的要求下载】

【官方地址】
 https://www.scala-sbt.org/download.html

 【版本】
 sbt 1.3.8 (.zip)

Jdk11下载

【3.0+ 依赖 jdk11，编译环境也必须配套Jdk11】

【官方地址】
 https://www.oracle.com/java/technologies/javase-jdk11-downloads.html

 【版本】
 jdk-11.0.7_linux-x64_bin.tar.gz

配置Jdk11

【3.0+ 依赖 jdk11】

【解压】
 # tar zxvf jdk-11.0.7_linux-x64_bin.tar.gz -C /home/java11

【环境变量】
# vim ~/.bashrc

#jdk#
export JAVA_HOME=/home/java11
export PATH=$JAVA_HOME/bin:$PATH

【检查】
# source ~/.bashrc
# java -version

检查sbt版本

【检查kafka Manager依赖的sbt版本】

【解压压缩包】
# unzip CMAK-3.0.0.0.zip
# vim CMAK-3.0.0.0/sbt

[可见该版本依赖1.3.8版本的sbt]
declare -r sbt_release_version="1.3.8"

安装sbt

【解压】
# unzip -q -d /usr/share/ sbt-1.3.8.zip
# cd /usr/share/sbt/

【启动脚本】
# vim /usr/share/sbt/sbt
#!/bin/bash
BT_OPTS="-Xms2048M -Xmx4096M -Xss1M -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=256M"
java $SBT_OPTS -jar /usr/share/sbt/bin/sbt-launch.jar "$@"
# chmod u+x /usr/share/sbt/sbt

【环境变量】
# vim ~/.bashrc
#SBT#
export SBT_HOME=/usr/share/sbt
export PATH=$PATH:$SBT_HOME/bin

# source ~/.bashrc

修改 sbt-launch.jar

【解包修改】
# cd /usr/share/sbt/bin/
# mkdir tmpd;cd tmpd
# cp ../sbt-launch.jar .
# mv ../sbt-launch.jar ../sbt-launch.jar.bak
# unzip -q sbt-launch.jar
# mkdir conf
# vim conf/repo.properties
[repositories]
  local
  aliyun: http://maven.aliyun.com/nexus/content/groups/public/
  central: http://repo1.maven.org/maven2/

【打包jar】
# jar -cfM ./sbt-launch.jar .
# chmod a+x sbt-launch.jar
# mv sbt-launch.jar .. && cd ../
# rm -rf tmpd

下载sbt依赖

# sbt version

copying runtime jar...
[info] [launcher] getting org.scala-sbt sbt 1.3.8  (this may take some time)...
waitting ......

【失败则修改源后重试，下载完后重新执行命令测试】
【如果下载卡在那个包，已经没有下行网速了，则ctrl+c退出后重新下载】
【修改源】
# vim ~/.sbt/repositories
[repositories]
local
aliyun: http://maven.aliyun.com/nexus/content/groups/public/
typesafe: http://repo.typesafe.com/typesafe/ivy-releases/, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext], bootOnly
sonatype-oss-releases
maven-central
sonatype-oss-snapshots

编译KafkaManager

# cd /home/CMAK-3.0.0.0
# ./sbt clean dist

编译完成后会在：/home/kafkamanager/CMAK-3.0.0.0/target/universal 生成安装包 cmak-3.0.0.0.zip

安装KfkaManager

【解压】
# unzip -q -d /home/alpha cmak-3.0.0.0.zip
# mv cmak-3.0.0.0 cmak

目录结构

【目录结构】

【项目】
cmak/
├── bin
├── conf
├── java11
├── lib
├── README.md
├── share
├── start.sh
└── stop.sh

【日志】
/data01/alpha/cmak/logs/
└── application.log

部署目录

/home/alpha/cmak

服务端口

Port：9000

配置服务

【登陆鉴权】

$ vim conf/application.conf
【1】
kafka-manager.zkhosts="10.66.40.97:3181,10.66.40.98:3181,10.66.40.99:3181"
【2】
basicAuthentication.enabled=true
【3】
akka {
  loggers = ["akka.event.slf4j.Slf4jLogger"]
  loglevel = "OFF"
}
【4】
basicAuthentication.password="Jet#2dg%S@A"

日志配置

【默认关闭所有日志，异常时日志会很多】

【=================== logger.xml ==========================】
$ vim conf/logger.xml

<configuration>

  <conversionRule conversionWord="coloredLevel" converterClass="play.api.Logger$ColoredLevel" />

  <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <file>/data01/alpha/cmak/logs/application.log</file>
    <encoder>
       <pattern>%date - [%level] - from %logger in %thread %n%message%n%xException%n</pattern>
    </encoder>

    <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
      <fileNamePattern>/data01/alpha/cmak/logs/application.%d{yyyy-MM-dd}.log</fileNamePattern>
      <maxHistory>3</maxHistory>
      <totalSizeCap>100MB</totalSizeCap>
    </rollingPolicy>
  </appender>

  <logger name="play" level="OFF" />
  <logger name="application" level="OFF" />

  <!-- Off these ones as they are annoying, and anyway we manage configuration ourself -->
  <logger name="com.avaje.ebean.config.PropertyMapLoader" level="OFF" />
  <logger name="com.avaje.ebeaninternal.server.core.XmlConfigLoader" level="OFF" />
  <logger name="com.avaje.ebeaninternal.server.lib.BackgroundThread" level="OFF" />
  <logger name="com.gargoylesoftware.htmlunit.javascript" level="OFF" />
  <logger name="org.apache.zookeeper.ClientCnxn" level="OFF" />
  <logger name="org.apache.zookeeper" level="OFF"/>
  <logger name="org.apache.curator.ConnectionState" level="OFF"/>
  <logger name="kafka.manager.KafkaManager" level="OFF"/>

  <logger name="akka" level="OFF" />
  <logger name="kafka" level="OFF" />

  <root level="OFF">
    <appender-ref ref="FILE" />
  </root>

</configuration>

【=================== logback.xml ==========================】
$ vim conf/logback.xml
<!--
  ~ Copyright (C) 2009-2015 Typesafe Inc. <http://www.typesafe.com>
  -->
<!-- The default logback configuration that Play uses if no other configuration is provided -->
<configuration>

    <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>/data01/alpha/cmak/logs/application.log</file>
        <encoder>
           <pattern>%date - [%level] - from %logger in %thread %n%message%n%xException%n</pattern>
        </encoder>

        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>/data01/alpha/cmak/logs/application.%d{yyyy-MM-dd}.log</fileNamePattern>
            <maxHistory>3</maxHistory>
            <totalSizeCap>100MB</totalSizeCap>
        </rollingPolicy>
    </appender>

    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%date - [%level] %logger{15} - %message%n%xException{10}</pattern>
        </encoder>
    </appender>

    <appender name="ASYNCFILE" class="ch.qos.logback.classic.AsyncAppender">
        <appender-ref ref="FILE" />
    </appender>

    <appender name="ASYNCSTDOUT" class="ch.qos.logback.classic.AsyncAppender">
        <appender-ref ref="STDOUT" />
    </appender>

    <logger name="play" level="OFF" />
    <logger name="application" level="OFF" />
    <logger name="kafka.manager" level="OFF" />

    <!-- Off these ones as they are annoying, and anyway we manage configuration ourself -->
    <logger name="com.avaje.ebean.config.PropertyMapLoader" level="OFF" />
    <logger name="com.avaje.ebeaninternal.server.core.XmlConfigLoader" level="OFF" />
    <logger name="com.avaje.ebeaninternal.server.lib.BackgroundThread" level="OFF" />
    <logger name="com.gargoylesoftware.htmlunit.javascript" level="OFF" />
    <logger name="org.apache.zookeeper" level="OFF"/>
    <logger name="org.apache.curator.ConnectionState" level="OFF"/>

    <root level="OFF">
        <appender-ref ref="ASYNCFILE" />
        <appender-ref ref="ASYNCSTDOUT" />
    </root>

</configuration>

存储目录

【创建日志目录】

$ mkdir -p /data01/alpha/cmak/logs

服务管理

【启动脚本】

$ vim cmak/start.sh
#!/bin/bash
if [ -f RUNNING_PID ];then echo -e "Delete RUNNING_PID !\n";rm RUNNING_PID;fi
/home/alpha/cmak/bin/cmak -Dconfig.file=/home/alpha/cmak/conf/application.conf -java-home /home/alpha/cmak/java11 -Dhttp.port=9000 2>&1 >/dev/null &

【停止脚本】

$ cat cmak/stop.sh
#!/bin/bash
jps|grep -w 'ProdServerStart'|awk '{print $1}'|xargs kill -9
if [ -f RUNNING_PID ];then echo -e "Delete RUNNING_PID !\n";rm RUNNING_PID;fi

启动服务

【启动】
$ cd ~/cmak
$ ./start.sh

【检查】
$ jps
1234 ProdServerStart

登陆页面

【web控制台】
http://10.66.40.97:9000

【用户名/密码】
admin / Jet#2dg%S@A

新建集群

【WEB创建集群】

配置项	参数
Cluster Name	lnyd-1
Cluster Zookeeper Hosts	10.66.40.97:3181,10.66.40.98:3181,10.66.40.99:3181
Kafka Version	0.10.1.0
勾选	Enable JMX Polling
勾选	Poll consumer information
勾选	Filter out inactive consumers
勾选	Enable Active OffsetCache
勾选	Display Broker and Topic Size

【注意】勾选“Enable JMX Polling” 后，kafka的启动脚本 “kafka-server-start.sh” 需要增加 “export JMX_PORT=9099”

# vim kafka/bin/kafka-server-start.sh
if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
    export KAFKA_HEAP_OPTS="-Xmx4G -Xms4G"
    export JMX_PORT=9099
fi

krb 认证

【配置】
# vim conf/application.conf

【kerberos】
-Djava.security.auth.login.config=/etc/ksecurity/my-jaas.conf