Linux搭建kafka集群

1首先安装zookeper集群

wget http://mirror.bit.edu.cn/apache/zookeeper/zookeeper-3.4.14/zookeeper-3.4.14.tar.gz
tar -zxvf zookeeper-3.4.14.tar.gz
mv zookeeper-3.4.14 zk
vi ~/.bashrc
export ZOOKEEPER_HOME=/usr/local/zk
export PATH=$PATH:$ZOOKEEPER_HOME/bin:$ZOOKEEPER_HOME/conf
source ~/.bashrc
cd zk/conf
cp zoo_sample.cfg zoo.cfg
vi zoo.cfg
修改：dataDir=/usr/local/zk/data
新增：（redis1,3,4为主机别名 vi /etc/hosts配置 ）
server.0=redis1:2888:3888	
server.1=redis3:2888:3888
server.2=redis4:2888:3888
cd zk
mkdir data
cd data

#配置集群唯一标识第一个为0，第二个为1

vi myid
0

#启动zookeeper集群

zkServer.sh start。

#设置zk 开机自启动
cd /etc/init.d
vi zk.sh
#!/bin/bash    
#chkconfig:2345 20 90 
#description:zookeeper
#processname:zookeeper
export JAVA_HOME=/usr/local/java/jdk1.8.0_11
case $1 in
        start) su root /usr/local/zk/bin/zkServer.sh start;;
        stop) su root /usr/local/zk/bin/zkServer.sh stop;;
        status) su root /usr/local/zk/bin/zkServer.sh status;;
        restart) su root /usr/local/zk/bin/zkServer.sh restart;;
        *) echo "require start|stop|status|restart" ;;
esac

#赋权
chmod 777 zk
chkconfig --add zk
#查看是否有zk自启动的权限
chkconfig --list 

#其次安装scala
tar -zxvf scala-2.11.4.tgz
mv scala-2.11.4 scala
vi ~/.bashrc
export SCALA_HOME=/usr/local/scala
export PATH=$SCALA_HOME/bin
source ~/.bashrc

#查看是否安装成功

scala -version

安装kafka

tar -zxvf kafka_2.9.2-0.8.1.tgz
mv kafka_2.9.2-0.8.1 kafka
vi /usr/local/kafka/config/server.properties
broker.id：依次增长的整数，0、1、2，集群中Broker的唯一id

#添加集群机器

zookeeper.connect=192.168.1.102:2181,192.168.1.105:2181,192.168.1.106:2181

#advertised.host.name=
改为advertised.host.name=192.168.1.102（当前服务器ip地址）
安装slf4j，把slf4j中的slf4j-nop-1.7.6.jar复制到kafka的libs目录下面

vi /usr/local/kafka/bin/kafka-run-class.sh 

if [ -z "$KAFKA_JVM_PERFORMANCE_OPTS" ]; then
  KAFKA_JVM_PERFORMANCE_OPTS="-server  -XX:+UseCompressedOops -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true"
fi

去掉-XX:+UseCompressedOops即可

#后台启动kafka
nohup bin/kafka-server-start.sh config/server.properties &

#验证是否安装成功
bin/kafka-topics.sh --zookeeper 192.168.1.102:2181,192.168.1.105:2181,192.168.1.106:2181 --topic test --replication-factor 1 --partitions 1 --create
bin/kafka-console-producer.sh --broker-list 192.168.1.102:9092,192.168.1.105:9092,192.168.1.106:9092 --topic test
bin/kafka-console-consumer.sh --zookeeper   192.168.1.102:2181,192.168.1.105:2181,192.168.1.106:2181 --topic test --from-beginning
#新版kafka不能使用zookeeper创建消费者
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning

错误总结：
1.错误是说没有分区元数据
1.No partition metadata for topic test due to kafka.common.LeaderNotAvailableException

这种错误修改server.properties

#advertised.host.name=<hostname routable by clients>

改为advertised.host.name=192.168.1.102（当前服务器ip地址）

2.启动报未知名称或服务
2.kafka启动报java.net.UnknownHostException: 主机名: 主机名: 未知的名称或服务
hostname 查看当前主机名称(我的主机名称为root)

vi /etc/hosts

将主机名 添加到本地ip127.0.0.1的后面

127.0.0.1   root  localhost localhost.localdomain localhost4 localhost4.localdomain4

3.启动报错kafka Unrecognized VM option ‘UseCompressedOops’

vi /usr/local/kafka/bin/kafka-run-class.sh 
if [ -z "$KAFKA_JVM_PERFORMANCE_OPTS" ]; then
  KAFKA_JVM_PERFORMANCE_OPTS="-server  -XX:+UseCompressedOops -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true"
fi
去掉-XX:+UseCompressedOops即可