Linux下kafka和zookeeper环境搭配

一、安装zookeeper

1.下载zookeeper-3.3.6.tar.gz

2.解压 到/usr/server/zookeeper    ---这里是你的zookeeper的安装目录

3.配置zoo.cfg

cp -rf conf/zoo_sample.cfg conf/zoo.cfg

vim zoo.cfg

# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
dataDir=/data/zookeeper/data
# 日志文件夹
dataLogDir=/data/zookeeper/logs
# the port at which the clients will connect
clientPort=2181

4.配置profile

vim /etc/profile

在后面追加

# idea - zookeeper-3.3.6 config start - 2017-03-08
export ZOOKEEPER_HOME=/usr/server/zookeeper
export PATH=$ZOOKEEPER_HOME/bin:$PATH
export PATH
# idea - zookeeper-3.3.6 config start - 2017-03-08

source /etc/profile  使生效

5.启动zookeeper

bin/zkServer.sh start

如打印以下信息说明成功

ZooKeeper JMX enabled by default

Using config: /usr/local/services/zookeeper/zookeeper-3.4.9/bin/../conf/zoo.cfg

Starting zookeeper ... STARTED

(1) 查询zookeeper状态

bin/zkServer.sh status

(2)关闭zookeeper

bin/zkServer.sh stop

如打印如下信息则表明成功关闭：

ZooKeeper JMX enabled by default

Using config: /usr/local/services/zookeeper/zookeeper-3.4.9/bin/../conf/zoo.cfg

Stopping zookeeper ... STOPPED

(3)重启zookeeper

bin/zkServer.sh restart

如打印如下信息则表明重启成功：

ZooKeeper JMX enabled by default

Using config: /usr/local/services/zookeeper/zookeeper-3.4.9/bin/../conf/zoo.cfg

ZooKeeper JMX enabled by default

Using config: /usr/local/services/zookeeper/zookeeper-3.4.9/bin/../conf/zoo.cfg

Stopping zookeeper ... STOPPED

ZooKeeper JMX enabled by default

Using config: /usr/local/services/zookeeper/zookeeper-3.4.9/bin/../conf/zoo.cfg

Starting zookeeper ... STARTED

二、安装kafka

1.下载kafka_2.10-0.9.0.0

2.解压到/usr/server/kafka  ---这里是你的安装目录

3.启动

bin/kafka-server-start.sh config/server.properties

(要挂到后台使用： bin/kafka-server-start.sh config/server.properties &)

NOTE:

当Kafka broker启动时，它会在ZK上注册自己的IP和端口号，客户端就通过这个IP和端口号来连接。

在AWS这种IaaS环境下，由于java.net.InetAddress.getCanonicalHostName调用拿到的HostName是类似ip-192-168-45-160这样的只有内网才能访问到的主机名，所以默认注册到ZK上的IP是内网才能访问的内网IP。

此时就需要显示指定 host.name, .listeners参数，让注册到ZK上的IP是外网IP。

############################# Socket Server Settings #############################
listeners=PLAINTEXT://192.168.45.160:9092
# The port the socket server listens on
#port=9092
# Hostname the broker will bind to. If not set, the server will bind to all interfaces
host.name=192.168.45.160

否则启动kafka时会报错：

17/03/08 16:50:42 INFO client.ClientUtils$: Fetching metadata from broker id:0,host:192.168.45.160,port:9092 with correlation id 0 for 1 topic(s) Set(shaw)

17/03/08 16:50:42 INFO producer.SyncProducer: Connected to 192.168.45.160:9092 for producing

17/03/08 16:50:42 INFO producer.SyncProducer: Disconnecting from 192.168.45.160:9092

17/03/08 16:50:44 INFO producer.SyncProducer: Connected to h0045160:9092 for producing

17/03/08 16:50:44 INFO producer.SyncProducer: Disconnecting from h0045160:9092

17/03/08 16:50:44 WARN async.DefaultEventHandler: Failed to send producer request with correlation id 2 to broker 0 with data for partitions [shaw,0]

java.nio.channels.ClosedChannelException

    at kafka.network.BlockingChannel.send(BlockingChannel.scala:100)

    at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73)

    at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72)