CentOS6.5下kafka＋ZooKeeper下载与安装详细教程-单机版

（注：在大数据分析系统中，不会安装单机版，所以可以直接去看多机版教程，多机版教程也是完整的。）

下载zookeeper：

下载地址：http://mirrors.cnnic.cn/apache/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz

 

安装zookeeper：

将zookeeper-3.4.6.tar.gz拿U盘复制到/home/hadoop目录下

 

cd /home/hadoop

打开目录/home/hadoop

 

tar -zxvf zookeeper-3.4.6.tar.gz -C /usr/local

解压到指定目录/usr/local

 

cd /usr/local

打开目录/usr/local

 

mv zookeeper-3.4.6 zookeeper

将zookeeper-3.4.6名字变为zookeeper

 

配置zookeeper：

cd /usr/local/zookeeper

进入zookeeper目录

cp -rf conf/zoo_sample.cfg conf/zoo.cfg

复制一个备份

cd conf

进入conf目录

vim zoo.cfg

编辑zoo.cfg

 

修改后的内容如下：

# The number of milliseconds of each tick

tickTime=2000

# The number of ticks that the initial 

# synchronization phase can take

initLimit=10

# The number of ticks that can pass between 

# sending a request and getting an acknowledgement

syncLimit=5

# the directory where the snapshot is stored.

# do not use /tmp for storage, /tmp here is just 

# example sakes.

dataDir=/usr/local/zookeeper/zkdata #这个目录是预先创建的

# the port at which the clients will connect

clientPort=2181

# the maximum number of client connections.

# increase this if you need to handle more clients

#maxClientCnxns=60

#

# Be sure to read the maintenance section of the 

# administrator guide before turning on autopurge.

#

# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance

#

# The number of snapshots to retain in dataDir

#autopurge.snapRetainCount=3

# Purge task interval in hours

# Set to "0" to disable auto purge feature

#autopurge.purgeInterval=1

 

红色文字原来的内容

# do not use /tmp for storage, /tmp here is just 

# example sakes.

dataDir=/tmp/zookeeper

上面两行的意思是：

不要使用/tmp/zookeeper作为目录

因此，需要对它做出修改

 

建立上面需要的文件夹：

mkdir /usr/local/zookeeper/zkdata

 

 

启动zookeeper：

具体指令如下所示：

[root@Slave1 conf]# cd ../

[root@Slave1 zookeeper]# cd bin

[root@Slave1 bin]# sh zkServer.sh start

JMX enabled by default

Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg

Starting zookeeper ... STARTED

[root@Slave1 bin]# ps ax| grep zookeeper.out 

  7684 pts/0    S+     0:00 grep zookeeper.out

[root@Slave1 bin]#  ps ax| grep zookeeper

  7655 pts/0    Sl     0:02 /usr/local/java/jdk1.7.0_51/bin/java -Dzookeeper.log.dir=. -Dzookeeper.root.logger=INFO,CONSOLE -cp /usr/local/zookeeper/bin/../build/classes:/usr/local/zookeeper/bin/../build/lib/*.jar:/usr/local/zookeeper/bin/../lib/slf4j-log4j12-1.6.1.jar:/usr/local/zookeeper/bin/../lib/slf4j-api-1.6.1.jar:/usr/local/zookeeper/bin/../lib/netty-3.7.0.Final.jar:/usr/local/zookeeper/bin/../lib/log4j-1.2.16.jar:/usr/local/zookeeper/bin/../lib/jline-0.9.94.jar:/usr/local/zookeeper/bin/../zookeeper-3.4.6.jar:/usr/local/zookeeper/bin/../src/java/lib/*.jar:/usr/local/zookeeper/bin/../conf::.:/usr/local/java/jdk1.7.0_51/lib:/usr/local/java/jdk1.7.0_51/jre/lib:.:/usr/local/java/jdk1.7.0_51/lib:/usr/local/java/jdk1.7.0_51/jre/lib:.:/usr/local/java/jdk1.7.0_51/lib:/usr/local/java/jdk1.7.0_51/jre/lib -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false org.apache.zookeeper.server.quorum.QuorumPeerMain /usr/local/zookeeper/bin/../conf/zoo.cfg

  7687 pts/0    S+     0:00 grep zookeeper

[root@Slave1 bin]# 

 

 

下载Kafka：

下载地址：http://apache.fayea.com/kafka/0.8.2.1/kafka_2.10-0.8.2.1.tgz

 

 

安装Kafka：

将kafka_2.10-0.8.2.1.tgz拿U盘复制到/home/hadoop目录下

 

cd /home/hadoop

打开目录/home/hadoop

 

tar -zxvf kafka_2.10-0.8.2.1.tgz -C /usr/local

解压到指定目录/usr/local

 

cd /usr/local

打开目录/usr/local

 

mv kafka_2.10-0.8.2.1 kafka

将kafka_2.10-0.8.2.1名字变为kafka

 

启动Kafka：

cd /usr/local/kafka

打开目录

sh bin/kafka-server-start.sh config/server.properties

启动

 

备注：要挂到后台使用：sh bin/kafka-server-start.sh config/server.properties

 

因为运行信息如下所示，运行结束后，将不能再输入命令：

[root@Slave1 kafka]# sh bin/kafka-server-start.sh config/server.properties

[2015-09-09 10:27:21,451] INFO Verifying properties (kafka.utils.VerifiableProperties)

[2015-09-09 10:27:21,559] INFO Property broker.id is overridden to 0 (kafka.utils.VerifiableProperties)

[2015-09-09 10:27:21,559] INFO Property log.cleaner.enable is overridden to false (kafka.utils.VerifiableProperties)

[2015-09-09 10:27:21,559] INFO Property log.dirs is overridden to /tmp/kafka-logs (kafka.utils.VerifiableProperties)

[2015-09-09 10:27:21,560] INFO Property log.retention.check.interval.ms is overridden to 300000 (kafka.utils.VerifiableProperties)

[2015-09-09 10:27:21,560] INFO Property log.retention.hours is overridden to 168 (kafka.utils.VerifiableProperties)

[2015-09-09 10:27:21,560] INFO Property log.segment.bytes is overridden to 1073741824 (kafka.utils.VerifiableProperties)

[2015-09-09 10:27:21,560] INFO Property num.io.threads is overridden to 8 (kafka.utils.VerifiableProperties)

[2015-09-09 10:27:21,560] INFO Property num.network.threads is overridden to 3 (kafka.utils.VerifiableProperties)

[2015-09-09 10:27:21,560] INFO Property num.partitions is overridden to 1 (kafka.utils.VerifiableProperties)

[2015-09-09 10:27:21,560] INFO Property num.recovery.threads.per.data.dir is overridden to 1 (kafka.utils.VerifiableProperties)

[2015-09-09 10:27:21,561] INFO Property port is overridden to 9092 (kafka.utils.VerifiableProperties)

[2015-09-09 10:27:21,561] INFO Property socket.receive.buffer.bytes is overridden to 102400 (kafka.utils.VerifiableProperties)

[2015-09-09 10:27:21,561] INFO Property socket.request.max.bytes is overridden to 104857600 (kafka.utils.VerifiableProperties)

[2015-09-09 10:27:21,561] INFO Property socket.send.buffer.bytes is overridden to 102400 (kafka.utils.VerifiableProperties)

[2015-09-09 10:27:21,561] INFO Property zookeeper.connect is overridden to localhost:2181 (kafka.utils.VerifiableProperties)

[2015-09-09 10:27:21,562] INFO Property zookeeper.connection.timeout.ms is overridden to 6000 (kafka.utils.VerifiableProperties)

[2015-09-09 10:27:21,692] INFO [Kafka Server 0], starting (kafka.server.KafkaServer)

[2015-09-09 10:27:21,729] INFO [Kafka Server 0], Connecting to zookeeper on localhost:2181 (kafka.server.KafkaServer)

[2015-09-09 10:27:21,793] INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)

[2015-09-09 10:27:21,847] INFO Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT (org.apache.zookeeper.ZooKeeper)

[2015-09-09 10:27:21,847] INFO Client environment:host.name=Slave1 (org.apache.zookeeper.ZooKeeper)

[2015-09-09 10:27:21,848] INFO Client environment:java.version=1.7.0_51 (org.apache.zookeeper.ZooKeeper)

[2015-09-09 10:27:21,848] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper)

[2015-09-09 10:27:21,848] INFO Client environment:java.home=/usr/local/java/jdk1.7.0_51/jre (org.apache.zookeeper.ZooKeeper)

[2015-09-09 10:27:21,848] INFO Client environment:java.class.path=:.:/usr/local/java/jdk1.7.0_51/lib:/usr/local/java/jdk1.7.0_51/jre/lib:.:/usr/local/java/jdk1.7.0_51/lib:/usr/local/java/jdk1.7.0_51/jre/lib:.:/usr/local/java/jdk1.7.0_51/lib:/usr/local/java/jdk1.7.0_51/jre/lib:/usr/local/kafka/bin/../core/build/dependant-libs-2.10.4*/*.jar:/usr/local/kafka/bin/../examples/build/libs//kafka-examples*.jar:/usr/local/kafka/bin/../contrib/hadoop-consumer/build/libs//kafka-hadoop-consumer*.jar:/usr/local/kafka/bin/../contrib/hadoop-producer/build/libs//kafka-hadoop-producer*.jar:/usr/local/kafka/bin/../clients/build/libs/kafka-clients*.jar:/usr/local/kafka/bin/../libs/jopt-simple-3.2.jar:/usr/local/kafka/bin/../libs/kafka_2.10-0.8.2.1.jar:/usr/local/kafka/bin/../libs/kafka_2.10-0.8.2.1-javadoc.jar:/usr/local/kafka/bin/../libs/kafka_2.10-0.8.2.1-scaladoc.jar:/usr/local/kafka/bin/../libs/kafka_2.10-0.8.2.1-sources.jar:/usr/local/kafka/bin/../libs/kafka_2.10-0.8.2.1-test.jar:/usr/local/kafka/bin/../libs/kafka-clients-0.8.2.1.jar:/usr/local/kafka/bin/../libs/log4j-1.2.16.jar:/usr/local/kafka/bin/../libs/lz4-1.2.0.jar:/usr/local/kafka/bin/../libs/metrics-core-2.2.0.jar:/usr/local/kafka/bin/../libs/scala-library-2.10.4.jar:/usr/local/kafka/bin/../libs/slf4j-api-1.7.6.jar:/usr/local/kafka/bin/../libs/slf4j-log4j12-1.6.1.jar:/usr/local/kafka/bin/../libs/snappy-java-1.1.1.6.jar:/usr/local/kafka/bin/../libs/zkclient-0.3.jar:/usr/local/kafka/bin/../libs/zookeeper-3.4.6.jar:/usr/local/kafka/bin/../core/build/libs/kafka_2.10*.jar (org.apache.zookeeper.ZooKeeper)

[2015-09-09 10:27:21,848] INFO Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper)

[2015-09-09 10:27:21,848] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)

[2015-09-09 10:27:21,848] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)

[2015-09-09 10:27:21,848] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)

[2015-09-09 10:27:21,848] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)

[2015-09-09 10:27:21,848] INFO Client environment:os.version=2.6.32-431.el6.x86_64 (org.apache.zookeeper.ZooKeeper)

[2015-09-09 10:27:21,848] INFO Client environment:user.name=root (org.apache.zookeeper.ZooKeeper)

[2015-09-09 10:27:21,848] INFO Client environment:user.home=/root (org.apache.zookeeper.ZooKeeper)

[2015-09-09 10:27:21,848] INFO Client environment:user.dir=/usr/local/kafka (org.apache.zookeeper.ZooKeeper)

[2015-09-09 10:27:21,849] INFO Initiating client connection, connectString=localhost:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@250cf510 (org.apache.zookeeper.ZooKeeper)

[2015-09-09 10:27:22,106] INFO Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)

[2015-09-09 10:27:22,242] INFO Socket connection established to localhost/127.0.0.1:2181, initiating session (org.apache.zookeeper.ClientCnxn)

[2015-09-09 10:27:22,627] INFO Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x14fb1e5298a0000, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)

[2015-09-09 10:27:22,630] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)

[2015-09-09 10:27:22,935] INFO Log directory '/tmp/kafka-logs' not found, creating it. (kafka.log.LogManager)

[2015-09-09 10:27:23,024] INFO Loading logs. (kafka.log.LogManager)

[2015-09-09 10:27:23,048] INFO Logs loading complete. (kafka.log.LogManager)

[2015-09-09 10:27:23,049] INFO Starting log cleanup with a period of 300000 ms. (kafka.log.LogManager)

[2015-09-09 10:27:23,071] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)

[2015-09-09 10:27:25,561] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor)

[2015-09-09 10:27:25,606] INFO [Socket Server on Broker 0], Started (kafka.network.SocketServer)

[2015-09-09 10:27:27,278] INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$)

[2015-09-09 10:27:27,405] INFO 0 successfully elected as leader (kafka.server.ZookeeperLeaderElector)

[2015-09-09 10:27:28,794] INFO Registered broker 0 at path /brokers/ids/0 with address Slave1:9092. (kafka.utils.ZkUtils$)

[2015-09-09 10:27:28,877] INFO [Kafka Server 0], started (kafka.server.KafkaServer)

[2015-09-09 10:27:31,429] INFO New leader is 0 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)

 

新建立一个TOPIC：

cd /usr/local/kafka/bin

打开目录

sh kafka-topics.sh --create --topic kafkatopic --replication-factor 1 --partitions 1 --zookeeper Slave1:2181

此时报了一堆错误，尝试进行修改

sh kafka-topics.sh --create --topic kafkatopic --replication-factor 1 --partitions 1 --zookeeper localhost:2181

此时，正确

显示如下：

 

[root@Slave1 bin]# sh kafka-topics.sh --create --topic kafkatopic --replication-factor 1 --partitions 1 --zookeeper localhost:2181

Created topic "kafkatopic"

 

备注：要挂到后台使用：

sh kafka-topics.sh --create --topic kafkatopic --replication-factor 1 --partitions 1 --zookeeper localhost:2181

 

把KAFKA的生产者启动起来：

 cd /usr/local/kafka/bin/

sh kafka-console-producer.sh --broker-list localhost:9092 --sync --topic kafkatopic

sh kafka-console-producer.sh --broker-list Slave1:9092 --sync --topic kafkatopic

 

 

显示信息如下所示：

[root@Slave1 hadoop]# cd /usr/local/kafka/bin/

[root@Slave1 bin]# sh kafka-console-producer.sh --broker-list localhost:9092 --sync --topic kafkatopic

[2015-09-09 11:06:54,831] WARN Property topic is not valid (kafka.utils.VerifiableProperties)

 

会出现一个警告

原教程中也出现，继续

 

备注：要挂到后台使用：

sh kafka-console-producer.sh --broker-list localhost:9092 --sync --topic kafkatopic

 

另开一个终端，把消费者启动起来：

（这一步骤还在同一台电脑或虚拟机上操作）

cd /usr/local/kafka/bin/

sh kafka-console-consumer.sh --zookeeper localhost:2181 --topic kafkatopic --from-beginning

 

此时也会报错
并与原教材不符合，待寻找原因

 

参考教程：http://www.cnblogs.com/super-d2/p/4534323.html

 

此教程可用性存疑，开始配置多机器环境

 

在后续的文章中，这个问题得到了解决

详情参见：CentOS6.5下kafka＋ZooKeeper下载与安装详细教程-多机版本2