1、Kafka伪分布安装

1、Kafka伪分布安装

下载的地址： http://kafka.apache.org/downloads.html


下载的软件：kafka_2.10-0.8.2.1.tgz


1、编辑文件：config/server.properties


[root@baozi kafka_2.10-0.8.2.1]# pwd
/usr/local/kafka_2.10-0.8.2.1
[root@baozi kafka_2.10-0.8.2.1]# ll
总用量 28
drwxr-xr-x. 3 root root  4096 2月  27 06:17 bin
drwxr-xr-x. 2 root root  4096 6月  20 20:35 config
drwxr-xr-x. 2 root root  4096 2月  27 06:17 libs
-rw-r--r--. 1 root root 11358 2月  27 06:02 LICENSE
-rw-r--r--. 1 root root   162 2月  27 06:02 NOTICE
[root@baozi kafka_2.10-0.8.2.1]#


[root@baozi kafka_2.10-0.8.2.1]# vim config/server.properties

host.name=192.168.1.200
log.dirs=/mykafka/kafka-logs


2、启动zookeeper：
[root@baozi kafka_2.10-0.8.2.1]# bin/kafka-server-start.sh config/zookeeper.properties &


[root@baozi kafka_2.10-0.8.2.1]# jps
2028 QuorumPeerMain
2122 Jps
[root@baozi kafka_2.10-0.8.2.1]#


3、后台启动kafka：
[root@baozi kafka_2.10-0.8.2.1]# bin/kafka-server-start.sh config/server.properties >/dev/null 2>&1 &

[root@baozi kafka_2.10-0.8.2.1]# jps
2134 Kafka
2222 Jps
2028 QuorumPeerMain
[root@baozi kafka_2.10-0.8.2.1]#


4、查看所有topic列表：
[root@baozi kafka_2.10-0.8.2.1]# bin/kafka-topics.sh --list --zookeeper 192.168.1.200:2181

//这个是在下面执行完创建topic命令之后输入的：
[root@baozi kafka_2.10-0.8.2.1]# bin/kafka-topics.sh --list --zookeeper 192.168.1.200:2181
[2015-06-20 20:59:52,034] INFO Accepted socket connection from /192.168.1.200:48535 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2015-06-20 20:59:52,040] INFO Client attempting to establish new session at /192.168.1.200:48535 (org.apache.zookeeper.server.ZooKeeperServer)
[2015-06-20 20:59:52,042] INFO Established session 0x14e10fd34b3000b with negotiated timeout 30000 for client /192.168.1.200:48535 (org.apache.zookeeper.server.ZooKeeperServer)
test1
test2
[2015-06-20 20:59:52,236] INFO Processed session termination for sessionid: 0x14e10fd34b3000b (org.apache.zookeeper.server.PrepRequestProcessor)
[2015-06-20 20:59:52,240] INFO Closed socket connection for client /192.168.1.200:48535 which had sessionid 0x14e10fd34b3000b (org.apache.zookeeper.server.NIOServerCnxn)
[root@baozi kafka_2.10-0.8.2.1]#


5、创建topic：
[root@baozi kafka_2.10-0.8.2.1]# bin/kafka-topics.sh --create --zookeeper 192.168.1.200:2181 --replication-factor 1 --partitions 3 --topic test1


6、describe命令：


[root@baozi kafka_2.10-0.8.2.1]# bin/kafka-topics.sh --describe --zookeeper 192.168.1.200:2181 --topic test1


显示：
[2015-06-20 20:55:07,901] INFO Accepted socket connection from /192.168.1.200:48529 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2015-06-20 20:55:07,907] INFO Client attempting to establish new session at /192.168.1.200:48529 (org.apache.zookeeper.server.ZooKeeperServer)
[2015-06-20 20:55:07,909] INFO Established session 0x14e10fd34b30005 with negotiated timeout 30000 for client /192.168.1.200:48529 (org.apache.zookeeper.server.ZooKeeperServer)
Topic:test1     PartitionCount:3        ReplicationFactor:1     Configs:
        Topic: test1    Partition: 0    Leader: 0       Replicas: 0     Isr: 0
        Topic: test1    Partition: 1    Leader: 0       Replicas: 0     Isr: 0
        Topic: test1    Partition: 2    Leader: 0       Replicas: 0     Isr: 0
[2015-06-20 20:55:08,831] INFO Processed session termination for sessionid: 0x14e10fd34b30005 (org.apache.zookeeper.server.PrepRequestProcessor)
[2015-06-20 20:55:08,833] INFO Closed socket connection for client /192.168.1.200:48529 which had sessionid 0x14e10fd34b30005 (org.apache.zookeeper.server.NIOServerCnxn)


7、查看所有topic分区情况：
创建topic ->test2
bin/kafka-topics.sh --create --zookeeper 192.168.1.200:2181 --replication-factor 1 --partitions 3 --topic test2


命令：
[root@baozi kafka_2.10-0.8.2.1]# bin/kafka-topics.sh --describe --zookeeper 192.168.1.200:2181
[2015-06-20 20:57:22,513] INFO Accepted socket connection from /192.168.1.200:48533 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2015-06-20 20:57:22,521] INFO Client attempting to establish new session at /192.168.1.200:48533 (org.apache.zookeeper.server.ZooKeeperServer)
[2015-06-20 20:57:22,524] INFO Established session 0x14e10fd34b30009 with negotiated timeout 30000 for client /192.168.1.200:48533 (org.apache.zookeeper.server.ZooKeeperServer)
Topic:test1     PartitionCount:3        ReplicationFactor:1     Configs:
        Topic: test1    Partition: 0    Leader: 0       Replicas: 0     Isr: 0
        Topic: test1    Partition: 1    Leader: 0       Replicas: 0     Isr: 0
        Topic: test1    Partition: 2    Leader: 0       Replicas: 0     Isr: 0
Topic:test2     PartitionCount:3        ReplicationFactor:1     Configs:
        Topic: test2    Partition: 0    Leader: 0       Replicas: 0     Isr: 0
        Topic: test2    Partition: 1    Leader: 0       Replicas: 0     Isr: 0
        Topic: test2    Partition: 2    Leader: 0       Replicas: 0     Isr: 0
[2015-06-20 20:57:23,678] INFO Processed session termination for sessionid: 0x14e10fd34b30009 (org.apache.zookeeper.server.PrepRequestProcessor)
[2015-06-20 20:57:23,682] INFO Closed socket connection for client /192.168.1.200:48533 which had sessionid 0x14e10fd34b30009 (org.apache.zookeeper.server.NIOServerCnxn)
[root@baozi kafka_2.10-0.8.2.1]#


8、删除主题topic：
[root@baozi kafka_2.10-0.8.2.1]# bin/kafka-run-class.sh kafka.admin.TopicCommand --delete --topic test1 --zookeeper 192.168.1.200:2181


查看所有主题：
[root@baozi kafka_2.10-0.8.2.1]# bin/kafka-topics.sh --list --zookeeper 192.168.1.200:2181                                            
[2015-06-20 21:08:51,864] INFO Accepted socket connection from /192.168.1.200:48538 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2015-06-20 21:08:51,870] INFO Client attempting to establish new session at /192.168.1.200:48538 (org.apache.zookeeper.server.ZooKeeperServer)
[2015-06-20 21:08:51,872] INFO Established session 0x14e10fd34b3000e with negotiated timeout 30000 for client /192.168.1.200:48538 (org.apache.zookeeper.server.ZooKeeperServer)
test1 - marked for deletion
test2
[2015-06-20 21:08:52,066] INFO Processed session termination for sessionid: 0x14e10fd34b3000e (org.apache.zookeeper.server.PrepRequestProcessor)
[2015-06-20 21:08:52,071] INFO Closed socket connection for client /192.168.1.200:48538 which had sessionid 0x14e10fd34b3000e (org.apache.zookeeper.server.NIOServerCnxn)
[root@baozi kafka_2.10-0.8.2.1]#


9、进入生产者状态：
[root@baozi kafka_2.10-0.8.2.1]# bin/kafka-console-producer.sh --broker-list 192.168.1.200:9092 --topic test2
[2015-06-20 21:11:18,523] WARN Property topic is not valid (kafka.utils.VerifiableProperties)


[root@baozi ~]# jps
2134 Kafka
2780 Jps
2028 QuorumPeerMain
2759 ConsoleProducer
[root@baozi ~]#


10、进入消费者：
[root@baozi kafka_2.10-0.8.2.1]# bin/kafka-console-consumer.sh --zookeeper 192.168.1.200:2181 --topic test2 --from-beginning


[root@baozi ~]# jps
2793 ConsoleConsumer
2134 Kafka
2843 Jps
2028 QuorumPeerMain
2759 ConsoleProducer
[root@baozi ~]#


11、在生产者这里写入：
hello world
hadoop hive hbase i am baolibin


[root@baozi kafka_2.10-0.8.2.1]# bin/kafka-console-producer.sh --broker-list 192.168.1.200:9092 --topic test2
[2015-06-20 21:11:18,523] WARN Property topic is not valid (kafka.utils.VerifiableProperties)
[2015-06-20 21:14:05,037] INFO Accepted socket connection from /192.168.1.200:48539 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2015-06-20 21:14:05,045] INFO Client attempting to establish new session at /192.168.1.200:48539 (org.apache.zookeeper.server.ZooKeeperServer)
[2015-06-20 21:14:05,048] INFO Established session 0x14e10fd34b3000f with negotiated timeout 30000 for client /192.168.1.200:48539 (org.apache.zookeeper.server.ZooKeeperServer)
[2015-06-20 21:14:05,621] INFO Accepted socket connection from /192.168.1.200:48540 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2015-06-20 21:14:05,622] INFO Client attempting to establish new session at /192.168.1.200:48540 (org.apache.zookeeper.server.ZooKeeperServer)
[2015-06-20 21:14:05,626] INFO Established session 0x14e10fd34b30010 with negotiated timeout 6000 for client /192.168.1.200:48540 (org.apache.zookeeper.server.ZooKeeperServer)
[2015-06-20 21:14:05,858] INFO Got user-level KeeperException when processing sessionid:0x14e10fd34b30010 type:create cxid:0x2 zxid:0x50 txntype:-1 reqpath:n/a Error Path:/consumers/console-consumer-43150/ids Error:KeeperErrorCode = NoNode for /consumers/console-consumer-43150/ids (org.apache.zookeeper.server.PrepRequestProcessor)
[2015-06-20 21:14:05,870] INFO Got user-level KeeperException when processing sessionid:0x14e10fd34b30010 type:create cxid:0x3 zxid:0x51 txntype:-1 reqpath:n/a Error Path:/consumers/console-consumer-43150 Error:KeeperErrorCode = NoNode for /consumers/console-consumer-43150 (org.apache.zookeeper.server.PrepRequestProcessor)
[2015-06-20 21:14:06,547] INFO Got user-level KeeperException when processing sessionid:0x14e10fd34b30010 type:create cxid:0x1b zxid:0x55 txntype:-1 reqpath:n/a Error Path:/consumers/console-consumer-43150/owners/test2 Error:KeeperErrorCode = NoNode for /consumers/console-consumer-43150/owners/test2 (org.apache.zookeeper.server.PrepRequestProcessor)
[2015-06-20 21:14:06,549] INFO Got user-level KeeperException when processing sessionid:0x14e10fd34b30010 type:create cxid:0x1c zxid:0x56 txntype:-1 reqpath:n/a Error Path:/consumers/console-consumer-43150/owners Error:KeeperErrorCode = NoNode for /consumers/console-consumer-43150/owners (org.apache.zookeeper.server.PrepRequestProcessor)
[2015-06-20 21:15:05,679] INFO Got user-level KeeperException when processing sessionid:0x14e10fd34b30010 type:setData cxid:0x28 zxid:0x5c txntype:-1 reqpath:n/a Error Path:/consumers/console-consumer-43150/offsets/test2/0 Error:KeeperErrorCode = NoNode for /consumers/console-consumer-43150/offsets/test2/0 (org.apache.zookeeper.server.PrepRequestProcessor)
[2015-06-20 21:15:05,681] INFO Got user-level KeeperException when processing sessionid:0x14e10fd34b30010 type:create cxid:0x29 zxid:0x5d txntype:-1 reqpath:n/a Error Path:/consumers/console-consumer-43150/offsets Error:KeeperErrorCode = NoNode for /consumers/console-consumer-43150/offsets (org.apache.zookeeper.server.PrepRequestProcessor)
[2015-06-20 21:15:05,747] INFO Got user-level KeeperException when processing sessionid:0x14e10fd34b30010 type:setData cxid:0x2d zxid:0x61 txntype:-1 reqpath:n/a Error Path:/consumers/console-consumer-43150/offsets/test2/2 Error:KeeperErrorCode = NoNode for /consumers/console-consumer-43150/offsets/test2/2 (org.apache.zookeeper.server.PrepRequestProcessor)
[2015-06-20 21:15:05,754] INFO Got user-level KeeperException when processing sessionid:0x14e10fd34b30010 type:create cxid:0x2e zxid:0x62 txntype:-1 reqpath:n/a Error Path:/consumers/console-consumer-43150/offsets/test2 Error:KeeperErrorCode = NodeExists for /consumers/console-consumer-43150/offsets/test2 (org.apache.zookeeper.server.PrepRequestProcessor)
[2015-06-20 21:15:05,771] INFO Got user-level KeeperException when processing sessionid:0x14e10fd34b30010 type:setData cxid:0x30 zxid:0x64 txntype:-1 reqpath:n/a Error Path:/consumers/console-consumer-43150/offsets/test2/1 Error:KeeperErrorCode = NoNode for /consumers/console-consumer-43150/offsets/test2/1 (org.apache.zookeeper.server.PrepRequestProcessor)
[2015-06-20 21:15:05,774] INFO Got user-level KeeperException when processing sessionid:0x14e10fd34b30010 type:create cxid:0x31 zxid:0x65 txntype:-1 reqpath:n/a Error Path:/consumers/console-consumer-43150/offsets/test2 Error:KeeperErrorCode = NodeExists for /consumers/console-consumer-43150/offsets/test2 (org.apache.zookeeper.server.PrepRequestProcessor)
hello world
hadoop hive hbase i am baolibin



在消费者这里会接收到：
[root@baozi kafka_2.10-0.8.2.1]# bin/kafka-console-consumer.sh --zookeeper 192.168.1.200:2181 --topic test2 --from-beginning
hello world
hadoop hive hbase i am baolibin


12、关闭kafka：
关闭broker：
[root@baozi kafka_2.10-0.8.2.1]# bin/kafka-server-stop.sh config/server.properties &


[root@baozi kafka_2.10-0.8.2.1]# bin/kafka-server-stop.sh config/server1.properties &


关闭zookeeper：
[root@baozi kafka_2.10-0.8.2.1]# bin/zookeeper-server-stop.sh config/zookeeper.properties >/dev/null 2>&1 &


[root@baozi kafka_2.10-0.8.2.1]# jps
3751 Jps

[root@baozi kafka_2.10-0.8.2.1]#