kafka的安装和基本使用

kafka概述

详见：http://kafka.apache.org/intro

Kafka架构：

producer：生产者，就是生产馒头（老妈）

cunsumer：消费者，吃馒头（你）

broker：篮子

topic：主题，给馒头带一个标签，topic a是给你吃的，topic b 是给你弟弟吃的

Kafka部署以及使用

本文主要讲Kafka单节点部署和使用

参考http://kafka.apache.org/quickstart

安装kafka之前要先安装zookeeper

安装过程：

1. 下载：

官网

或 这里

2. 解压：

tar -zvxf kafka_2.11-0.9.0.0.tgz  -C ~/app/

3. 配置环境变量：

export KAFKA_HOME=/home/hadoop/app/kafka_2.11-0.9.0.0
export PATH=$KAFKA_HOME/bin:$PATH

4. 刷新source ~/.bash_profile

5. 配置server.properties

cd /home/hadoop/app/kafka_2.11-0.9.0.0/config
vi server.properties

要修改的地方

#修改主机名
host.name=hadoop000
#修改kafka日志文件目录
log.dirs=/home/hadoop/app/tmp/kafka-logs
#修改zookeeper的连接信息
zookeeper.connect=hadoop000:2181

6. 启动：

zkServer.sh start        #先启动zookeeper：

#bin/kafka-server-start.sh  #尝试单独启动
#USAGE: bin/kafka-server-start.sh [-daemon] server.properties [--override property=value]*   #提示应该指定server.properties	
#启动kafka
kafka-server-start.sh $KAFKA_HOME/config/server.properties

查看进程
jps

[hadoop@hadoop000 ~]$ jps
8917 ZooKeeperMain
9158 Jps
9063 Kafka    #kafka进程
9004 QuorumPeerMain
8525 ZooKeeperMain
[hadoop@hadoop000 ~]$ 

jps -m

[hadoop@hadoop000 ~]$ jps -m
8917 ZooKeeperMain
9206 Jps -m
9063 Kafka /home/hadoop/app/kafka_2.11-0.9.0.0/config/server.properties    #加载的配置
9004 QuorumPeerMain /home/hadoop/app/zookeeper-3.4.5-cdh5.7.0/bin/../conf/zoo.cfg
8525 ZooKeeperMain

7. kafka单节点练习

7.1 创建topic（指定zookeeper）

 kafka-topics.sh --create --zookeeper hadoop000:2181 --replication-factor 1 --partitions 1 --topic topic_hello

7.2 查看所有topic

[hadoop@hadoop000 ~]$ kafka-topics.sh --list --zookeeper hadoop000:2181
hello_topic
kafka_streaming_topic
my-replicated-topic
streamingtopic
topic_hello

7.3 生产消息(和broker-list打交道：9092端口，详情可以看config/server.properties)

[hadoop@hadoop000 ~]$ kafka-console-producer.sh --broker-list hadoop000:9092 --topic topic_hello
hello hadoop
hello 
spark

7.4 消费消息：（zookeeper）

[hadoop@hadoop000 ~]$ kafka-console-consumer.sh --zookeeper hadoop000:2181 --topic topic_hello  --from-beginning
hello hadoop
hello 
spark

--from-beginning代表从头开始消费

单节点操作的其他命令

kafka-topic.sh --describe --zookeeper hadoop000:2181   #查看主题信息

[hadoop@hadoop000 bin]$ kafka-topics.sh --describe --zookeeper hadoop000:2181
Topic:hello_topic	PartitionCount:1	ReplicationFactor:1	Configs:
	Topic: hello_topic	Partition: 0	Leader: 0	Replicas: 0	Isr: 0
Topic:kafka_streaming_topic	PartitionCount:1	ReplicationFactor:1	Configs:
	Topic: kafka_streaming_topic	Partition: 0	Leader: 0	Replicas: 0	Isr: 0
Topic:my-replicated-topic	PartitionCount:1	ReplicationFactor:3	Configs:
	Topic: my-replicated-topic	Partition: 0	Leader: 1	Replicas: 3,1,2	Isr: 1
Topic:streamingtopic	PartitionCount:1	ReplicationFactor:1	Configs:
	Topic: streamingtopic	Partition: 0	Leader: 0	Replicas: 0	Isr: 0
Topic:topic_hello	PartitionCount:1	ReplicationFactor:1	Configs:
	Topic: topic_hello	Partition: 0	Leader: 0	Replicas: 0	Isr: 0

注:Partition: 0 代表分区从0开始

看单独的一个topic

[hadoop@hadoop000 bin]$ kafka-topics.sh --describe --zookeeper hadoop000:2181 --topic hello_topic
Topic:hello_topic	PartitionCount:1	ReplicationFactor:1	Configs:
	Topic: hello_topic	Partition: 0	Leader: 0	Replicas: 0	Isr: 0