Kafka 史上最全面的大数据学习（十一） 这是重点 重点 Kafka的安装及一些shell操作 不会只有我没过七夕吧

三、Kafka 安装

单节点模式

[root@HadoopNode00 ~]# mkdir /home/kafka
[root@HadoopNode00 ~]# tar -zxvf kafka_2.11-0.11.0.0.tgz -C /home/kafka/

# /home/kafka/kafka_2.11-0.11.0.0/config/server.properties

# Switch to enable topic deletion or not, default value is false 允许删除topic
delete.topic.enable=true

# 配置kafak sever 的地址
listeners=PLAINTEXT://HadoopNode00:9092

# A comma seperated list of directories under which to store log files  日志存储目录
log.dirs=/home/kafka/kafka_2.11-0.11.0.0/kafka-logs/

#zk地址 
zookeeper.connect=HadoopNode00:2181

# Timeout in ms for connecting to zookeeper 超时时间
zookeeper.connection.timeout.ms=6000

# The minimum age of a log file to be eligible for deletion due to age 消息存储时长
log.retention.hours=168

启动

zk 正常启动

[root@HadoopNode00 kafka_2.11-0.11.0.0]# ./bin/kafka-server-start.sh ./config/server.properties # 前台方式启动 


[root@HadoopNode00 kafka_2.11-0.11.0.0]# ./bin/kafka-server-start.sh -daemon ./config/server.properties
# 后台台方式启动 

关闭

[root@HadoopNode00 kafka_2.11-0.11.0.0]# ./bin/kafka-server-stop.sh

集群模式

# /home/kafka/kafka_2.11-0.11.0.0/config/server.properties

# The id of the broker. This must be set to a unique integer for each broker. # broker id 必须不一样
broker.id=[0,1,2]

# Switch to enable topic deletion or not, default value is false 允许删除topic
delete.topic.enable=true

# 配置kafak sever 的地址
listeners=PLAINTEXT://[HadoopNode01,HadoopNode02，HadoopNode03]:9092

# A comma seperated list of directories under which to store log files  日志存储目录
log.dirs=/home/kafka/kafka_2.11-0.11.0.0/kafka-logs/

#zk地址 
zookeeper.connect=HadoopNode01:2181,HadoopNode02:2181,HadoopNode03:2181

# Timeout in ms for connecting to zookeeper 超时时间
zookeeper.connection.timeout.ms=6000

# The minimum age of a log file to be eligible for deletion due to age 消息存储时长
log.retention.hours=168

验证启动

此时如果使用的是单节点 创建分区副本数量一般就指定为1，因为指定为大于1 的情况后，会发生报错，因为Broker的数量不够

或者说分区副本数量要小于等等于Broker的数量

[root@HadoopNode00 kafka_2.11-0.11.0.0]# ./bin/kafka-topics.sh --zookeeper HadoopNode01:2181,HadoopNode02:2181,HadoopNode03:2181 --create --topic t01 --partitions 3 -replication-factor 3
Created topic "t01".
# 消费者
[root@HadoopNode00 kafka_2.11-0.11.0.0]# ./bin/kafka-console-consumer.sh --bootstrap-server HadoopNode01:9092,HadoopNode02:9092,HadoopNode03:9092 --topic t01 
# 生产者
[root@HadoopNode00 kafka_2.11-0.11.0.0]# ./bin/kafka-console-producer.sh --broker-list HadoopNode01:9092,HadoopNode02:9092,HadoopNode03:9092 --topic t01

添加Topic

[root@HadoopNode00 kafka_2.11-0.11.0.0]# ./bin/kafka-topics.sh --zookeeper HadoopNode01:2181,HadoopNode02:2181,HadoopNode03:2181 --create --topic t01 --partitions 3 -replication-factor 3
Created topic "t01".

查看Topic 详情

[root@HadoopNode00 kafka_2.11-0.11.0.0]# ./bin/kafka-topics.sh --zookeeper HadoopNode00:2181 --describe --topic t01 # 单节点
Topic:t01       PartitionCount:3        ReplicationFactor:1     Configs:
        Topic: t01      Partition: 0    Leader: 0       Replicas: 0     Isr: 0
        Topic: t01      Partition: 1    Leader: 0       Replicas: 0     Isr: 0
        Topic: t01      Partition: 2    Leader: 0       Replicas: 0     Isr: 0

[root@HadoopNode00 kafka_2.11-0.11.0.0]# ./bin/kafka-topics.sh --zookeeper HadoopNode01:2181,HadoopNode02:2181,HadoopNode03:2181 --describe --topic t01 # 集群

删除Topic

[root@HadoopNode01 kafka_2.11-0.11.0.0]# ./bin/kafka-topics.sh --zookeeper HadoopNode01:2181,HadoopNode02:2181,HadoopNode03:2181 --delete --topic t04
Topic t04 is marked for deletion.
Note: This will have no impact if delete.topic.enable is not set to true.

修改分区信息

[root@HadoopNode01 kafka_2.11-0.11.0.0]# ./bin/kafka-topics.sh --zookeeper HadoopNode01:2181,HadoopNode02:2181,HadoopNode03:2181 --alter --topic t01 --partitions 4
WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected
Adding partitions succeeded!

显示所有Topic

[root@HadoopNode01 kafka_2.11-0.11.0.0]# ./bin/kafka-topics.sh --zookeeper HadoopNode01:2181,HadoopNode02:2181,HadoopNode03:2181 --list
__consumer_offsets
t01
t04

分区内先进先出

由此可见，kafka虽然也是FIFO，但是这个范围就是在分区内。

另外消费者在消费消息的时候，不确定是先消费那个分区。

删除某一则消息

这种操作极少，删除已经发布的一则或者多则消息

[root@HadoopNode02 kafka_2.11-0.11.0.0]# ./bin/kafka-delete-records.sh --bootstrap-server HadoopNode01:9092,HadoopNode02:9092,HadoopNode03:9092 --offset-json-file /root/del.json

{"partitions":
	[
	{"topic":"t04","partition":1,"offset":3}
	]
}

删除t04中分区1的3以前的所有的数据

献给每一个正在努力的我们，就算在忙，也要注意休息和饮食哦！我就是我，一个在互联网跌跌撞撞，摸爬滚打的热忱，给个三连吧~ 还有就是不要只看，多动手才行！努力呀！