kafka的应用

kafka的安装

1.将kafka压缩包上传的文件夹

2.解压kafka     tar -xvf---------

[root@master software]$ mv /usr、software/kafka_2.11-1.1.0 kafka

3.配置环境变量

[root@master software]$ sudo vi /etc/profile

#kafka

export KAFKA_HOME=/usr/software/kafka

export PATH=$PATH:$KAFKA_HOME/bin

[root@master software]$ source /etc/profile

4.创建节点

在config中的 vi server.properties 创建节点

master broker id =0 logs.dir=/usr/software/kafka/ogs

slave1 broker id =1

slave2 broker id =2

zookeeper.connect=master:2181,slave1:2181,slave2:2181

分发

[root@master config]$ xrsync.sh /usr/software/kafka/

修改slave1、slave2的server.properties 分别为 1， 2

启动Kafka集群

1.[root@master ~]# zkServer.sh
2.[root@master bin]# ./kafka-server-start.sh --daemon /usr/software/kafka_2.11-1.1.0/config/server.properties 

kafka的基本命令

查看topic列表

[root@master bin]$ kafka-topics.sh --zookeeper master:2181 --list

__consumer_offsets

创建topic

[root@master logs]$ kafka-topics.sh --zookeeper master:2181 --create --replication-factor 3 --partitions 2 --topic mytopic1

Created topic "mytopic1".

//查看主题

[root@master ~]$ cd /usr/software/kafka/logs

查看topic

[root@master logs]$ kafka-topics.sh --zookeeper master:2181 --describe --topic mytopic2

Topic:mytopic2 PartitionCount:3 ReplicationFactor:2 Configs:

Topic: mytopic2 Partition: 0 Leader: 2 Replicas: 2,0 Isr: 2,0

Topic: mytopic2 Partition: 1 Leader: 0 Replicas: 0,1 Isr: 0,1

Topic: mytopic2 Partition: 2 Leader: 1 Replicas: 1,2 Isr: 1,2

删除topic

[root@master ~]$ kafka-topics.sh --zookeeper master:2181 --delete --topic mytopic1

Topic mytopic1 is marked for deletion.

Note: This will have no impact if delete.topic.enable is not set to true.

Kafka+flume集成

Kafka作为Channel（没有Sink）

[root@master bin]$ kafka-topics.sh --zookeeper master:2181 --create --topic netcat-kafka --partitions 1 --replication-factor 1

Created topic "netcat-kafka".

启动flume

[root@master flume]$ bin/flume-ng agent -c conf/ -n a1 -f conf/kafka/netcat-kafka.conf  -Dflume.root.logger=INFO,console

启动kafka控制台消费者

[root@master bin]$ ./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic netcat-kafka

测试

[hadoop@hadoop1 ~]$ netcat localhost 44444