kafka-zookeper-01[linux从零开始-环境搭建与试用]

基础环境搭建

1.安装java环境

https://blog.csdn.net/csfun1/article/details/107075438

2.安装zookerper

步骤如下
1)下载zookeeper安装包http://mirror.bit.edu.cn/apache/zookeeper/zookeeper-3.4.14/
2)解压 tar -zxvf zookeeper.xxxx
3)启动脚本要读取conf下的zoo.cfg文件 故修改文件名
mv zoo_simple.cfg zoo.cfg
4)配置zoo.cfg 中 data文件、日志的存储位置
5)启动bin 下的zkServer.sh start
6)查看进程及端口号
ps -ef | grep zookeeper
jsp -l

3.安装kafka

1)下载kafka

wget http://mirror.bit.edu.cn/apache/kafka/2.3.0/kafka_2.12-2.3.0.tgz 

	tar zxvf kafka_2.12-2.3.0.tgz   linux下直接下载也可以
	【不要直接去Apache官网下载，用镜像即可】

2. 解压tgz文件

tar zxvf kafka_2.12-2.3.0.tgz 

3)配置server.properties参数
broker.id=0 #集群中 broker的编号 若集群中有多个broker 需要设置多个broker
(The id of the broker. This must be set to a unique integer for each broker.)
listeners=PLAINTEXT://:9092 brokers对外提供服务的入口地址
log.dirs=/opt/huawei/logs/kafka kafka集群存放消息信息的目录
zookeeper.connect=localhost:2182 kafka所需Zookeeper集群地址 安装本机 故localhost
# The minimum age of a log file to be eligible for deletion due to age kafka消息保存的时长默认7天
log.retention.hours=168

4)启动kafka
如果时刚安装的linux系统会出现类似 未知系统hostname的问题，需要在etc/hosts 里面配置一下当前的hostname
启动命令 ./kafka-server-start.sh ../config/server.properties
5)检查kafka启动是否成功
ps -ef | grep kafka
jps -l 查看所有进程

6）关闭kafka 杀死kafka进程
修改kafka-server-stop.sh
将 PIDS=$(ps ax | grep -i ‘kafka.Kafka’ | grep java | grep -v grep | awk ‘{print $1}’)
修改为 PIDS=$(jps -lm | grep -i ‘kafka.Kafka’ | awk ‘{print $1}’)

4.kafka 生产和消费DMQ

1)创建topic

bin/kafka-topic.sh  --zookeeper localhost:2181  --create --topic hm.test  --partition 2 
--replication-factor 1
# --create  创建动作
# --zookeeper  指定所连接的zookeeper的服务地址
# --topic 创建主题名称 hm.test
# --partition 分区个数 
# --replication-factor 副本因子

2)展示当前zk下已创建的所有Topic

 bin/kafka-topic.sh  --zookeeper localhost:2181  --list

3)查看某个Topic的详细信息
包含分区、副本、leader等信息

 bin/kafka-topic.sh  --zookeeper localhost:2181  --describe  --topic hm.test

4)启动消费者client
消费端开始是监听 Topic hm.test 的消息
9092 为brokers 为对外提供服务的入口地址

./kafka-console-consumer.sh  --bootstrap-server localhost:9092 --topic hm.test

5)启动生产端

./kafka-console-producer.sh  --broker-list localhost:9092 --topic hm.test

5.java 程序生产和消费kafka消息

pom.xml

 <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <maven.compiler.source>1.8</maven.compiler.source>
    <maven.compiler.target>1.8</maven.compiler.target>
    <kafka.version>2.0.0</kafka.version>
    <scala.version>2.11</scala.version>
    <slf4j.version>1.7.21</slf4j.version>
    <lombok.version>1.18.8</lombok.version>
    <junit.version>4.11</junit.version>
    <gson.version>2.2.4</gson.version>
    <protobuff.version>1.5.4</protobuff.version>
    <spark.version>2.3.1</spark.version>
  </properties>

  <dependencies>
    <dependency>
      <groupId>org.apache.kafka</groupId>
      <artifactId>kafka-clients</artifactId>
      <version>${kafka.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.kafka</groupId>
      <artifactId>kafka_${scala.version}</artifactId>
      <version>${kafka.version}</version>
      <exclusions>
          <exclusion>
              <artifactId>org.apache.zookeeper</artifactId>
              <groupId>zookeepeer</groupId>
          </exclusion>
        <exclusion>
              <groupId>org.slf4j</groupId>
              <artifactId>slf4j-log4j12</artifactId>
        </exclusion>
        <exclusion>
                <artifactId>log4j</artifactId>
                <groupId>log4j</groupId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>org.slf4j</groupId>
      <artifactId>slf4j-log4j12</artifactId>
      <version>${slf4j.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-streaming_${scala.version}</artifactId>
      <version>${spark.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-streaming-kafka-0-10_${scala.version}</artifactId>
      <version>${spark.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-sql_${scala.version}</artifactId>
      <version>${spark.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-sql-kafka-0-10_${scala.version}</artifactId>
      <version>${spark.version}</version>
    </dependency>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>4.11</version>
      <scope>test</scope>
    </dependency>
  </dependencies>