pyspark结合kafka实现wordcount

I.Kafka的生产消费

01.启动Zookeeper

(base) [root@192 ~]# cd $ZOOKEEPER_HOME
(base) [root@192 zookeeper-3.4.14]# cd bin/
(base) [root@192 bin]# ls
README.txt  zkCleanup.sh  zkCli.cmd  zkCli.sh  zkEnv.cmd  zkEnv.sh  zkServer.cmd  zkServer.sh  zkTxnLogToolkit.cmd  zkTxnLogToolkit.sh  zookeeper.out
(base) [root@192 bin]# zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper-3.4.14/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
(base) [root@192 bin]# jps

45555 Jps
45530 QuorumPeerMain

02.后台启动Kafka

(base) [root@192 kafka_2.11-2.4.0]# nohup bin/kafka-server-start.sh config/server.properties 2>&1 &
[1] 49457
(base) [root@192 kafka_2.11-2.4.0]# nohup: 忽略输入并把输出追加到"nohup.out"

(base) [root@192 kafka_2.11-2.4.0]# jps
49457 Kafka
49869 Jps
48543 QuorumPeerMain

03.创建kafka主题topic

​ 我这里已经存在了就不需要再创建了

(base) [root@192 kafka_2.11-2.4.0]# ./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic sparkapp
Error while executing topic command : Topic 'sparkapp' already exists.
[2022-03-31 20:46:43,976] ERROR org.apache.kafka.common.errors.TopicExistsException: Topic 'sparkapp' already exists.
 (kafka.admin.TopicCommand$)

04.启动producer

​ >:(这边可以输入内容，不要关闭当前shell窗口)

(base) [root@192 kafka_2.11-2.4.0]# bin/kafka-console-producer.sh --broker-list localhost:9092 --topic sparkapp
>

05.启动consumer

​ 需要另起一个shell窗口

(base) [root@192 kafka_2.11-2.4.0]# cd $KAFKA_HOME
(base) [root@192 kafka_2.11-2.4.0]# bin/kafka-console-consumer.sh --bootstrap-server localhost:9092  --topic sparkapp
(这一行会输出信息)

06.测试再producer种输入，结果就会在consumer中输出

producer的shell中输入hello kafka

consumer的shell中输出hello kafka

producer的shell中再输入hello xiaobao

consumer的shell中输出：

II.pysparkstream对kafka的消费处理

01.相关jar包准备

​ 需要对应版本，放置在$SPARK_HOME/jars目录下

​

02.使用pyspark编写代码

​ 以wordcount为例

from pyspark import SparkContext
from pyspark.streaming import StreamingContext
from pyspark.streaming.kafka import KafkaUtils

sc = SparkContext(appName="streamingkafka")
sc.setLogLevel("WARN") #设置日志级别
ssc = StreamingContext(sc, 5) # 计算时间间隔
brokers='192.168.1.10:9092' # IP：端口
topic = 'sparkapp' #kafka主题
# 使用streaming使用直连模式消费kafka
kafka_streaming_rdd = KafkaUtils.createDirectStream(ssc, [topic], {"metadata.broker.list": brokers})
lines_rdd = kafka_streaming_rdd.map(lambda x: x[1])
counts = lines_rdd.flatMap(lambda line: line.split(" ")) \
    .map(lambda word: (word, 1)) \
    .reduceByKey(lambda a, b: a+b)
# 将workcount结果打印到当前shell
counts.pprint()
ssc.start()
ssc.awaitTermination()

property = property()

​ pycharm中启动效果如下（时间为每隔5秒）

03.在producer的shell中输入内容

04.在pycharm终端中找到这条信息结果

III.jar包下载

文中jar如下：版本不对应可能需要下载相应jar(设置的0C币不知道能不能免费下)
spark-streaming-kafka-0-8-assembly_2.11-2.4.5

spark-streaming-kafka-0-8_2.11-2.4.4.jar