I.Kafka的生产消费
01.启动Zookeeper
(base) [root@192 ~]# cd $ZOOKEEPER_HOME
(base) [root@192 zookeeper-3.4.14]# cd bin/
(base) [root@192 bin]# ls
README.txt zkCleanup.sh zkCli.cmd zkCli.sh zkEnv.cmd zkEnv.sh zkServer.cmd zkServer.sh zkTxnLogToolkit.cmd zkTxnLogToolkit.sh zookeeper.out
(base) [root@192 bin]# zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper-3.4.14/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
(base) [root@192 bin]# jps
45555 Jps
45530 QuorumPeerMain
02.后台启动Kafka
(base) [root@192 kafka_2.11-2.4.0]# nohup bin/kafka-server-start.sh config/server.properties 2>&1 &
[1] 49457
(base) [root@192 kafka_2.11-2.4.0]# nohup: 忽略输入并把输出追加到"nohup.out"
(base) [root@192 kafka_2.11-2.4.0]# jps
49457 Kafka
49869 Jps
48543 QuorumPeerMain
03.创建kafka主题topic
我这里已经存在了就不需要再创建了
(base) [root@192 kafka_2.11-2.4.0]# ./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic sparkapp
Error while executing topic command : Topic 'sparkapp' already exists.
[2022-03-31 20:46:43,976] ERROR org.apache.kafka.common.errors.TopicExistsException: Topic 'sparkapp' already exists.
(kafka.admin.TopicCommand$)
04.启动producer
>:(这边可以输入内容,不要关闭当前shell窗口)
(base) [root@192 kafka_2.11-2.4.0]# bin/kafka-console-producer.sh --broker-list localhost:9092 --topic sparkapp
>
05.启动consumer
需要另起一个shell窗口
(base) [root@192 kafka_2.11-2.4.0]# cd $KAFKA_HOME
(base) [root@192 kafka_2.11-2.4.0]# bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic sparkapp
(这一行会输出信息)
06.测试再producer种输入,结果就会在consumer中输出
producer的shell中输入hello kafka
consumer的shell中输出hello kafka
producer的shell中再输入hello xiaobao
consumer的shell中输出:
II.pysparkstream对kafka的消费处理
01.相关jar包准备
需要对应版本,放置在$SPARK_HOME/jars目录下
02.使用pyspark编写代码
以wordcount为例
from pyspark import SparkContext
from pyspark.streaming import StreamingContext
from pyspark.streaming.kafka import KafkaUtils
sc = SparkContext(appName="streamingkafka")
sc.setLogLevel("WARN") #设置日志级别
ssc = StreamingContext(sc, 5) # 计算时间间隔
brokers='192.168.1.10:9092' # IP:端口
topic = 'sparkapp' #kafka主题
# 使用streaming使用直连模式消费kafka
kafka_streaming_rdd = KafkaUtils.createDirectStream(ssc, [topic], {"metadata.broker.list": brokers})
lines_rdd = kafka_streaming_rdd.map(lambda x: x[1])
counts = lines_rdd.flatMap(lambda line: line.split(" ")) \
.map(lambda word: (word, 1)) \
.reduceByKey(lambda a, b: a+b)
# 将workcount结果打印到当前shell
counts.pprint()
ssc.start()
ssc.awaitTermination()
property = property()
pycharm中启动效果如下(时间为每隔5秒)
03.在producer的shell中输入内容
04.在pycharm终端中找到这条信息结果
III.jar包下载
文中jar如下:版本不对应可能需要下载相应jar(设置的0C币不知道能不能免费下)
spark-streaming-kafka-0-8-assembly_2.11-2.4.5