环境: win7 + Anaconda3 + pyspark 2.3.1 + kafka 1.0.1
一,win7安装kafka(单机)
1.下载压缩包
Index of /dist/kafka/1.0.1archive.apache.org2.解压并进入 KAFKA_HOME/bin/windows
3,通过CMD启动zookeeper 和 kafka
启动zookeeper:
zookeeper-server-start.bat 你的kafka根目录configzookeeper.properties (启动成功后不要关掉cmd对话框)
启动kafka:
kafka-server-start.bat 你的kafka根目录configserver.properties(启动成功后不要关掉cmd对话框)
5.测试
创建topic test:
你的kafka根目录binwindowskafka-topics.bat --zookeeper localhost:2181 --create --topic test --partitions 1 --replication-factor 1
控制台生产数据:
你的kafka根目录binwindowskafka-console-producer.bat --broker-list localhost:9092 --topic test
控制台消费数据:
你的kafka根目录binwindowskafka-console-consumer.bat --zookeeper localhost:2181 --topic test --from-beginning
kafka搭建成功!!!
二,安装pyspark
通过 anaconda安装
1.打开cmd并切换Python空间
activate tensorflow (你的python空间)
2.查看pyspark安装渠道:
anaconda search -t conda pyspark
3.查看安装命令
anaconda show conda-forge/pyspark
4.安装pyspark
等号后面接你要安装的版本号
conda install --channel Package repository for conda-forge pyspark=2.3.1
5.测试 cmd -> python -> import pyspark
6.下载spark依赖
spark-2.3.1-bin-hadoop2.7.rar
莫名原因官网无法下载,下面是网盘链接
链接:https://pan.baidu.com/s/1Jtt8M_p3gYIBbwZNHZsfMw
提取码:mx4c
--
下载之后解压,pycharm运行时需要指定 spark-2.3.1-bin-hadoop2.7 路径
7.测试
from
如果报错 : FileNotFoundError: [WinError 2] 系统找不到指定的文件。
检查是否spark依赖添加成功
检查pyspark 和spark依赖的版本是否对应。
三.整和pyspark 和 kafka
1.下载spark kafka整合包
spark-streaming-kafka-0-8-assembly, Version = 2.3.1
spark-streaming-kafka-0-8,Version = 2.3.1
在以下地址分别 search spark-streaming-kafka-0-8-assembly 和spark-streaming-kafka-0-8,下载jar包,切换版本参照下方图片。
Maven Central Repository Searchsearch.maven.org下载好两个jar包放到 spark-2.3.1-bin-hadoop2.7(二-6的目录下面)/jars目录下
如果没有这两个jar包会出现以下错误。
Spark Streaming's Kafka libraries not found in class path. Try one of the following.
1. Include the Kafka library and its dependencies with in the
spark-submit command as
$ bin/spark-submit --packages org.apache.spark:spark-streaming-kafka-0-8:2.3.1 ...
2. Download the JAR of the artifact from Maven Central Maven Central Repository Search,
Group Id = org.apache.spark, Artifact Id = spark-streaming-kafka-0-8-assembly, Version = 2.3.1.
Then, include the jar in the spark-submit command as
$ bin/spark-submit --jars <spark-streaming-kafka-0-8-assembly.jar> ...
四.使用pyspark消费kafka
from
ERROR 1.
2020-10-22 14:36:46 ERROR Executor:91 - Exception in task 0.0 in stage 1.0 (TID 1)
org.apache.spark.api.python.PythonException: Traceback (most recent call last):
File " E:Anaconda3installtensorflowenvstensorflowlibsite-packagespysparkrdd.py", line 1371, in takeUpToNumLeft
yield next(iterator)
StopIteration
The above exception was the direct cause of the following exception:
分析 :
使用python3.7版本迭代会出现一些问题
解决:
编辑 E:Anaconda3installtensorflowenvstensorflowlibsite-packagespysparkrdd.py 1392行
将
yield next(iterator)
修改为
try:
yield next(iterator)
except StopIteration:
return
问题解决!!!