pyspark 链接MongoDB

连接部分代码

• config：指定链接的Mongo库和表

from pyspark import SparkContext, SQLContext, SparkConf
from pyspark.sql import SparkSession
import time, sys, warnings

MONGOURI = 'mongodb://172.00.100.11:27017/{}.{}'.format('ad_portrayal', 'user_portrayal_test')

sc = SparkContext(appName="deal_with_portrayal_data")
sqlContext = SQLContext(sc)
ss = SparkSession(sc).builder \
    .config("spark.mongodb.input.uri", MONGOURI) \
    .config("spark.mongodb.output.uri", MONGOURI) \
    .getOrCreate()

warnings.simplefilter(action='ignore')
sc.setLogLevel('ERROR')

spark提交代码

• packages指定驱动，如下，2.11为Scala版本，2.4.3为spark版本。版本号一定要和集群的对应

PYSPARK_PYTHON=/data/anaconda3/bin/python3 \
/opt/spark/bin/spark-submit \
   --master yarn \
    --deploy-mode client \
    --driver-memory 4g \
    --executor-memory 4g \
    --driver-cores 4 \
    --executor-cores 4 \
    --packages org.mongodb.spark:mongo-spark-connector_2.11:2.4.3 \
    --name 'deal_with_portrayal_data' \
    deal_portData.py >>./log/deal_portData.log 2>>./log/deal_portData.err

然和可以读数据测试了

# coll = ss.read.format('mongo').option('database', 'ad_portrayal').option('collection', 'user_portrayal_test').load()
coll = ss.read.format('mongo').load()
coll.show()
print(type(coll))

+-----+---------+-------+
|EVENT|    REQID|    _id|
+-----+---------+-------+
|click|DWADLAWMD|ABCDEFG|
+-----+---------+-------+

<class 'pyspark.sql.dataframe.DataFrame'>