pyspark 读mysql数据_使用PySpark读取MySQL

I have the following test code:

from pyspark import SparkContext, SQLContext

sc = SparkContext('local')

sqlContext = SQLContext(sc)

print('Created spark context!')

if __name__ == '__main__':

df = sqlContext.read.format("jdbc").options(

url="jdbc:mysql://localhost/mysql",

driver="com.mysql.jdbc.Driver",

dbtable="users",

user="user",

password="****",

properties={"driver": 'com.mysql.jdbc.Driver'}

).load()

print(df)

When I run it, I get the following error:

java.lang.ClassNotFoundException: com.mysql.jdbc.Driver

In Scala, this is solved by importing the .jar mysql-connector-java into the project.

However, in python I have no idea how to tell the pyspark module to link the mysql-connector file.

I have seen this solved with examples like

spark --package=mysql-connector-java testfile.py

But I don't want this since it forces me to run my script in a weird way. I would like an all python solution or copy a file somewhere or, add something to the Path.

解决方案

You can pass arguments to spark-submit when creating your sparkContext before SparkConf is initialized:

import os

from pyspark import SparkConf, SparkContext

SUBMIT_ARGS = "--packages mysql:mysql-connector-java:5.1.39 pyspark-shell"

os.environ["PYSPARK_SUBMIT_ARGS"] = SUBMIT_ARGS

conf = SparkConf()

sc = SparkContext(conf=conf)

or you can add them to your $SPARK_HOME/conf/spark-defaults.conf