spark之mysql驱动配置

1、直接启动时加载配置

可以参考http://dblab.xmu.edu.cn/blog/1724-2/；启动pyspark时，必须指定mysql连接驱动jar包。

cd /usr/local/spark
./bin/pyspark \
--jars /usr/local/spark/jars/mysql-connector-java-5.1.40/mysql-connector-java-5.1.40-bin.jar \
--driver-class-path /usr/local/spark/jars/mysql-connector-java-5.1.40/mysql-connector-java-5.1.40-bin.jar

2、启动jupyter，在os中加载

如何在jupyter加载jar包可以参考 

https://stackoverflow.com/questions/35946868/adding-custom-jars-to-pyspark-in-jupyter-notebook

在加载jdbc生成DataFrame，会报找不到驱动错误，添加os.environ配置就可以解决了

from pyspark import SparkContext
from pyspark.sql import SparkSession
import os
os.environ['PYSPARK_SUBMIT_ARGS'] = '--jars /home/jsy/spark_test/mysql/mysql-connector-java-5.1.46/mysql-connector-java-5.1.46-bin.jar,/home/jsy/spark_test/mysql/mysql-connector-java-5.1.46/mysql-connector-java-5.1.46.jar pyspark-shell'
sc = SparkContext( 'local', 'test')

from pyspark.sql import SparkSession
import pyspark
spark=SparkSession(sc)