1、直接启动时加载配置
可以参考http://dblab.xmu.edu.cn/blog/1724-2/;启动pyspark时,必须指定mysql连接驱动jar包。
cd /usr/local/spark
./bin/pyspark \
--jars /usr/local/spark/jars/mysql-connector-java-5.1.40/mysql-connector-java-5.1.40-bin.jar \
--driver-class-path /usr/local/spark/jars/mysql-connector-java-5.1.40/mysql-connector-java-5.1.40-bin.jar
2、启动jupyter,在os中加载
如何在jupyter加载jar包可以参考
https://stackoverflow.com/questions/35946868/adding-custom-jars-to-pyspark-in-jupyter-notebook
在加载jdbc生成DataFrame,会报找不到驱动错误,添加os.environ配置就可以解决了
from pyspark import SparkContext
from pyspark.sql import SparkSession
import os
os.environ['PYSPARK_SUBMIT_ARGS'] = '--jars /home/jsy/spark_test/mysql/mysql-connector-java-5.1.46/mysql-connector-java-5.1.46-bin.jar,/home/jsy/spark_test/mysql/mysql-connector-java-5.1.46/mysql-connector-java-5.1.46.jar pyspark-shell'
sc = SparkContext( 'local', 'test')
from pyspark.sql import SparkSession
import pyspark
spark=SparkSession(sc)