spark-submit \
--master yarn \
--verbose \
--deploy-mode cluster \
--num-executors 1 \
--executor-memory 1G \
--executor-cores 1 \
test.py -table 'ods.tabe' -fields 'dt' -prov hl -dt 20201122
在spark-default.conf
配置
spark.pyspark.python python3
spark.driver.extraClassPath /opt/ops/spark/jars/mysql-connector-java-5.1.42-bin.jar
spark.driver.extraLibraryPath /opt/ops/spark/jars/mysql-connector-java-5.1.42-bin.jar
或者在提交的时候配置
--conf spark.pyspark.python=python3 \
--driver-class-path /opt/ops/spark/jars/mysql-connector-java-5.1.42-bin.jar \
--driver-library-path /opt/ops/spark/jars/mysql-connector-java-5.1.42-bin.jar \
否则会报驱动找不到问题