调用pyspark脚本

脚本内部 

spark = SparkSession.builder.master('yarn-client').config('spark.executor.instances', 10).config('spark.executor.cores',2). \
        config('spark.executor.memory', '4g').config('spark.sql.shuffle.partitions', 20).config('spark.default.parallelism',20). \
        config('spark.driver.memory', '15g').config('spark.sql.autoBroadcastJoinThreshold', -1).appName(u'xieguoliang_trancode_module').enableHiveSupport().getOrCreate()

启动命令

spark-submit --master yarn --deploy-mode client --queue root.sqoop /home/online_base.py