Spark设置setMaster=local,不提交集群,在本地启用多线程模拟运行
object SparkUtil {
private val logger = Logger.getLogger(getClass.getName, true)
def getSparkContext(appName:String, local:Boolean=false, threadNum:Int=4):SparkContext = {
val conf = new SparkConf().setAppName(appName)
if(local)
conf.setMaster(s"""local[$threadNum]""")
val sc = new SparkContext(conf)
return sc
}
}
SparkUtil.getSparkContext("testApp",true) // 本地启4个线程模拟运行
本地模式运行可以测试主要的程序逻辑,并不能一定在集群中正确运行.
使用spark-submit提交的几种方式如下
local本地模式
spark-submit --class org.apache.spark.examples.SparkPi
--master local
spark-examples-*.jar 10
yarn-clien模式
spark-submit --class org.apache.spark.examples.SparkPi \
--master yarn \
--deploy-mode client \
--driver-memory 4g \
--executor-memory 2g \
--executor-cores 1 \
spark-examples-*.jar \
10
yarn-cluster模式
spark-submit --class org.apache.spark.examples.SparkPi \
--master yarn \
--deploy-mode cluster \
--driver-memory 4g \
--executor-memory 2g \
--executor-cores 1 \
spark-examples-*.jar \
10