两种模式:
cluster mode
client mode
启动shell:
bin/spark-shell --master yarn-client
bin/spark-shell --master yarn-cluster - 不支持
例子:
提交Spark Job方式:
./bin/spark-submit --class org.apache.spark.examples.SparkPi \
--master yarn \
--deploy-mode client \
--driver-memory 2g \
--executor-memory 2g \
--executor-cores 1 \
--queue default \
examples/jars/spark-examples_2.11-2.0.2.jar \
10
./bin/spark-submit --class org.apache.spark.examples.SparkPi \
--master yarn \
--deploy-mode cluster \
--driver-memory 2g \
--executor-memory 2g \
--executor-cores 1 \
--queue default \
examples/jars/spark-examples_2.11-2.0.2.jar \
10
提交任务,cluster模式,添加jar方式:
./bin/spark-submit --class my.main.Class \
--master yarn \
--deploy-mode cluster \
--jars my-other-jar.jar,my-other-other-jar.jar \
my-main-jar.jar \
app_arg1 app_arg2
配置hive-metastore服务:
<property>
<name>hive.metastore.local</name>
<value>false</value>
</property>
<property>
<name>hive.metastore.uris</name>
<value>thrift://pc1:9083,thrift://pc2:9083</value>
</property>
两台都启动:
nohup bin/hive --service metastore 2>&1 >> /dev/null &
同步hive-site.xml文件到hive的conf目录下
spark sql shell:
bin/spark-sql
执行spark sql方式:
./bin/spark-sql --master yarn --deploy-mode cluster --name mysql_test --queue default -f test.sql --不支持
./bin/spark-sql --master yarn --deploy-mode client --name mysql_test --queue default -f test.sql
配置historyserver:
方式一 spark-defaults.conf:
#地址
park.yarn.historyServer.address http://pc3:18080
#端口
spark.history.ui.port 18080
#是否记录Spark事件,用于应用程序在完成后重构webUI
spark.eventLog.enabled true
#历史任务存放的hdfs地址
spark.eventLog.dir hdfs://cluster1/tmp/spark/history-job
#开启压缩
spark.eventLog.compress true
#job信息地址,必须预先建立 - 实际测试中这个参数生效,这个spark.eventLog.dir貌似不生效
spark.history.fs.logDirectory hdfs://pc3:9000/tmp/spark/history-job
方式二 spark-env.sh:
export SPARK_HISTORY_OPTS="-Dspark.history.ui.port=18080 -Dspark.history.retainedApplications=3 -Dspark.history.fs.logDirectory=hdfs://cluster1/tmp/spark/history-job"
启动:
sbin/start-history-server.sh
jps:
102274 HistoryServer
spark on yarn 基本用法
最新推荐文章于 2024-07-02 07:15:00 发布