hive on spark 配置文件

最新推荐文章于 2024-10-14 16:20:38 发布

Ebaugh

最新推荐文章于 2024-10-14 16:20:38 发布

阅读量351

点赞数

分类专栏： hive 文章标签： hive hiveonspark

本文链接：https://blog.csdn.net/AntKengElephant/article/details/84326646

版权

hive 专栏收录该内容

28 篇文章 0 订阅

订阅专栏

<property>
    <name>hive.execution.engine</name>
    <value>spark</value>
</property>
<property>
    <name>hive.enable.spark.execution.engine</name>
    <value>true</value>
</property>
<property>
    <name>spark.home</name>
    <value>/data/appcom/spark-2.0.2-bin-hadoop2.7</value>
</property>
<property>
    <name>spark.master</name>
    <value>yarn-client</value>
</property>
<property>
    <name>spark.enentLog.enabled</name>
    <value>true</value>
</property>
<property>
    <name>spark.enentLog.dir</name>
    <value>hdfs://master:9000/spark-logs</value>
</property>
<property>
    <name>spark.serializer</name>
    <value>org.apache.spark.serializer.KryoSerializer</value>
</property>
<property>
    <name>spark.executor.memeory</name>
    <value>5g</value>
</property>
<property>
    <name>spark.driver.memeory</name>
    <value>1g</value>
</property>
<property>
    <name>spark.executor.cores</name>
    <value>6</value>
</property>
<property>
    <name>spark.executor.instances</name>
    <value>20</value>
</property>
<property>
    <name>spark.default.parallelism</name>
    <value>50</value>
</property>

或者直接交互界面set

set spark.master=yarn-client;
set hive.execution.engine=spark;
set spark.eventLog.enabled=true;
set spark.eventLog.dir=hdfs://master:9000/spark-logs;
set spark.executor.memory=6g;
set spark.executor.cores=6;
set spark.executor.instances=40;
set spark.default.parallelism= 50;
set spark.serializer=org.apache.spark.serializer.KryoSerializer;