Spark安装配置04--YARN模式安装

------------------------------------------------

1.修改hadoop,yarn-site.xml,添加内容如下：

<configuration>
<!-- Site specific YARN configuration properties -->
        <property>
                <name>yarn.resourcemanager.hostname</name>
                <value>s101/value>
        </property>
        <property>
                <name>yarn.nodemanager.aux-services</name>
                <value>mapreduce_shuffle</value>
        </property>
        <!--是否启动一个线程检查每个任务正使用的物理内存量，如果任务超出分配值，则直接将其杀掉，默认是true -->
        <property>
                <name>yarn.nodemanager.pmem-check-enabled</name>
                <value>false</value>
        </property>
        <!--是否启动一个线程检查每个任务正使用的虚拟内存量，如果任务超出分配值，则直接将其杀掉，默认是true -->
        <property>
                <name>yarn.nodemanager.vmem-check-enabled</name>
                <value>false</value>
        </property>
</configuration>

------------------------------------------------

2.修改Spark-env.sh 添加内容如下：

HADOOP_CONF_DIR=/soft/hadoop/etc/hadoop
YARN_CONF_DIR=/soft/hadoop/etc/hadoop

------------------------------------------------

3.分发配置文件

cd /soft/hadoop/etc/hadoop
scp -r yarn-site.xml centos@s102:/soft/hadoop/etc/hadoop
scp -r yarn-site.xml centos@s103:/soft/hadoop/etc/hadoop
scp -r yarn-site.xml centos@s104:/soft/hadoop/etc/hadoop

cd /soft/spark/conf
scp -r spark-env.sh centos@s102:/soft/spark/conf
scp -r spark-env.sh centos@s103:/soft/spark/conf
scp -r spark-env.sh centos@s104:/soft/spark/conf

------------------------------------------------

4.启动hadoop集群

cd /soft/hadoop/sbin

./start-all.sh

或：
./start-dfs.sh
./start-yarn.sh

------------------------------------------------

5.执行一个示例程序

cd /soft/spark/bin

示例1：

./spark-submit \
--class org.apache.spark.examples.SparkPi \
--master yarn \
--deploy-mode client \
./examples/jars/spark-examples_2.11-2.2.3.jar \
100

示例2，WordCount：

前提：配置好了HADOOP_HOME,HADOOP_HOME/bin,HADOOP_HOME/sbin

scala>hdfs dfs -put ./testWordCount

scala> sc.textFile("hdfs://s101:9000/testWordCount").flatMap(_.split(" "))
.map((_,1)).reduceByKey(_+_).saveAsTextFile("hdfs://s101:9000/out")

scala> hdfs dfs -cat hdfs://master01:9000/out/p*