------------------------------------------------
1.修改hadoop,yarn-site.xml,添加内容如下:
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.resourcemanager.hostname</name>
<value>s101/value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<!--是否启动一个线程检查每个任务正使用的物理内存量,如果任务超出分配值,则直接将其杀掉,默认是true -->
<property>
<name>yarn.nodemanager.pmem-check-enabled</name>
<value>false</value>
</property>
<!--是否启动一个线程检查每个任务正使用的虚拟内存量,如果任务超出分配值,则直接将其杀掉,默认是true -->
<property>
<name>yarn.nodemanager.vmem-check-enabled</name>
<value>false</value>
</property>
</configuration>
------------------------------------------------
2.修改Spark-env.sh 添加内容如下:
HADOOP_CONF_DIR=/soft/hadoop/etc/hadoop
YARN_CONF_DIR=/soft/hadoop/etc/hadoop
------------------------------------------------
3.分发配置文件
cd /soft/hadoop/etc/hadoop
scp -r yarn-site.xml centos@s102:/soft/hadoop/etc/hadoop
scp -r yarn-site.xml centos@s103:/soft/hadoop/etc/hadoop
scp -r yarn-site.xml centos@s104:/soft/hadoop/etc/hadoop
cd /soft/spark/conf
scp -r spark-env.sh centos@s102:/soft/spark/conf
scp -r spark-env.sh centos@s103:/soft/spark/conf
scp -r spark-env.sh centos@s104:/soft/spark/conf
------------------------------------------------
4.启动hadoop集群
cd /soft/hadoop/sbin
./start-all.sh
或:
./start-dfs.sh
./start-yarn.sh
------------------------------------------------
5.执行一个示例程序
cd /soft/spark/bin
示例1:
./spark-submit \
--class org.apache.spark.examples.SparkPi \
--master yarn \
--deploy-mode client \
./examples/jars/spark-examples_2.11-2.2.3.jar \
100
示例2,WordCount:
前提:配置好了HADOOP_HOME,HADOOP_HOME/bin,HADOOP_HOME/sbin
scala>hdfs dfs -put ./testWordCount
scala> sc.textFile("hdfs://s101:9000/testWordCount").flatMap(_.split(" "))
.map((_,1)).reduceByKey(_+_).saveAsTextFile("hdfs://s101:9000/out")
scala> hdfs dfs -cat hdfs://master01:9000/out/p*