Spark On Yarn 详细配置流程

最新推荐文章于 2024-04-16 07:28:44 发布
最新推荐文章于 2024-04-16 07:28:44 发布
阅读量421 收藏
点赞数
分类专栏： Spark Hadoop YARN 文章标签： hadoop spark yarn
Spark 同时被 3 个专栏收录
9 篇文章 0 订阅
订阅专栏
Hadoop
7 篇文章 1 订阅
订阅专栏
YARN
1 篇文章 0 订阅
订阅专栏

1、系统与软件准备

  • 系统：centos7
  • 软件：

  1.  hadoop-2.7.2.tar.gz，请勿选择src版本，否则需要自行编译
  2.  jdk-7u79-linux-x64.tar.gz
  3.   scala-2.10.3.tgz
  4.   spark-1.6.1-bin-hadoop2.6.tgz，选择hadooppre-built版本，否则需要安装后编译
  5.   zookeeper-3.3.6.tar.gz
2、详细配置流程说明

1，JDK配置

# tar xvzf jdk-7u45-linux-x64.tar.gz -C/usr/local 

# cd /usr/local 

# ln -s jdk1.7.0_45 jdk 

 

# vim /etc/profile 加入以下内容

[html]  view plain  copy
  1. export JAVA_HOME=/usr/local/jdk   
  2. export CLASS_PATH=$JAVA_HOME/lib:$JAVA_HOME/jre/lib   
  3. export PATH=$PATH:$JAVA_HOME/bin   

# source /etc/profile 

2. Scala安装

# tar xvzf scala-2.10.3.tgz -C/usr/local 

# cd /usr/local 

# ln -s scala-2.10.3 scala 

  

# vim /etc/profile 加入以下内容

[html]  view plain  copy
  1. export SCALA_HOME=/usr/local/scala   
  2. export PATH=$PATH:$SCALA_HOME/bin   

3. SSH免登录配置

#ssh-keygen

在node1下生成的密钥对：id_rsa和id_rsa.pub，默认存储在"~/.ssh"目录下，包括两个文件，id_rsa和id_rsa.pub，分别为私钥和公钥

  将公钥写入信任文件中

#cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

       然后修改authorized_keys文件的权限

#chmod 644 ~/.ssh/authorized_keys

       node1中的authorized_keys拷贝至其余节点的~/.ssh目录下，修改SSH配置文件"/etc/ssh/sshd_config"的下列内容（这步可以省略）

RSAAuthentication yes # 启用 RSA 认证 

PubkeyAuthentication yes # 启用公钥私钥配对认证方式 

AuthorizedKeysFile .ssh/authorized_keys # 公钥文件路径（和上面生成的文件同）

#service sshd restart

重启SSH服务

4.主机名设置

# vim /etc/hosts 

[html]  view plain  copy
  1. 10.25.248.1    node1  
  2. 10.25.248.29   node2  
  3. 10.25.248.17   node3  

#vi /etc/sysconfig/network

       HOSTNAME=node1/node2/node3


5. Zookeeper安装

# mkdir –p /root/data/hadoop/zookeeper/{data,logs}

两个文件夹都需要预先建立好，否则会运行时会报错

 

# vim /usr/local/zookeeper/conf/zoo.cfg

[html]  view plain  copy
  1. tickTime=2000  
  2. initLimit=10  
  3. syncLimit=5  
  4.    
  5. dataDir=/root/data/hadoop/zookeeper/data  
  6. clientPort=2181  
  7.    
  8. server.1=10.25.248.1:2888:3888  
  9. server.2=10.25.248.29:2888:3888  
  10. server.3=10.25.248.17:2888:3888  


接着在/root/data/hadoop/zookeeper/data下分别建立名为myid文件，文件内容为上述zoo.cfg中IP地址对应server.[number]中的number

node1 : echo 1 > /root/data/hadoop/zookeeper/data/myid

node2 : echo 2 > /root/data/hadoop/zookeeper/data/myid

node3 : echo 3 > /root/data/hadoop/zookeeper/data/myid

 

执行zkServer.sh start启动zookeeper，相应zkServer.shstop退出，zkServer.sh status查看状态

分别在node1，node2，node3执行zkServer.sh start，然后通过zkServer.sh status查看状态，如果发现每个node当前状态标记为follower或者leader，那么测试通过


6. Hadoop（HDFS HA）集群部署

# tar xvzf hadoop-2.2.0.tgz -C/usr/local

# cd /usr/local

# ln -s hadoop-2.2.0 hadoop

 

#vim /etc/profile

[html]  view plain  copy
  1. export HADOOP_HOME=/usr/local/hadoop  
  2. export HADOOP_PID_DIR=/root/data/hadoop/pids  
  3. export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native  
  4. export HADOOP_OPTS="$HADOOP_OPTS-Djava.library.path=$HADOOP_HOME/lib/native"  
  5. export HADOOP_MAPRED_HOME=$HADOOP_HOME  
  6. export HADOOP_COMMON_HOME=$HADOOP_HOME  
  7. export HADOOP_HDFS_HOME=$HADOOP_HOME  
  8. export YARN_HOME=$HADOOP_HOME  
  9. export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop  
  10. export HDFS_CONF_DIR=$HADOOP_HOME/etc/hadoop  
  11. export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop  
  12. export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native  
  13. export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin  

接下来有8个配置文件需要修改，配置文件均在$HADOOP_HOME/etc/hadoop/目录下

  1. hadoop-env.sh, mapred-env.sh, yarn-env.sh中加入以下内容

[html]  view plain  copy
  1. export JAVA_HOME=/usr/local/jdk   
  2. export CLASS_PATH=$JAVA_HOME/lib:$JAVA_HOME/jre/lib   
  3.     
  4. export HADOOP_HOME=/usr/local/hadoop   
  5. export HADOOP_PID_DIR=/root/data/hadoop/pids   
  6. export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native   
  7. export HADOOP_OPTS="$HADOOP_OPTS-Djava.library.path=$HADOOP_HOME/lib/native"   
  8.     
  9. export HADOOP_PREFIX=$HADOOP_HOME   
  10.     
  11. export HADOOP_MAPRED_HOME=$HADOOP_HOME   
  12. export HADOOP_COMMON_HOME=$HADOOP_HOME   
  13. export HADOOP_HDFS_HOME=$HADOOP_HOME   
  14. export YARN_HOME=$HADOOP_HOME   
  15.     
  16. export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop   
  17. export HDFS_CONF_DIR=$HADOOP_HOME/etc/hadoop   
  18. export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop   
  19.     
  20. export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native   
  21.     
  22. export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin  

2，core-site.xml


[html]  view plain  copy
  1. <configuration>  
  2.     <property>   
  3.         <name>fs.defaultFS</name>   
  4.         <value>hdfs://lab369</value> <!--此处不能有‘-’符号-->  
  5.     </property>   
  6.                                                                                                                                                                 
  7.     <property>   
  8.         <name>io.file.buffer.size</name>   
  9.         <value>131072</value>   
  10.     </property>   
  11.                                                                                                                                                                 
  12.     <property>   
  13.         <name>hadoop.tmp.dir</name>   
  14.         <value>file:/root/data/hadoop/storage/tmp</value>   
  15.     </property>   
  16.                                                                                                                                                                 
  17.     <property>   
  18.         <name>ha.zookeeper.quorum</name>   
  19.         <value>node1:2181,node2:2181,node3:2181</value>   
  20.     </property>   
  21.                                                                                                                                                                 
  22.     <property>   
  23.         <name>ha.zookeeper.session-timeout.ms</name>   
  24.         <value>2000</value>   
  25.     </property>   
  26.                                                                                                                                                                 
  27.     <property>   
  28.         <name>fs.trash.interval</name>   
  29.         <value>4320</value>   
  30.     </property>   
  31.                                                                                                                                                                 
  32.     <property>   
  33.          <name>hadoop.http.staticuser.use</name>   
  34.          <value>root</value>   
  35.     </property>   
  36.                                                                                                                                                                 
  37.     <property>   
  38.         <name>hadoop.proxyuser.hadoop.hosts</name>   
  39.         <value>*</value>   
  40.     </property>   
  41.                                                                                                                                                                 
  42.     <property>   
  43.         <name>hadoop.proxyuser.hadoop.groups</name>   
  44.         <value>*</value>   
  45.     </property>   
  46.                                                                                                                                                                 
  47.     <property>   
  48.         <name>hadoop.native.lib</name>   
  49.         <value>true</value>   
  50.     </property>   
  51. </configuration>  


3， hdfs-site.xml

[html]  view plain  copy
  1. <?xml version="1.0" encoding="UTF-8"?>  
  2. <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>  
  3. <!--  
  4.   Licensed under the Apache License, Version 2.0 (the "License");  
  5.   you may not use this file except in compliance with the License.  
  6.   You may obtain a copy of the License at  
  7.   
  8.     http://www.apache.org/licenses/LICENSE-2.0  
  9.   
  10.   Unless required by applicable law or agreed to in writing, software  
  11.   distributed under the License is distributed on an "AS IS" BASIS,  
  12.   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.  
  13.   See the License for the specific language governing permissions and  
  14.   limitations under the License. See accompanying LICENSE file.  
  15. -->  
  16.   
  17. <!-- Put site-specific property overrides in this file. -->  
  18.   
  19. <configuration>  
  20.     <property>   
  21.         <name>dfs.namenode.name.dir</name>   
  22.         <value>file:/root/data/hadoop/storage/hdfs/name</value>   
  23.     </property>   
  24.                                                                                                                                                     
  25.     <property>   
  26.         <name>dfs.datanode.data.dir</name>   
  27.         <value>file:/root/data/hadoop/storage/hdfs/data</value>   
  28.     </property>   
  29.                                                                                                                                                     
  30.     <property>   
  31.         <name>dfs.replication</name>   
  32.         <value>2</value>   
  33.     </property>   
  34.                                                                                                                                                     
  35.     <property>   
  36.         <name>dfs.blocksize</name>   
  37.         <value>67108864</value>   
  38.     </property>   
  39.                                                                                                                                                     
  40.     <property>   
  41.         <name>dfs.datanode.du.reserved</name>   
  42.         <value>10737418240</value>   
  43.     </property>   
  44.                                                                                                                                                     
  45.     <property>   
  46.         <name>dfs.webhdfs.enabled</name>   
  47.         <value>true</value>   
  48.     </property>   
  49.                                                                                                                                                     
  50.     <property>   
  51.         <name>dfs.permissions</name>   
  52.         <value>true</value>   
  53.     </property>   
  54.                                                                                                                                                     
  55.     <property>   
  56.         <name>dfs.permissions.enabled</name>   
  57.         <value>true</value>   
  58.     </property>   
  59.                                                                                                                                                     
  60.     <property>   
  61.         <name>dfs.nameservices</name>   
  62.         <value>lab369</value>   
  63.     </property>   
  64.                                                                                                                                                     
  65.     <property>   
  66.         <name>dfs.ha.namenodes.lab369</name>   
  67.         <value>nn1,nn2</value>   
  68.     </property>   
  69.                                                                                                                                                     
  70.     <property>   
  71.         <name>dfs.namenode.rpc-address.lab369.nn1</name>   
  72.         <value>node1:8020</value>   
  73.     </property>   
  74.                                                                                                                                                     
  75.     <property>   
  76.         <name>dfs.namenode.rpc-address.lab369.nn2</name>   
  77.         <value>node2:8020</value>   
  78.     </property>   
  79.                                                                                                                                                     
  80.     <property>   
  81.         <name>dfs.namenode.servicerpc-address.lab369.nn1</name>   
  82.         <value>node1:53310</value>   
  83.     </property>   
  84.                                                                                                                                                     
  85.     <property>   
  86.         <name>dfs.namenode.servicerpc-address.lab369.nn2</name>   
  87.         <value>node2:53310</value>   
  88.     </property>   
  89.                                                                                                                                                     
  90.     <property>   
  91.         <name>dfs.namenode.http-address.lab369.nn1</name>   
  92.         <value><span style="color:#ff0000;">node1:<span style="font-family:Arial, Helvetica, sans-serif;">50070</span></span></value> <!-- 该处不建议占掉8080端口，很多教程上都直接8080-->  
  93.     </property>   
  94.                                                                                                                                                     
  95.     <property>   
  96.         <name>dfs.namenode.http-address.lab369.nn2</name>   
  97.         <value><span style="color:#ff0000;">node2:50070</span></value>   
  98.     </property>   
  99.                                                                                                                                                     
  100.     <property>   
  101.         <name>dfs.datanode.http.address</name>   
  102.         <value><span style="color:#ff0000;">0.0.0.0:50070</span></value>   
  103.     </property>   
  104.                                                                                                                                                     
  105.     <property>   
  106.         <name>dfs.namenode.shared.edits.dir</name>   
  107.         <value>qjournal://node1:8485;node2:8485;node3:8485/lab369</value>  
  108.     </property>   
  109.                                                                                                                                                     
  110.     <property>   
  111.         <name>dfs.client.failover.proxy.provider.lab369</name>   
  112.         <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>   
  113.     </property>   
  114.                                                                                                                                                     
  115.     <property>   
  116.         <name>dfs.ha.fencing.methods</name>   
  117.         <value>sshfence</value>   
  118.     </property>   
  119.                                                                                                                                                     
  120.     <property>   
  121.         <name>dfs.ha.fencing.ssh.private-key-files</name>   
  122.         <value>/root/.ssh/id_rsa</value>   
  123.     </property>   
  124.                                                                                                                                                     
  125.     <property>   
  126.         <name>dfs.ha.fencing.ssh.connect-timeout</name>   
  127.         <value>30000</value>   
  128.     </property>   
  129.                                                                                                                                                     
  130.     <property>   
  131.         <name>dfs.journalnode.edits.dir</name>   
  132.         <value>/root/data/hadoop/storage/hdfs/journal</value>   
  133.     </property>   
  134.                                                                                                                                                     
  135.     <property>   
  136.         <name>dfs.ha.automatic-failover.enabled</name>   
  137.         <value>true</value>   
  138.     </property>   
  139.                                                                                                                                                     
  140.     <property>   
  141.         <name>ha.failover-controller.cli-check.rpc-timeout.ms</name>   
  142.         <value>60000</value>   
  143.     </property>   
  144.                                                                                                                                                     
  145.     <property>   
  146.         <name>ipc.client.connect.timeout</name>   
  147.         <value>60000</value>   
  148.     </property>   
  149.                                                                                                                                                     
  150.     <property>   
  151.         <name>dfs.image.transfer.bandwidthPerSec</name>   
  152.         <value>41943040</value>   
  153.     </property>   
  154.                                                                                                                                                     
  155.     <property>   
  156.         <name>dfs.namenode.accesstime.precision</name>   
  157.         <value>3600000</value>   
  158.     </property>   
  159.                                                                                                                                                     
  160.     <property>   
  161.         <name>dfs.datanode.max.transfer.threads</name>   
  162.         <value>4096</value>   
  163.     </property>   
  164. </configuration>  


4，mapred-site.xml

[html]  view plain  copy
  1. <configuration>   
  2.     <property>   
  3.         <name>mapreduce.framework.name</name>   
  4.         <value>yarn</value>   
  5.     </property>   
  6.                                                                                                                                          
  7.     <property>   
  8.         <name>mapreduce.jobhistory.address</name>   
  9.         <value>node1:10020</value>   
  10.     </property>   
  11.                                                                                                                                          
  12.     <property>   
  13.         <name>mapreduce.jobhistory.webapp.address</name>   
  14.         <value>node1:19888</value>   
  15.     </property>  
  16.       
  17.     <property>  
  18.         <name>mapreduce.application.classpath</name>  
  19.         <value>  
  20.         /usr/local/hadoop/etc/hadoop,  
  21.         /usr/local/hadoop/share/hadoop/common/*,  
  22.         /usr/local/hadoop/share/hadoop/common/lib/*,  
  23.         /usr/local/hadoop/share/hadoop/hdfs/*,  
  24.         /usr/local/hadoop/share/hadoop/hdfs/lib/*,  
  25.         /usr/local/hadoop/share/hadoop/mapreduce/*,  
  26.         /usr/local/hadoop/share/hadoop/mapreduce/lib/*,  
  27.         /usr/local/hadoop/share/hadoop/yarn/*,  
  28.         /usr/local/hadoop/share/hadoop/yarn/lib/*  
  29.         </value>  
  30.     </property>  
  31. </configuration>  


5，yarn-site.xml


[html]  view plain  copy
  1. <configuration>  
  2.   
  3. <!-- Site specific YARN configuration properties -->  
  4.     <property>   
  5.         <name>yarn.nodemanager.aux-services</name>   
  6.         <value>mapreduce_shuffle</value>   
  7.     </property>   
  8.                                                                                                            
  9.     <property>   
  10.         <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>   
  11.         <value>org.apache.hadoop.mapred.ShuffleHandler</value>   
  12.     </property>   
  13.                                                                   
  14.     <property>   
  15.         <name>yarn.resourcemanager.scheduler.address</name>   
  16.         <value>node1:8030</value>   
  17.     </property>   
  18.                                                                                                            
  19.     <property>   
  20.         <name>yarn.resourcemanager.resource-tracker.address</name>   
  21.         <value>node1:8031</value>  
  22.     </property>   
  23.                                                                                                            
  24.     <property>   
  25.         <name>yarn.resourcemanager.address</name>   
  26.         <value>node1:8032</value>   
  27.     </property>   
  28.                                                                                                            
  29.     <property>   
  30.         <name>yarn.resourcemanager.admin.address</name>   
  31.         <value>node1:8033</value>   
  32.     </property>                                                                                                                                                                                                                 
  33.                                                                                                            
  34.     <property>   
  35.         <name>yarn.resourcemanager.webapp.address</name>   
  36.         <value>node1:80</value>   
  37.     </property>   
  38.   
  39.     <property>  
  40.         <name>yarn.nodemanager.hostname</name>  
  41.         <value>node3</value> <!-- 每个slave应该对应自己的hostName-->  
  42.         <description>the nodemanagers bind to this port</description>  
  43.     </property>  
  44.       
  45.     <property>   
  46.         <name>yarn.nodemanager.webapp.address</name>   
  47.         <value>${yarn.nodemanager.hostname}:80</value>   
  48.     </property>   
  49.       
  50.     <property>  
  51.         <name>yarn.nodemanager.address</name>  
  52.         <value>${yarn.nodemanager.hostname}:8034</value>  
  53.         <description>the nodemanagers bind to this port</description>  
  54.     </property>  
  55.                                                                                                            
  56.     <property>   
  57.         <name>yarn.nodemanager.local-dirs</name>   
  58.         <value>${hadoop.tmp.dir}/nodemanager/local</value>   
  59.     </property>   
  60.                                                                                                            
  61.     <property>   
  62.         <name>yarn.nodemanager.remote-app-log-dir</name>   
  63.         <value>${hadoop.tmp.dir}/nodemanager/remote</value>   
  64.     </property>   
  65.                                                                                                            
  66.     <property>   
  67.         <name>yarn.nodemanager.log-dirs</name>   
  68.         <value>${hadoop.tmp.dir}/nodemanager/logs</value>   
  69.     </property>   
  70.                                                                                                            
  71.     <property>   
  72.         <name>yarn.nodemanager.log.retain-seconds</name>   
  73.         <value>604800</value>   
  74.     </property>   
  75.                                                                                                            
  76.     <property>   
  77.         <name>yarn.nodemanager.resource.cpu-vcores</name>   
  78.         <value>2</value>   
  79.     </property>   
  80.                                                                                                            
  81.     <property>   
  82.         <name>yarn.nodemanager.resource.memory-mb</name>   
  83.         <value>10240</value>   
  84.     </property>   
  85.                                                                                                            
  86.     <property>   
  87.         <name>yarn.scheduler.minimum-allocation-mb</name>   
  88.         <value>256</value>   
  89.     </property>   
  90.                                                                                                            
  91.     <property>   
  92.         <name>yarn.scheduler.maximum-allocation-mb</name>   
  93.         <value>40960</value>   
  94.     </property>   
  95.                                                                                                            
  96.     <property>   
  97.         <name>yarn.scheduler.minimum-allocation-vcores</name>   
  98.         <value>1</value>   
  99.     </property>   
  100.                                                                                                            
  101.     <property>   
  102.         <name>yarn.scheduler.maximum-allocation-vcores</name>   
  103.         <value>8</value>   
  104.     </property>   
  105. </configuration>  

6，slaves（此处如果超过一个节点，不要填主机名，要填IP）

[html]  view plain  copy
  1. 10.25.248.17  


接下来依次执行以下命令：


a)        在namenode1上执行，创建命名空间

# hdfs zkfc -formatZK

b)       在对应的节点上启动日志程序journalnode

# cd /usr/local/hadoop && ./sbin/hadoop-daemon.sh start journalnode

c)        格式化主NameNode节点（node1）

# hdfs namenode -format

d)       启动主NameNode节点

# cd /usr/local/hadoop && sbin/hadoop-daemon.sh start namenode

e)        格式备NameNode节点（node2）

# hdfs namenode -bootstrapStandby

f)         启动备NameNode节点（node2）

# cd /usr/local/hadoop && sbin/hadoop-daemon.sh start namenode

g)       在两个NameNode节点（node1、node2）上执行

# cd /usr/local/hadoop && sbin/hadoop-daemon.sh start zkfc

h)       启动所有的DataNode节点（node3）

# cd /usr/local/hadoop && sbin/hadoop-daemon.sh start datanode

i)         启动Yarn（node1）

# cd /usr/local/hadoop &&sbin/start-yarn.sh



7，Spark安装与配置

# tar xvzf spark-0.9.0-incubating.tgz-C/usr/local

# cd /usr/local

# ln -s spark-0.9.0-incubating spark

 

# vim /etc/profile

export SPARK_HOME=/usr/local/spark

export PATH=$PATH:$SPARK_HOME/bin

 

# source /etc/profile

 

# cd /usr/local/spark/conf

# mkdir -p /data/spark/tmp

接下来，配置两个配置文件，文件在SPARK_HOME/conf/目录下

1，spark-env.sh

[html]  view plain  copy
  1. export JAVA_HOME=/usr/local/jdk  
  2. export SCALA_HOME=/usr/local/scala  
  3. export HADOOP_HOME=/usr/local/hadoop  
  4. export SPARK_MASTER_PORT=7077  
  5. export SPARK_MASTER_WEBUI_PORT=7070  
  6. export SPARK_WORKER_CORES=2  
  7. export SPARK_WORKER_MEMORY=1024m  
  8. export SPARK_WORKER_INSTANCES=2  
  9.   
  10. export SPARK_LOCAL_DIR="/root/data/spark/tmp"  
  11. export SPARK_JAVA_OPTS="-Dspark.storage.blockManagerHeartBeatMs=60000-Dspark.local.dir=$SPARK_LOCAL_DIR -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:$SPARK_HOME/logs/gc.log -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:CMSInitiatingOccupancyFraction=60"  


2，slaves（多个节点不能是主机名）

[html]  view plain  copy
  1. <span style="font-size: 13.3333px;">10.25.248.29</span>  
  2. <span style="font-size: 13.3333px;">10.25.248.17</span>  

分别进入每个节点的/usr/local/spark/sbin目录下，主节点执行#./start-master.sh，子节点执行#./start-slaves.sh，或者在主节点直接执行sbin/start-all.sh


8，测试流程

1，进程运行情况测试

在每个节点执行#jps指令，若输出结果为以下内容，则测试通过，否则进入/usr/local/hadoop/logs或者/usr/local/spark/logs目录下查看log文件进行检查

主namenode


从namenode



datanode节点




其中Master和Worker是Spark相应进程
Hmaster和HregionServer为Hbase相应进程(这里不需要)
DFSZKFailoverController为Zookeeper相应进程
ResourceManager和NodeManager为Yarn相应进程

其余为Hadoop相应进程


2，HDFS测试


在任意节点下执行

#hadoopfs –mkdir /test

#hadoopfs –ls /

#hadoopfs –put /test/test.txt

若不报错，则说明测试通过


3，Spark测试

1)       Spark本地模式测试（Spark Standalone）

#run-exampleorg.apache.spark.examples.SparkPi 100

#spark-submit--class org.apache.spark.examples.JavaWordCount --master spark://node1:6066 --deploy-modecluster /usr/local/spark/lib/spark-examples-1.6.1-hadoop2.6.0.jar ./test.txt

在http://node1:4040中观察输出结果

 

#spark-submit--class org.apache.spark.examples.JavaWordCount --master spark://node1:6066 --deploy-modecluster /usr/local/spark/lib/spark-examples-1.6.1-hadoop2.6.0.jar hdfs://[hdfsnamespace]/test/test.txt

在http://node1:4040中观察输出结果


2)        Spark集群模式测试（Spark on Yarn）

#spark-submit --class org.apache.spark.examples.JavaWordCount--master yarn --deploy-mode client/usr/local/spark/lib/spark-examples-1.6.1-hadoop2.6.0.jar hdfs://[hdfsnamespace]/test/test.txt

 

#spark-submit --classorg.apache.spark.examples.JavaWordCount --master yarn --deploy-mode cluster/usr/local/spark/lib/spark-examples-1.6.1-hadoop2.6.0.jar hdfs://[hdfsnamespace]/test/test.txt

可通过http://node1:80进入UI界面查看

李天泉
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
Spark实验：On Yarn模式安装部署（带答案）1
08-03
1. 解压Spark安装包 2. 配置Hadoop生态组件相关环境变量 2. 在 master 节点上，关闭HDFS的安全模式： 3. 在 master 节点上
Spark集群环境搭建( Spark On YARN模式)
天涯芳草
04-08 2117
Spark On YARN模式的搭建比较简单,仅仅需要在YARN集群的”一个节点”上安装Spark即可, 该节点可作为提交Spark应用程序到YARN集群的客户端。Spark本身的Master节点和Worker节点不需要启动,Spark On Yarn集群的部署不依赖Standalone集群。 一：下载scala安装包 下载地址如下： https://www.scala-lang.org/d...
Spark单机伪分布式环境搭建、完全分布式环境搭建、Spark-on-yarn模式搭建
遇安的博客
03-09 2033
本文详细的写有：Spark伪分布式环境搭建、Spark完全分布式环境搭建、Spark-on-yarn模式环境搭建、Scala环境搭建的详细步骤
spark on yarn 配置
weixin_51683372的博客
12-20 621
spark on yarn 配置解压安装1.1解压Scala1.2解压Spark1.3配置环境变量1.4配置Spark1.4.1重命名1.4.2配置spark-env.sh1.4.3向yarn-site.xml添加配置1.5启动Hadoop1.5.1查看yarn web 解压安装 1.1解压Scala tar -zxvf scala-2.11.8.tgz -C /usr/project/ 1.2解压Spark tar -zxvf spark-2.4.4-bin-hadoop2.7.tgz -C /us
Spark On YARN环境配置
feizuiku0116的博客
02-06 2405
一、准备工作 点击查看Spark Standalone HA环境配置教程 二、修改配置文件 一、修改spark-env.sh cd /export/server/spark/conf vim /export/server/spark/conf/spark-env.sh # 添加以下内容 HADOOP_CONF_DIR=/export/server/hadoop-3.3.0/etc/hadoop/ YARN_CONF_DIR=/export/server/hadoop-3.3.0/etc/hadoop/
Spark on YARN 部署搭建详细图文教程
Stars.Sky 的博客
09-15 5376
Spark on YARN 部署搭建详细图文教程
Spark on yarn 配置
水彩橘子
07-04 1797
1、环境介绍 操作系统：centos 7.9 jdk版本：8u291 hadoop版本:2.10.1 spark版本：2.4.8 （因为自建了hadoop 所以使用without-hadoop包） spark下载地址：https://archive.apache.org/dist/spark/spark-2.4.8/ 注： spark on yarn 运行方式有两种 client和cluster模式： client方式：任务调度driver，运行在当前提交程序的客户机上，作业完成前不能关闭，显示输出在客
Spark on Yarn配置和Anaconda的安装
qq_56437391的博客
04-02 964
Spark on Yarn配置 1、在搭建好的Spark上修改spark-env.sh文件： 首先打开spark文件,修改spark-env.sh文件 在/opt/spark-2.3.3-bin-hadoop2.7/conf 在spark-env.sh添加以下配置： export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop export YARN_CONF_DIR=${HADOOP_HOME}/etc/hadoop ...
Spark on Yarn模式部署.docx
10-03
Spark on Yarn模式部署.docx
Spark源码系列（七）Sparkonyarn具体实现
01-30
本来不打算写的了，但是真的是闲来无事，整天看美剧也没啥意思。这一章打算讲一下Spark onyarn的实现，1.0.0里面...在第一章《spark-submit提交作业过程》的时候，我们讲过Sparkonyarn的在cluster模式下它的main clas
Spark on Yarn集群搭建手册
08-29
Spark on Yan集群搭建的详细过程，减少集群搭建的时间
03_MapReduce 和Spark on YARN.docx
02-24
■ 计算框架在Hadoop 中的作用 ■ YARN 的设计目的和基本架构 ...■ Apache Spark 概念 ■ YARN 如何分配集群资源 ■ YARN 如何处理故障 ■ 如何查看和管理YARN 应用程序 ■ 如何访问YARN 应用程序日志
Spark基础入门-第五章：环境搭建-Spark on YARN
黑马程序员官方博客
09-07 1993
Client模式和Cluster模式最最本质的区别是： Driver程序运行在哪里。Client模式：学习测试时使用，生产不推荐(要用也可以,性能略低,稳定性略低)1.Driver运行在Client上,和集群的通信成本高2.Driver输出结果会在客户端显示Cluster模式：生产环境中使用该模式1.Driver程序在YARN集群中，和集群的通信成本低2.Driver输出结果不能在客户端显示。
Spark on Yarn 环境搭建
weixin_37536020的博客
07-02 790
当Spark Application应用提交运行在YARN上时，默认情况下，每次提交应用都需要将依赖Spark相关jar包上传到YARN 集群中，为了节省提交时间和存储空间，将Spark相关jar包上传到HDFS目录中，设置属性告知Spark Application应用。计算圆周率PI的Spark程序，提交到Yarn上运行，在Centos平台上直接用spark-sumbit脚本提交程序。三台Centos7机器: 主机名分别为：node1、node2、node3。设置资源信息，提交运行pi程序至YARN上。
spark on yarn 配置、部署模式DeployMode
2202_75347029的博客
09-27 248
（2）随后Resource Manager分配Container，在合适的Node Manager上启动Application Master，此时的Application Master相当于一个Executor Laucher，只负责向Resource Manager申请Executor内存；（2）随后Resource Manager分配Container，在合适的Node Manager上启动Application Master，此时的Application Master就是Driver；
Spark on Yarn任务yarn-site.xml配置参数说明
多头注意力探索Now
04-28 1289
spark on yarn job 关于yarn参数配置问题
配置spark在yarn运行
weixin_41765352的博客
03-16 399
本文详细演示了如何部署spark在yarn上运行
.NET/C#汇总 —— WPF
最新发布
夜飞鼠的专栏
04-16 748
WPF英⽂全称是Windows Presentation Foundation，中⽂为Windows表现层基础。是微软最初 在.NET Framework3.5平台推出的⼀个图形平台。1 XAML是eXtensible Application Markup Language的英⽂缩写，相应于中⽂称为可扩展应⽤程序标 记语⾔，它是微软公司为构建应⽤程序⽤户界⾯⽽创建的⼀种新的描述性语⾔。2 XAML是XML语⾔的⼀个衍⽣物，它的语法与XML语⾔完全⼀致，它的功能就是专⻔⽤来设计和实现 程序的UI。
spark on yarn环境配置
04-30
在使用 Spark on YARN 之前，您需要完成以下环境配置： 1. 安装 Hadoop 集群，并且保证 HDFS 和 YARN 服务正常运行。 2. 下载 Spark 并解压到您的计算机上，并在所有节点上安装 Spark。 3. 配置 Spark 和 YARN 的环境变量，例如在 ~/.bashrc 中添加以下内容： ``` export SPARK_HOME=/path/to/spark export PATH=$SPARK_HOME/bin:$PATH export HADOOP_CONF_DIR=/path/to/hadoop/etc/hadoop ``` 其中，`/path/to/spark` 为您解压 Spark 的路径，`/path/to/hadoop` 为您的 Hadoop 安装路径。 4. 配置 Spark 的 YARN 集成，例如在 Spark 的 `conf/spark-defaults.conf` 文件中添加以下内容： ``` spark.master yarn spark.eventLog.enabled true spark.eventLog.dir hdfs:///spark-logs spark.yarn.historyServer.address http://your-history-server:18080 ``` 其中，`yarn` 表示您将使用 YARN 作为 Spark 的资源管理器，`/spark-logs` 为 Spark 事件日志的保存路径，`http://your-history-server:18080` 为 Spark 历史服务器的地址。 完成以上环境配置后，您可以使用 Spark on YARN 进行大规模分布式计算了。

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值