安装Spark+hadoop，spark、hadoop分布式集群搭建...(亲自搭建过！！)

最新推荐文章于 2024-09-18 10:39:18 发布
置顶 最新推荐文章于 2024-09-18 10:39:18 发布
阅读量1.5w 收藏 93
点赞数 9
分类专栏： 大数据 数据挖掘 文章标签： hadoop spark 服务器 大数据环境 分布式
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/u014552678/article/details/78584998
版权

首先说一下我所用的版本： 

spark-2.1.1-bin-hadoop2.7.tgz
hadoop-2.7.3.tar.gz
jdk-8u131-linux-x64.rpm

我们实验室有4台服务器：每个节点硬盘：300GB，内存：64GB。四个节点的hostname分别是master，slave01，slave02，slave03。

我用的是Spark做并行计算，用HDFS作为数据的分布式存储，这样的话就得安装hadoop利用里面的HDFS。如果你不用hadoop的话可以直接跳到第7步，直接安装spark即可！

1。先装java1.8环境：给各个节点上传jdk-8u131-linux-x64.rpm到/home里面。用rpm安装。

[root@localhost home]# rpm -ivh jdk-8u131-linux-x64.rpm
Preparing...                          ################################# [100%]
Updating / installing...
   1:jdk1.8.0_131-2000:1.8.0_131-fcs  ################################# [100%]
Unpacking JAR files...
        tools.jar...
        plugin.jar...
        javaws.jar...
        deploy.jar...
        rt.jar...
        jsse.jar...
        charsets.jar...
        localedata.jar...
[root@localhost home]# java -version
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)

如上：java1.8 安装成功！！

2。集群核准时间：(如果集群时间一致的话，此步略过！)
时间必须同步，因为节点之间要发送心跳，如果时间不一致的话，会产生错误。
用date -s 命令也行！(下面是ntp服务器来同步时间)

##在每个节点上执行安装ntp服务
[hadoop@master ~]$ sudo yum install -y ntp
##在每个节点上同时执行`sudo ntpdate us.pool.ntp.org`
[hadoop@master ~]$ sudo ntpdate us.pool.ntp.org
 5 Oct 18:19:41 ntpdate[2997]: step time server 138.68.46.177 offset -6.006070 sec

或者也可以在某个节点上启动一个ntp服务器：

##在每个节点上执行安装ntp服务
[hadoop@master ~]$ sudo yum install -y ntp
##在192.168.2.219节点上执行`sudo ntpdate us.pool.ntp.org`把这个节点作为ntp同步服务器
[hadoop@master ~]$ sudo ntpdate us.pool.ntp.org
 5 Oct 18:19:41 ntpdate[2997]: step time server 138.68.46.177 offset -6.006070 sec
##在各个节点上开启ntp服务
[hadoop@master ~]$ sudo service ntpd start
Redirecting to /bin/systemctl start  ntpd.service
##在其他节点上同步192.168.2.219节点ntp服务器上的时间。
[hadoop@slave01 ~]$ sudo ntpdate 192.168.2.219
 5 Oct 18:27:45 ntpdate[3014]: adjust time server 192.168.147.6 offset -0.001338 sec

3。添加用户hadoop：

[root@localhost etc]# useradd -m hadoop -s /bin/bash
useradd: user 'hadoop' already exists
[root@localhost etc]# passwd hadoop
Changing password for user hadoop.
New password: 
BAD PASSWORD: The password fails the dictionary check - it is too simplistic/systematic
Retype new password: 
passwd: all authentication tokens updated successfully.
[root@localhost etc]# su - hadoop
[hadoop@localhost ~]$

4。给hadoop用户增加管理员权限，方便部署

[root@localhost ~]#visudo

找到 root ALL=(ALL) ALL 这行（应该在第98行，可以先按一下键盘上的 ESC 键，然后输入 :98 (按一下冒号，接着输入98，再按回车键)，可以直接跳到第98行 ），然后在这行下面增加一行内容：hadoop ALL=(ALL) ALL （当中的间隔为tab），如下图所示：
这里写图片描述

5。SSH无密通信：

[root@master .ssh]#su - hadoop
[hadoop@master ~]$
[hadoop@master ~]$ ssh localhost    # 如果没有该目录，先执行一次ssh localhost
[hadoop@master ~]$ cd ~/.ssh 
[hadoop@master ~]$ rm ./id_rsa*    # 删除之前生成的公匙（如果有）
[hadoop@master ~]$ ssh-keygen -t rsa  # 一直按回车就可以
让 Master 节点需能无密码 SSH 本机，在 Master 节点上执行：
[hadoop@master .ssh]$ cat ./id_rsa.pub >> ./authorized_keys
完成后可执行 ssh Master 验证一下（可能需要输入 yes，成功后执行 exit 返回原来的终端）。接着在 master 节点将上公匙传输到 slave01,slave02,slave03 节点：
[hadoop@master .ssh]$ scp ~/.ssh/id_rsa.pub hadoop@slave01:/home/hadoop/
[hadoop@master .ssh]$ scp ~/.ssh/id_rsa.pub hadoop@slave02:/home/hadoop/
[hadoop@master .ssh]$ scp ~/.ssh/id_rsa.pub hadoop@slave03:/home/hadoop/
接着在 slave01,slave02,slave03节点上，将 ssh 公匙加入授权：【分别在其他三个节点上执行以下命令：】
[hadoop@slave03 ~]$ mkdir ~/.ssh   
[hadoop@slave03 ~]$ cat ~/id_rsa.pub >> ~/.ssh/authorized_keys
[hadoop@slave03 ~]$ rm ~/id_rsa.pub
然后在master上执行ssh slave01  但是还是不行解决方法：

常见免密码登录失败分析
配置问题
检查配置文件/etc/ssh/sshd_config是否开启了AuthorizedKeysFile选项
检查AuthorizedKeysFile选项指定的文件是否存在并内容正常
目录权限问题
~权限设置为700
~/.ssh权限设置为700
~/.ssh/authorized_keys的权限设置为600

sudo chmod 700 ~
sudo chmod 700 ~/.ssh
sudo chmod 600 ~/.ssh/authorized_keys

6。安装hadoop：
这里写图片描述
下面是hosts文件内容：
192.168.2.189 slave01
192.168.2.240 slave02
192.168.2.176 slave03
192.168.2.219 master
hadoop-2.7.3.tar.gz放在~下。【一般安装文件都放在~下面】
我们选择将 Hadoop 安装至 /usr/local/ 中：

[hadoop@master ~]$  sudo tar -zxf ~/hadoop-2.7.3.tar.gz -C /usr/local   # 解压到/usr/local中
[hadoop@master ~]$ cd /usr/local/
[hadoop@master ~]$ sudo mv ./hadoop-2.7.3/ ./hadoop         # 将文件夹名改为hadoop
[hadoop@master ~]$ sudo chown -R hadoop:hadoop ./hadoop           # 修改文件权限

Hadoop 解压后即可使用。输入如下命令来检查 Hadoop 是否可用，成功则会显示 Hadoop 版本信息：

[hadoop@master local]$ cd /usr/local/hadoop
[hadoop@master hadoop]$ ./bin/hadoop version
Hadoop 2.7.3
Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r baa91f7c6bc9cb92be5982de4719c1c8af91ccff
Compiled by root on 2016-08-18T01:41Z
Compiled with protoc 2.5.0
From source with checksum 2e4ce5f957ea4db193bce3734ff29ff4
This command was run using /usr/local/hadoop/share/hadoop/common/hadoop-common-2.7.3.jar

6.1。 Hadoop单机配置(非分布式) ，注：先把一个节点的hadoop装好后，然后再依次拷贝到其他的节点上。
Hadoop 默认模式为非分布式模式，无需进行其他配置即可运行。非分布式即单 Java 进程，方便进行调试。
现在我们可以执行例子来感受下 Hadoop 的运行。Hadoop 附带了丰富的例子（运行 ./bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar 可以看到所有例子），包括 wordcount、terasort、join、grep 等。
在此我们选择运行 grep 例子，我们将 input 文件夹中的所有文件作为输入，筛选当中符合正则表达式 dfs[a-z.]+ 的单词并统计出现的次数，最后输出结果到 output 文件夹中。

[hadoop@master hadoop]$ cd /usr/local/hadoop
[hadoop@master hadoop]$ mkdir ./input
[hadoop@master hadoop]$ cp ./etc/hadoop/*.xml ./input # 将配置文件作为输入文件
[hadoop@master hadoop]$./bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar grep ./input ./output 'dfs[a-z.]+'
[hadoop@master hadoop]$ cat ./output/*       # 查看运行结果
1       dfsadmin

6.2。修改/usr/local/hadoop/etc/hadoop/slaves:这里配置的是三个运行节点，master节点只做master不作为运行节点。

[hadoop@master hadoop]$ vi slaves      #里面内容是：
slave01
slave02
slave03

6.3。文件 core-site.xml 改为下面的配置：

<configuration>
      <property>
          <name>hadoop.tmp.dir</name>
          <value>/usr/local/hadoop/tmp</value>
          <description>Abase for other temporary directories.</description>
      </property>
      <property>
          <name>fs.defaultFS</name>
          <value>hdfs://master:9000</value>
      </property>
</configuration>

6.4。文件 hdfs-site.xml，dfs.replication 一般设为 3：

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>3</value>
    </property>
</configuration>

6.5。文件 mapred-site.xml （可能需要先重命名，默认文件名为 mapred-site.xml.template），然后配置修改如下：

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
</configuration>

6.6。文件 yarn-site.xml：

<configuration>
  <!-- Site specific YARN configuration properties -->
      <property>
          <name>yarn.nodemanager.aux-services</name>
          <value>mapreduce_shuffle</value>
      </property>
      <property>
          <name>yarn.resourcemanager.hostname</name>
          <value>master</value>
      </property>
       <property>
          <name>yarn.nodemanager.pmem-check-enabled</name>
          <value>false</value>
      </property>
      <property>
          <name>yarn.nodemanager.vmem-check-enabled</name>
          <value>false</value>
      </property>
</configuration>

6.7。配置好后，将 master 上的 /usr/local/Hadoop 文件夹复制到各个节点上。在 master 节点上执行：

之前有跑过伪分布式模式，建议在切换到集群模式前先删除之前的临时文件。
[hadoop@master ~]$cd /usr/local
[hadoop@master ~]$sudo rm -r ./hadoop/tmp     # 删除 Hadoop 临时文件
[hadoop@master ~]$sudo rm -r ./hadoop/logs/*   # 删除日志文件
[hadoop@master ~]$tar -zcf ~/hadoop.master.tar.gz ./hadoop # 先压缩再复制
[hadoop@master ~]$cd ~
[hadoop@master ~]$scp ./hadoop.master.tar.gz slave01:/home/hadoop
[hadoop@master ~]$scp ./hadoop.master.tar.gz slave02:/home/hadoop
[hadoop@master ~]$scp ./hadoop.master.tar.gz slave03:/home/hadoop

6.8。在各个节点执行：
在 slave01 节点上执行：

[hadoop@slave01 ~]$ sudo rm -r /usr/local/hadoop       # 删掉旧的（如果存在）
[hadoop@slave01 ~]$ sudo tar -zxf ~/hadoop.master.tar.gz -C /usr/local
[hadoop@slave01 ~]$ sudo chown -R hadoop /usr/local/hadoop

在 slave02 节点上执行：

[hadoop@slave02 ~]$ sudo rm -r /usr/local/hadoop       # 删掉旧的（如果存在）
[hadoop@slave02 ~]$ sudo tar -zxf ~/hadoop.master.tar.gz -C /usr/local
[hadoop@slave02 ~]$ sudo chown -R hadoop /usr/local/hadoop

在 slave03 节点上执行：

[hadoop@slave03 ~]$ sudo rm -r /usr/local/hadoop       # 删掉旧的（如果存在）
[hadoop@slave03 ~]$ sudo tar -zxf ~/hadoop.master.tar.gz -C /usr/local
[hadoop@slave03 ~]$ sudo chown -R hadoop /usr/local/hadoop

6.9。首次启动需要先在 Master 节点执行 NameNode 的格式化：

[hadoop@master ~]$ hdfs namenode -format     #首次运行需要执行初始化，之后不需要

6.10。CentOS系统需要关闭防火墙
CentOS系统默认开启了防火墙，在开启 Hadoop 集群之前，需要关闭集群中每个节点的防火墙。有防火墙会导致 ping 得通但 telnet 端口不通，从而导致 DataNode 启动了，但 Live datanodes 为 0 的情况。
在 CentOS 中，可以通过如下命令关闭防火墙：

在 CentOS 6.x 中，可以通过如下命令关闭防火墙：
sudo service iptables stop   # 关闭防火墙服务
sudo chkconfig iptables off  # 禁止防火墙开机自启，就不用手动关闭了
Shell 命令
若用是 CentOS 7，需通过如下命令关闭（防火墙服务改成了 firewall）：
systemctl stop firewalld.service    # 关闭firewall
systemctl disable firewalld.service # 禁止firewall开机启动

6.11。接着可以启动 hadoop 了，启动需要在 master 节点上进行：
注意修改：vi /usr/local/hadoop/etc/hadoop/hadoop-env.sh ，
把export JAVA_HOME=${JAVA_HOME}改为：export JAVA_HOME=/usr/java/jdk1.8.0_131/

在/usr/local/hadoop/sbin   启动hadoop      ./start-all.sh   
[hadoop@master sbin]$ ./start-all.sh 
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
Starting namenodes on [master]
master: starting namenode, logging to /usr/local/hadoop/logs/hadoop-hadoop-namenode-master.out
slave03: starting datanode, logging to /usr/local/hadoop/logs/hadoop-hadoop-datanode-slave03.out
slave01: starting datanode, logging to /usr/local/hadoop/logs/hadoop-hadoop-datanode-slave01.out
slave02: starting datanode, logging to /usr/local/hadoop/logs/hadoop-hadoop-datanode-slave02.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop/logs/hadoop-hadoop-secondarynamenode-master.out
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop/logs/yarn-hadoop-resourcemanager-master.out
slave01: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-hadoop-nodemanager-slave01.out
slave02: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-hadoop-nodemanager-slave02.out
slave03: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-hadoop-nodemanager-slave03.out

然后jps

[hadoop@master hadoop]$ jps
6194 ResourceManager
5717 NameNode
5960 SecondaryNameNode
6573 Jps    
[hadoop@slave01 hadoop]$ jps
4888 Jps
4508 DataNode
4637 NodeManager
[hadoop@slave02 hadoop]$ jps
3841 DataNode
3970 NodeManager
4220 Jps    
[hadoop@slave03 hadoop]$ jps
4032 NodeManager
4282 Jps
3903 DataNode

6.12. 打开hadoop WEBUI
在浏览器中输入http://192.168.2.219:50070 【注意浏览器要与192.168.2.219为局域网】
这里写图片描述

6.13。执行分布式实例

首先创建 HDFS 上的用户目录：
[hadoop@master hadoop]$ hdfs dfs -mkdir -p /user/hadoop
将 /usr/local/hadoop/etc/hadoop 中的配置文件作为输入文件复制到分布式文件系统中：
[hadoop@master hadoop]$ hdfs dfs -mkdir input
[hadoop@master hadoop]$ hdfs dfs -put /usr/local/hadoop/etc/hadoop/*.xml input

通过查看 DataNode 的状态（占用大小有改变），输入文件确实复制到了 DataNode 中，如下图所示：
这里写图片描述
接着就可以运行 MapReduce 作业了：【注意运行前要保证节点时间一致】

####命令：
hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar grep input output 'dfs[a-z.]+'
----------
####执行过程运行的log如下：
[hadoop@master hadoop]$ hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar grep input output 'dfs[a-z.]+'
17/11/13 22:26:21 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.2.219:8032
17/11/13 22:26:21 INFO input.FileInputFormat: Total input paths to process : 9
17/11/13 22:26:21 INFO mapreduce.JobSubmitter: number of splits:9
17/11/13 22:26:21 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1510581226826_0004
17/11/13 22:26:22 INFO impl.YarnClientImpl: Submitted application application_1510581226826_0004
17/11/13 22:26:22 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1510581226826_0004/
17/11/13 22:26:22 INFO mapreduce.Job: Running job: job_1510581226826_0004
17/11/13 22:26:28 INFO mapreduce.Job: Job job_1510581226826_0004 running in uber mode : false
17/11/13 22:26:28 INFO mapreduce.Job:  map 0% reduce 0%
17/11/13 22:26:32 INFO mapreduce.Job:  map 33% reduce 0%
17/11/13 22:26:33 INFO mapreduce.Job:  map 100% reduce 0%
17/11/13 22:26:37 INFO mapreduce.Job:  map 100% reduce 100%
17/11/13 22:26:37 INFO mapreduce.Job: Job job_1510581226826_0004 completed successfully
17/11/13 22:26:37 INFO mapreduce.Job: Counters: 50
        File System Counters
                FILE: Number of bytes read=51
                FILE: Number of bytes written=1190205
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=28817
                HDFS: Number of bytes written=143
                HDFS: Number of read operations=30
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=2
        Job Counters 
                Killed map tasks=1
                Launched map tasks=9
                Launched reduce tasks=1
                Data-local map tasks=9
                Total time spent by all maps in occupied slots (ms)=26894
                Total time spent by all reduces in occupied slots (ms)=2536
                Total time spent by all map tasks (ms)=26894
                Total time spent by all reduce tasks (ms)=2536
                Total vcore-milliseconds taken by all map tasks=26894
                Total vcore-milliseconds taken by all reduce tasks=2536
                Total megabyte-milliseconds taken by all map tasks=27539456
                Total megabyte-milliseconds taken by all reduce tasks=2596864
        Map-Reduce Framework
                Map input records=796
                Map output records=2
                Map output bytes=41
                Map output materialized bytes=99
                Input split bytes=1050
                Combine input records=2
                Combine output records=2
                Reduce input groups=2
                Reduce shuffle bytes=99
                Reduce input records=2
                Reduce output records=2
                Spilled Records=4
                Shuffled Maps =9
                Failed Shuffles=0
                Merged Map outputs=9
                GC time elapsed (ms)=762
                CPU time spent (ms)=7040
                Physical memory (bytes) snapshot=2680807424
                Virtual memory (bytes) snapshot=19690971136
                Total committed heap usage (bytes)=1957691392
        Shuffle Errors
                BAD_ID=0
                CONNECTION=0
                IO_ERROR=0
                WRONG_LENGTH=0
                WRONG_MAP=0
                WRONG_REDUCE=0
        File Input Format Counters 
                Bytes Read=27767
        File Output Format Counters 
                Bytes Written=143
17/11/13 22:26:37 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.2.219:8032
17/11/13 22:26:37 INFO input.FileInputFormat: Total input paths to process : 1
17/11/13 22:26:37 INFO mapreduce.JobSubmitter: number of splits:1
17/11/13 22:26:37 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1510581226826_0005
17/11/13 22:26:37 INFO impl.YarnClientImpl: Submitted application application_1510581226826_0005
17/11/13 22:26:37 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1510581226826_0005/
17/11/13 22:26:37 INFO mapreduce.Job: Running job: job_1510581226826_0005
17/11/13 22:26:48 INFO mapreduce.Job: Job job_1510581226826_0005 running in uber mode : false
17/11/13 22:26:48 INFO mapreduce.Job:  map 0% reduce 0%
17/11/13 22:26:52 INFO mapreduce.Job:  map 100% reduce 0%
17/11/13 22:26:57 INFO mapreduce.Job:  map 100% reduce 100%
17/11/13 22:26:58 INFO mapreduce.Job: Job job_1510581226826_0005 completed successfully
17/11/13 22:26:58 INFO mapreduce.Job: Counters: 49
        File System Counters
                FILE: Number of bytes read=51
                FILE: Number of bytes written=237047
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=271
                HDFS: Number of bytes written=29
                HDFS: Number of read operations=7
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=2
        Job Counters 
                Launched map tasks=1
                Launched reduce tasks=1
                Data-local map tasks=1
                Total time spent by all maps in occupied slots (ms)=2331
                Total time spent by all reduces in occupied slots (ms)=2600
                Total time spent by all map tasks (ms)=2331
                Total time spent by all reduce tasks (ms)=2600
                Total vcore-milliseconds taken by all map tasks=2331
                Total vcore-milliseconds taken by all reduce tasks=2600
                Total megabyte-milliseconds taken by all map tasks=2386944
                Total megabyte-milliseconds taken by all reduce tasks=2662400
        Map-Reduce Framework
                Map input records=2
                Map output records=2
                Map output bytes=41
                Map output materialized bytes=51
                Input split bytes=128
                Combine input records=0
                Combine output records=0
                Reduce input groups=1
                Reduce shuffle bytes=51
                Reduce input records=2
                Reduce output records=2
                Spilled Records=4
                Shuffled Maps =1
                Failed Shuffles=0
                Merged Map outputs=1
                GC time elapsed (ms)=110
                CPU time spent (ms)=1740
                Physical memory (bytes) snapshot=454008832
                Virtual memory (bytes) snapshot=3945603072
                Total committed heap usage (bytes)=347078656
        Shuffle Errors
                BAD_ID=0
                CONNECTION=0
                IO_ERROR=0
                WRONG_LENGTH=0
                WRONG_MAP=0
                WRONG_REDUCE=0
        File Input Format Counters 
                Bytes Read=143
        File Output Format Counters 
                Bytes Written=29
----------
查看运行结果：
[hadoop@master hadoop]$ hdfs dfs -cat output/*
1       dfsadmin
1       dfs.replication

运行时的输出信息与伪分布式类似，会显示 Job 的进度。
可能会有点慢，但如果迟迟没有进度，比如 5 分钟都没看到进度，那不妨重启 Hadoop 再试试。
6.14。关闭 Hadoop 集群也是在 Master 节点上执行的：./sbin/stop-all.sh 即可

[hadoop@master sbin]$ stop-all.sh 
This script is Deprecated. Instead use stop-dfs.sh and stop-yarn.sh
Stopping namenodes on [master]
master: stopping namenode
slave01: stopping datanode
slave03: stopping datanode
slave02: stopping datanode
Stopping secondary namenodes [0.0.0.0]
0.0.0.0: stopping secondarynamenode
stopping yarn daemons
stopping resourcemanager
slave01: stopping nodemanager
slave02: stopping nodemanager
slave03: stopping nodemanager
slave01: nodemanager did not stop gracefully after 5 seconds: killing with kill -9
slave02: nodemanager did not stop gracefully after 5 seconds: killing with kill -9
slave03: nodemanager did not stop gracefully after 5 seconds: killing with kill -9
no proxyserver to stop

附录：增加一个master节点作为slaves ，这样运行节点就变成4个了，进入/usr/local/hadoop/etc/hadoop/修改slaves为：

master
slave01
slave02
slave03

$PWD就是当前目录，把此目录的slaves拷贝到其他三个节点上进行覆盖。

[hadoop@master hadoop]$ scp slaves hadoop@slave01:$PWD
[hadoop@master hadoop]$ scp slaves hadoop@slave02:$PWD
[hadoop@master hadoop]$ scp slaves hadoop@slave03:$PWD

再启动：

[hadoop@master hadoop]$ ./sbin/start-all.sh 
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
Starting namenodes on [master]
master: starting namenode, logging to /usr/local/hadoop/logs/hadoop-hadoop-namenode-master.out
master: starting datanode, logging to /usr/local/hadoop/logs/hadoop-hadoop-datanode-master.out
slave02: starting datanode, logging to /usr/local/hadoop/logs/hadoop-hadoop-datanode-slave02.out
slave01: starting datanode, logging to /usr/local/hadoop/logs/hadoop-hadoop-datanode-slave01.out
slave03: starting datanode, logging to /usr/local/hadoop/logs/hadoop-hadoop-datanode-slave03.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop/logs/hadoop-hadoop-secondarynamenode-master.out
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop/logs/yarn-hadoop-resourcemanager-master.out
slave02: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-hadoop-nodemanager-slave02.out
slave01: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-hadoop-nodemanager-slave01.out
master: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-hadoop-nodemanager-master.out
slave03: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-hadoop-nodemanager-slave03.out

这里写图片描述
至此hadoop安装完成！！！！

7。安装spark
先把spark-2.1.1-bin-hadoop2.7.tgz传到~里。
执行命令：

[hadoop@master ~]$ sudo tar -zxf ~/spark-2.1.1-bin-hadoop2.7.tgz -C /usr/local/
[hadoop@master ~]$ cd /usr/local
[hadoop@master ~]$ sudo cp ./spark-2.1.1-bin-hadoop2.7.tgz/ ./spark
[hadoop@master ~]$ sudo chown -R hadoop:hadoop ./spark

7.1。 在/usr/local/spark/conf里，修改spark-env.sh添加：

export JAVA_HOME=/usr/java/jdk1.8.0_131
export SPARK_MASTER_IP=192.168.2.219
export SPARK_MASTER_PORT=7077

7.2。在/usr/local/spark/conf里，添加内容到slaves，这里有4个运行节点把master也算进去了，master既做管理又做计算

[hadoop@master conf]$ cat slaves
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# A Spark Worker will be started on each of the machines listed below.
master
slave01
slave02
slave03

7.3。 配置好后，将Master主机上的/usr/local/spark文件夹复制到各个节点上。在Master主机上执行如下命令：

[hadoop@master local]$ cd /usr/local/
压缩spark包
[hadoop@master local]$ tar -zcf ~/spark.master.tar.gz ./spark
[hadoop@master local]$ cd ~
把spark压缩包远程传到其他节点
[hadoop@master local]$ scp ./spark.master.tar.gz slave01:/home/hadoop
[hadoop@master local]$ scp ./spark.master.tar.gz slave02:/home/hadoop
[hadoop@master local]$ scp ./spark.master.tar.gz slave03:/home/hadoop

在slave01,slave02，slave03节点上分别执行下面同样的操作：

[hadoop@slave01 spark]$ sudo tar -zxf ~/spark.master.tar.gz -C /usr/local
[hadoop@slave01 spark]$ sudo chown -R hadoop:hadoop /usr/local/spark

7.4。启动Spark集群
启动Hadoop集群
启动Spark集群前，要先启动Hadoop集群。在Master节点主机上运行如下命令：

[hadoop@master ~]$ cd /usr/local/hadoop/
[hadoop@master ~]$ ./sbin/start-all.sh

启动Spark集群

[hadoop@master spark]$ ./sbin/start-all.sh 
starting org.apache.spark.deploy.master.Master, logging to /usr/local/spark/logs/spark-hadoop-org.apache.spark.deploy.master.Master-1-master.out
slave02: starting org.apache.spark.deploy.worker.Worker, logging to /usr/local/spark/logs/spark-hadoop-org.apache.spark.deploy.worker.Worker-1-slave02.out
master: starting org.apache.spark.deploy.worker.Worker, logging to /usr/local/spark/logs/spark-hadoop-org.apache.spark.deploy.worker.Worker-1-master.out
slave03: starting org.apache.spark.deploy.worker.Worker, logging to /usr/local/spark/logs/spark-hadoop-org.apache.spark.deploy.worker.Worker-1-slave03.out
slave01: starting org.apache.spark.deploy.worker.Worker, logging to /usr/local/spark/logs/spark-hadoop-org.apache.spark.deploy.worker.Worker-1-slave01.out
[hadoop@master spark]$ jps
4641 Master
4068 ResourceManager
3447 NameNode
4807 Worker
3608 DataNode
3832 SecondaryNameNode
4938 Jps
4207 NodeManager

在浏览器上查看Spark独立集群管理器的集群信息，在192.168.2.219主机上打开浏览器，访问http://192.168.2.219:8080,如下图：
这里写图片描述
安装成功！！！

7.5。关闭Spark集群
关闭spark：./stop-all.sh
关闭hadoop：./sbin/stop-all.sh

[hadoop@master spark]$ ./sbin/stop-all.sh 
master: stopping org.apache.spark.deploy.worker.Worker
slave03: stopping org.apache.spark.deploy.worker.Worker
slave02: stopping org.apache.spark.deploy.worker.Worker
slave01: stopping org.apache.spark.deploy.worker.Worker
stopping org.apache.spark.deploy.master.Master
[hadoop@master spark]$ cd ..
[hadoop@master local]$ cd hadoop
[hadoop@master spark]$ ./sbin/stop-all.sh 
master: stopping org.apache.spark.deploy.worker.Worker
slave03: stopping org.apache.spark.deploy.worker.Worker
slave02: stopping org.apache.spark.deploy.worker.Worker
slave01: stopping org.apache.spark.deploy.worker.Worker
stopping org.apache.spark.deploy.master.Master
[hadoop@master hadoop]$ ./sbin/stop-all.sh 
This script is Deprecated. Instead use stop-dfs.sh and stop-yarn.sh
Stopping namenodes on [master]
master: stopping namenode
master: stopping datanode
slave02: stopping datanode
slave03: stopping datanode
slave01: stopping datanode
Stopping secondary namenodes [0.0.0.0]
0.0.0.0: stopping secondarynamenode
stopping yarn daemons
stopping resourcemanager
slave03: stopping nodemanager
slave02: stopping nodemanager
slave01: stopping nodemanager
master: stopping nodemanager
no proxyserver to stop

集群已经停止！！

完成！有什么意见和问题请站内评价和提问。谢谢！

MENG哥
关注
专栏目录
spark与hadoop集成详解
reylee的博客
06-03 5292
我们经常被问到如何让spark集成到hadoop生态系统中，如何让spark在现有的hadoop集群中运行，本文的目的就是来回答上述问题。 1，spark主要用于提高而不是取代hadoop栈，从一开始spark就被设计从hdfs中读取存储数据，类似于其他的存储系统，例如Hbase, Amazon S3等，因此，hadoop用户可以通过结合spark来提高hadoop MR, Hbase 及其他大
搭建hadoop+spark完全分布式集群环境
weixin_46305053的博客
07-13 1354
tar -zxvf jdk-8u181-linux-x64.tar.gz -C /home/hadoop/apps #解压到apps下。进入/home/hadoop/apps/hadoop-2.7.6/etc/Hadoop。sudo vi /etc/profile #修改配置文件。start-slaves.sh //启动从节点。start-master.sh //启动主节点。拷贝到slave1和slave2中。拷贝到slaves1,slaves中。//将其拷贝到其他机子上。
Hadoop和spark集群搭建详解
12-04
hadoop与spark集群搭建，了解hadoop分布式、伪分布式等方式集群搭建
【大数据】深入了解Hadoop
最新发布
m0_46181896的博客
09-18 5596
hdfs的元数据两个部分内存上的元数据在服务运行期间,所有的元数据会先存储在内存上文件信息,datanode信息,块信息元数据文件为了避免内存上的元数据丢失,会将内存的上的元数据保存在磁盘上secondarynamenode完成元数据文件的保存存储位置在hadoop的指定数据edits_xxxxx 编辑日志文件,记录所有的文件操作信息fsimage_xxxxx 镜像文件,保存了元数据内容 文件和块内容datanode会汇报自身状态给NameNode,默认是3秒。
基于Hadoop集群的Spark集群搭建
青澜饮舟的博客
05-31 2649
基于Hadoop集群的Spark集群搭建 一. 单机版Spark搭建 (1)将下载下来的软件包拉至集群中(主节点上) (2)解压至指定目录 (3)将解压下的文件移至某个特定目录下并查看 (4)修改环境变量 (5)使环境变量生效并进入scala命令行测试； (6）解压scala压缩包并放置指定目录； (7)添加环境变量 (8)生效环境变量并测试spark(source .bash_pr...
Spark+Hadoop环境搭建
m0_67392811的博客
08-24 1454
注意： yarn 是作为资源管理器存在，每次提交一个任务时， spark 会做两件事，一个是将代码分成多个 stage ，每个 stage 相当于一个计算，然后再把 stage 分成多个 task ，每个 task 执行相同的操作，只不过操作的数据不同，这就和之前说的每个 DataNode 里面都存放着同一个文件的不同块联系起来了。1.选择几个ubuntu的虚拟机，其中一个作为spark的master节点以及hadoop的 NameNode节点，其他的作为slave节点和DataNode节点。
Spark入门——Hadoop与Spark
beilizhang的博客
08-12 440
参考链接 https://www.bilibili.com/video/BV11A411L7CK?p=1 Hadoop Hadoop是由Java语言编写的，在分布式服务器集群上存储海量数据并运行分布式分析应用的开源框架 作为Hadoop分布式文件系统，HDFS处于Hadoop生态圈的最下层，存储着所有的数据，支持着Hadoop的所有服务。它的理论基础源于Google的TheGoogleFileSystem这篇论文，它是GFS的开源实现 MapReduce是一种编程模型，Hadoop根据Google的Ma
Hadoop/Spark集群搭建图文全攻略
no_hot的博客
03-26 1027
完全分布式集群搭建一、安装VMware二、创建Linux虚拟机三、CentOS-7安装 一、安装VMware 双击打开安装包，点击“下一步” 勾选“我接受许可协议中的条款”，点击“下一步” 设置安装路径，建议提前新建文件夹，并重命名(不建议安装在C盘)，点击“下一步” 取消掉两个勾选，点击“下一步” 直接点击“下一步” 点击“安装”，开始安装 等待安装完成 安装完成 打开VMware...
Spark+hadoop+scala 完全分布式集群搭建过程及启动过程
07-08
对于想学习 Spark 的人而言，如何构建 Spark 集群是其最大的难点之一， 为了解决大家构建 Spark 集群的一切困难，Spark 集群的构建分为了五个步骤，从 零起步，不需要任何前置知识，涵盖操作的每一个细节，构建完整...
VM+CentOS+hadoop2.7搭建hadoop完全分布式集群
01-24
### 基于VM+CentOS+hadoop2.7搭建Hadoop完全分布式集群的知识点解析 #### 一、概述 本文旨在详细介绍如何使用VMware Workstation（简称VM）结合CentOS 6.5操作系统来搭建Hadoop 2.7的完全分布式集群。此文档是根据...
hadoop+spark分布式集群搭建及spark程序示例.doc
07-02
hadoop+spark分布式集群搭建及spark程序示例，例子程序为用二项逻辑斯蒂回归进行二分类分析和一个简单的求平均的程序，两种不同的运行方式
hadoop&spark; 环境搭建教程
02-01
自己整理的一份hadoop及spark 环境搭建的详细教程，真实环境下搭建过程实录， 可以为你免去很多麻烦
spark 与 hadoop的hdfs的连接（亲测有效）
热门推荐
AStar的博客
04-03 1万+
出处：：http://blog.csdn.net/oopsoom/article/details/24257981， 目前Spark的Run on的Hadoop版本大多是hadoop2以上，但是实际上各个公司的生产环境不尽相同，用到2.0以上的公司还是少数。 大多数公司还是停留在1代hadoop上，所以我就拿spark0.91 +  hadoop0.20.2-cdh3u
Hadoop系列 (六)：Spark搭建
Eric Ray的博客
10-14 1147
基于hadoop的spark搭建
【Hadoop】hadoop+spark配置
dominic_z的博客
03-28 5442
不是教程，教程请看http://www.cnblogs.com/zyrblog/p/8510506.html 只是个人笔记 准备材料 一台能联网的电脑 VMware 14 Centos 7 1801 64bit 我还是喜欢centos多一些，old fashion，好吧其实是装了一次ubuntu发现里面缺的东西有点多，ifconfig，vim都没有还得自己安，烦 jdk-8u191-linux-...
spark环境，hadoop集群搭建
Nick_Jackson的博客
07-01 707
spark环境，hadoop集群搭建 环境准备 因为Ubuntu服务器在国外，可能国内访问下载速度较慢，因此可以换成阿里源，通过以下命令对源进行更换 阿里源地址：http://mirrors.aliyun.com/ubuntu/dists/ 查看当前系统的代号 lsb_release -a # 1、备份原来的源 cp -ra /etc/apt/sources.list /etc/apt/sources.list.bak # 2、修改/etc/apt/sources.list文件，可将原来的内容删除或注
hadoop与spark---MyConclusion（3）
weixin_42051034的博客
07-13 124
hadoop与spark参考博客链接 参考博客链接 1.hadoop与spark学习笔记 2.spark 介绍 3.hadoop与spark区别和联系
Hadoop+Spark集群搭建
oKoukou12345678的博客
07-23 2515
搭建Hadoop+Spark集群的方法这篇文章主要是介绍了怎么在Linux平台下搭建Hadoop与Spark集群，得声明一点的是Spark并不依赖于Hadoop。
【 大数据分析Hadoop + Spark 】10分钟搭建Hadoop（伪分布式 ）+ Spark（Local模式）环境
qq_56022768的博客
05-28 4517
【 大数据分析Hadoop + Spark 】10分钟搭建Hadoop（伪分布式 ）+ Spark（Local模式）环境
评论 3
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值