spark从入门到放弃--hadoop集群安装

1、机器准备

192.168.10.149 hadoop-master
192.168.10.150 hadoop-salve1
192.168.10.151 hadoop-salve2

2、修改主机名

分别登陆到三台虚拟机 去修改主机名

vi /etc/hostname

hadoop-master

配置 host

vi /etc/hosts

192.168.10.149 hadoop-master
192.168.10.150 hadoop-salve1
192.168.10.151 hadoop-salve2

3、免密登陆

生成公钥和私钥

　　ssh-keygen -t rsa

导入公钥到认证文件

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys  
scp ~/.ssh/id_rsa.pub xxx@host:/home/xxx/id_rsa.pub  
cat ~/id_rsa.pub >> ~/.ssh/authorized_keys 

更改权限

chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys  

4.关闭SELINUX以及防火墙

防火墙会引起hadoop相关组件通讯的各种异常,需关闭防火墙。

用root用户权限登录进行操作：

1、关闭防火墙：

service iptables stop

验证：service iptables status

service iptables stop（临时关闭）

chkconfig iptables off（重启后生效）

2、关闭SELINUX

修改/etc/selinux/config文件

将SELINUX=enforcing改为SELINUX=disabled

5、修改hadoop配置

5.1、修改core.site.xml

<configuration>
    <property>
      <name>fs.default.name</name>
      <value>hdfs://hadoop-master:9000</value>
    </property>
</configuration>

5.2、hdfs-site.xml

<configuration>
    <property>
      <name>dfs.name.dir</name>
      <value>/mnt/data/namenode</value>
    </property>
    <property>
      <name>dfs.data.dir</name>
      <value>/mnt/data/datanode</value>
    </property>
    <property>
      <name>dfs.tmp.dir</name>
      <value>/mnt/data/tmp</value>
    </property>
    <property>
      <name>dfs.replication</name>
      <value>2</value>
    </property>
    <property>
      <name>dfs.permissions</name>
      <value>false</value>
    </property>
</configuration>

5.3、mapred-site.xml

<configuration>
    <property>
      <name>mapreduce.framework.name</name>
      <value>yarn</value>
    </property>
</configuration>

5.4、yarn-site.xml

<configuration>
    <property>
      <name>yarn.resourcemanager.hostname</name>
      <value>hadoop-master</value>
    </property>
    <property>
      <name>yarn.nodemanager.aux-services</name>
      <value>mapreduce_shuffle</value>
    </property>
</configuration>

其他节点配置同上

6、格式化namenode

hdfs namenode -format

7、启动hadoop集群

 sh start-dfs.sh 

8、master节点查看启动状态jps

23425 SecondaryNameNode
22743 Bootstrap
10056 QuorumPeerMain
7562 jar
23850 Jps
23580 ResourceManager
23245 NameNode

9、从服务器查看节点状态jps

25315 QuorumPeerMain
31763 Jps
31528 DataNode
31626 NodeManager