Centos7安装Hadoop

修改 Centos7为字符界面

Centos7取消了之前的 inittab文件的作用，改用命令实现，
- systemctl get-default ： 获得默认启动方式
- systemctl set-default multi-user.target ： 设置字符界面启动方式(level3)
- systemctl set-default graphical.target ： 设置图形化启动方式(level5)

修改主机名

• Centos7 中 ：hostnamectl set-hostname xxx
• Centos6 修改 network文件即可 ：vim /etc/sysconfig/network

修改hosts文件保证三台机器可以ping通

hadoop-master 192.168.178.1
slave1 192.168.178.1
slave2 192.168.178.1

安装SSH

检查是否已经安装ssh

• rpm –qa | grep openssh
• rpm –qa | grep rsync

如果没有安装 ssh 和 rsync，用下面的命令安装

• yum install ssh
• yum install rsync
• service sshd restart

配置无密码登陆

• 机器上生成密码对

  ssh-keygen –t rsa

• 把 id_rsa.pub 追加到授权的 key 里面 去

  cat id_rsa.pub >> authorized_keys

• 修改文件”authorized_keys”权限

  chmod 600 authorized_keys

• 修改 SSH 配置文件 /etc/ssh/sshd_config (所有机器都要配)

  RSAAuthentication yes
  PubkeyAuthentication yes
  AuthorizedKeysFile .ssh/authorized_keys

• 重启ssh服务

  service sshd restart

• 将公钥复制到所有slave机器上

  scp id_rsa.pub hadoop@slave1:/home/hadoop/

• 追加 authorized_keys

  cp id_rsa.pub .ssh
  cat id_rsa.pub >> .ssh/authorized_keys
  chmod 600 .ssh/authorized_keys
  service sshd restart

• 验证是否成功 ssh slave1 如果可以直接登陆则表示成功

  [hadoop@localhost .ssh]$ ssh slave1
  Last login: Fri Jan 5 21:11:48 2018 from hadoop-master

Java 环境安装

• 卸载已安装的java

  rpm -qa | grep java
  rpm -e –nodeps java-1.8.0-openjdk-1.8.0.102-4.b14.el7.x86_64
  rpm -e –nodeps javapackages-tools-3.4.1-11.el7.noarch
  rpm -e –nodeps java-1.8.0-openjdk-headless-1.8.0.102-4.b14.el7.x86_64
  rpm -e –nodeps tzdata-java-2016g-2.el7.noarch
  rpm -e –nodeps java-1.7.0-openjdk-1.7.0.111-2.6.7.8.el7.x86_64
  rpm -e –nodeps java-1.7.0-openjdk-headless-1.7.0.111-2.6.7.8.el7.x86_64

• 安装java

  解压
  tar -zxvf jdk-8u131-linux-x64.tar.gz
  将目录名改为 jdk1.8
  mv jdk1.8.0_131/ jdk1.8
  编辑环境变量
  vim /etc/profile
  
  
  // 添加以下内容
  # set java environment
  export JAVA_HOME=/usr/jdk1.8
  export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib:$JAVA_HOME/jre/lib export PATH=$PATH:$JAVA_HOME/bin:$JAVA_HOME/jre/bin

  
  
  使配置文件生效
  source /etc/profile

安装Hadoop

• 解压 hadoop-2.7.0.tar.gz，并重命名

  tar zxvf hadoop-2.7.5.tar.gz
  mv hadoop-2.7.0 hadoop

• 在/usr/hadoop下面创建tmp文件夹

  mkdir tmp

• 配置 /etc/profile 文件

  vim /etc/profile
  
  
  添加以下内容
  export HADOOP_INSTALL=/usr/hadoop
  export PATH=${HADOOP_INSTALL}/bin:${HADOOP_INSTALL}/sbin${PATH} export HADOOP_MAPRED_HOME=${HADOOP_INSTALL}
  export HADOOP_COMMON_HOME=${HADOOP_INSTALL} export HADOOP_HDFS_HOME=${HADOOP_INSTALL}
  export YARN_HOME=${HADOOP_INSTALLL} export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_INSTALL}/lib/natvie
  export HADOOP_OPTS=”-Djava.library.path=${HADOOP_INSTALL}/lib:${HADOOP_INSTALL}/lib/native”
  
  
  使生效
  source /etc/profile

配置hadoop

• 配置 hadoop-env.sh,配置java环境变量

  export JAVA_HOME=/usr/java/jdk1.8

• 配置core-site.xml文件

// 内容如下
<configuration>
<property>
  <name>hadoop.tmp.dir</name>
  <value>/usr/hadoop/tmp</value>
  <description>A base for other temporary directories.</description>
  </property>
  <property>
  <name>fs.default.name</name>
  <value>hdfs://hadoop-master:9000</value>
 </property>
</configuration>

• 配置mapred-site.xml文件

<configuration>
    <property>
           <name>mapreduce.framework.name</name>
           <value>yarn</value>
           <final>true</final>
    </property>

    <property>
        <name>mapreduce.jobtracker.http.address</name>
        <value>hadoop-master:50030</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.address</name>
        <value>hadoop-master:10020</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>hadoop-master:19888</value>
    </property>
        <property>
                <name>mapred.job.tracker</name>
                <value>http://hadoop-master:9001</value>
        </property>
</configuration>

• 配置yarn-site.xml文件

<configuration>
        <property>
                <name>yarn.resourcemanager.hostname</name>
                <value>hadoop-master</value>
        </property>

    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.resourcemanager.address</name>
        <value>hadoop-master:8032</value>
    </property>
    <property>
        <name>yarn.resourcemanager.scheduler.address</name>
        <value>hadoop-master:8030</value>
    </property>
    <property>
        <name>yarn.resourcemanager.resource-tracker.address</name>
        <value>hadoop-master:8031</value>
    </property>
    <property>
        <name>yarn.resourcemanager.admin.address</name>
        <value>hadoop-master:8033</value>
    </property>
    <property>
        <name>yarn.resourcemanager.webapp.address</name>
        <value>hadoop-master:8088</value>
    </property>
</configuration>

• 将hadoop传入slave中

  scp -r /usr/hadoop root@slave1:/usr/

• 修改Master主机上的slaves文件

vim /usr/hadoop/etc/hadoop/slaves

将内容改为：

slave1
slave2

启动hadoop

• 关闭防火墙

  systemctl stop firewalld.service

  关闭防火墙开机自启
  systemctl disable firewalld.service

• 格式化

  hadoop namenode -format

• 启动hadoop

  /usr/hadoop/sbin/start-all.sh

• 验证，jps， 若出现下列信息表示安装成功

主节点：
[hadoop@hadoop-master usr]$ jps
13138 Jps
12563 SecondaryNameNode
12708 ResourceManager
12375 NameNode

从节点：
[root@slave1 ~]# jps
12786 Jps
12205 DataNode
12317 NodeManager