HDFS

jdk:jdk-8u131-linux-x64.tar.gz
hadoop:hadoop-2.8.0.tar.gz
centos:CentOS-7-x86_64-Minimal-1611.iso

一.关闭防火墙和强制安全策略

systemctl stop firewalld.service
systemctl disable firewalld.service

修改配置vi /etc/selinux/config

 SELINUX=disabled

二.修改主机名

修改配置 vi /etc/sysconfig/network

HOSTNAME=centos201

三.配置三台主机名映射

修改配置 vi /etc/hosts

192.168.193.201 centos201
192.168.193.202 centos202
192.168.193.203 centos203

重启系统 生效

四.添加用户

groupadd -g 601 wuyang
useradd -g wuyang -u 601 wuyang
(echo '123456';sleep 1;echo '123456')| passwd wuyang

五.免密钥登录

su wuyang
ssh-keygen -t rsa
ssh-copy-id centos201
ssh-copy-id centos202
ssh-copy-id centos203

六.安装JDK

A:解压/usr/java
B:vi /etc/profile
C:添加

    export JAVA_HOME=/usr/java/jdk1.8.0_131
    export JRE_HOME=/usr/java/jdk1.8.0_131/jre
    export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
    export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH

D:source /etc/profile
E:alternatives --install /usr/bin/java java /usr/java/jdk1.8.0_131/bin/java 300
F:alternatives --config java
G:alternatives --install /usr/bin/javac javac /usr/java/jdk1.8.0_131/bin/java 300
H:alternatives --config javac

七.安装HDFS

A:解压到/usr/hadoop
B:把Hadoop的安装路径添加到”/etc/profile”中

    vi /etc/profile

    export HADOOP_HOME=/usr/hadoop  
    export PATH=$PATH:$HADOOP_HOME/bin

C:source /etc/profile
D:配置hadoop-env.sh，并确认生效

    vi /usr/hadoop/etc/hadoop/hadoop-env.sh

    export JAVA_HOME=/usr/java/jdk1.8.0_131
    export HADOOP_CONF_DIR=/usr/hadoop/etc/hadoop/

    source /usr/hadoop/etc/hadoop/hadoop-env.sh 

    hadoop version

E:在/usr/hadoop目录下创建子目录

    [root@centos201 hadoop]# mkdir tmp
    [root@centos201 hadoop]# mkdir hdfs
    [root@centos201 hadoop]# cd hdfs
    [root@centos201 hdfs]# mkdir name
    [root@centos201 hdfs]# mkdir tmp
    [root@centos201 hdfs]# mkdir data

F:配置core-site.xml文件 在/usr/hadoop/etc/hadoop”目录下。修改Hadoop核心配置文件core-site.xml，这里配置的是HDFS master（即namenode）的地址和端口号。

vi /usr/hadoop/etc/hadoop/core-site.xml 

<configuration>
<property>
    <name>hadoop.tmp.dir</name>
    <value>/usr/hadoop/tmp</value>
    <final>true</final>
<!--拢篓卤赂注拢潞??? /usr/hadoop 目录?陆篓b tmp ?录镁录校漏 -->
    <description>A base for other temporary directories.</description>
</property>
<property>
    <name>fs.default.name</name>
    <value>hdfs://centos201:9000</value>   
<!-- hdfs://Master.Hadoop:22-->
        <final>true</final>
</property>
<property>
     <name>io.file.buffer.size</name>
     <value>131072</value>
</property>  
</configuration>

G:配置hdfs-site.xml文件

vi /usr/hadoop/etc/hadoop/hdfs-site.xml

<property>   
        <name>dfs.replication</name>   
        <value>3</value>   
    </property>   
    <property>   
        <name>dfs.name.dir</name>   
        <value>/usr/hadoop/hdfs/name</value>   
    </property>   
    <property>   
        <name>dfs.data.dir</name>   
        <value>/usr/hadoop/hdfs/data</value>   
    </property>   
    <property>    
         <name>dfs.namenode.secondary.http-address</name>    
         <value>centos201:9001</value>    
    </property>    
    <property>    
         <name>dfs.webhdfs.enabled</name>    
         <value>true</value>    
    </property>    
    <property>    
         <name>dfs.permissions</name>    
         <value>false</value>    
    </property>

H:配置mapred-site.xml文件

cp /usr/hadoop/etc/hadoop/mapred-site.xml.template /usr/hadoop/etc/hadoop/mapred-site.xml
vi /usr/hadoop/etc/hadoop/mapred-site.xml

<configuration>  
    <property>    
          <name>mapreduce.framework.name</name>    
          <value>yarn</value>    
    </property>     
</configuration>

I:配置yarn-site.xml文件

vi /usr/hadoop/etc/hadoop/yarn-site.xml

<property>    
  <name>yarn.resourcemanager.address</name>    
  <value>centos201:18040</value>    
</property>    
<property>    
  <name>yarn.resourcemanager.scheduler.address</name>    
  <value>centos201:18030</value>    
</property>    
<property>    
  <name>yarn.resourcemanager.webapp.address</name>    
  <value>centos201:18088</value>    
</property>    
<property>    
  <name>yarn.resourcemanager.resource-tracker.address</name>    
  <value>centos201:18025</value>    
</property>    
<property>    
  <name>yarn.resourcemanager.admin.address</name>    
  <value>centos201:18141</value>    
</property>    
<property>    
  <name>yarn.nodemanager.aux-services</name>    
  <value>mapreduce_shuffle</value>    
</property>    
<property>    
  <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>    
  <value>org.apache.hadoop.mapred.ShuffleHandler</value>    
</property>

J:配置slaves文件

vi /usr/hadoop/etc/hadoop/slaves

centos201
centos202
centos203

K:给权限

chown -R wuyang:wuyang /usr/hadoop/
chown -R wuyang:wuyang /usr/java/
sudo chmod -R a+w /usr/hadoop/

L:切换用户

su wuyang

M:初始化

/usr/hadoop/bin/hdfs namenode -format

N:启动

/usr/hadoop/sbin/start-all.sh

O:测试

http://192.168.193.201:18088

重点内容

1. 资源设定

2.参数配置