Hadoop Single Node Cluster安装

Hadoop Single Node Cluster安装

一、软件准备

1. CentOS 虚拟机（这里我用的是CentOS 6.8 X64 + VMware）。

2. jdk-8u191-linux-x64.tar.gz

3. hadoop-2.8.5.tar.gz

二、配置要求

hostname	IP地址	OS	用户	密码
hadoop	10.13.9.9	CentOS 6.8	hdusr	Hdusr123!

三、安装步骤

1.配置hostname

a.编辑network文件修改hostname行（重启生效）

$ vim /etc/sysconfig/network

b. 设置当前的hostname(立即生效）

$ hostname hadoop

c. 编辑hosts文件，给127.0.0.1添加hostname

$ vim /etc/hosts

2.配置IP地址

# vim /etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE="eth0"

BOOTPROTO="static"

ONBOOT="yes"

TYPE="Ethernet"

IPADDR="10.13.9.9"

PREFIX="24"

GATEWAY="10.13.9.2"

DNS1="10.13.9.2"

3.关闭防火墙

$ sudo chkconfig iptables off

$ sudo service iptables stop

4.创建hdusr用户

创建用户

# useradd hdusr

# passwd hdusr

修改文件权限

# chmod u+w /etc/sudoers

编辑/etc/sudoers文件。

找到这一行：”root ALL=(ALL) ALL”在起下面添加”xxx ALL=(ALL) ALL”(这里的xxx是你的用户名)，然后保存。

# vim /etc/sudoers

撤销文件的写权限# chmod u-w /etc/sudoers

5.设定ssh免密登录

a.生成密钥

$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa

$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

b.配置sshd

$ sudo vim /etc/ssh/sshd_config

PasswordAuthentication yes

HostKey /etc/ssh/ssh_host_rsa_key

RSAAuthentication yes

PubkeyAuthentication yes

AuthorizedKeysFile .ssh/authorized_keys

$ sudo /etc/init.d/sshd restart

c.设置权限

$ chown -R hdusr:hdusr /home/hdusr/

$ chmod 700 /home/hdusr/

$ chmod 700 /home/hdusr/.ssh/

$ chmod 644 /home/hdusr/.ssh/authorized_keys

$ chmod 600 /home/hdusr/.ssh/id_dsa

c.测试ssh免密登录

$ ssh 127.0.0.1

Last login: Tue Oct 30 01:28:59 2018 from localhost

$ exit

logout

Connection to 127.0.0.1 closed.

6.安装并配置java环境变量

a.解压到安装目录

$ sudo mkdir /usr/local/java

$ sudo chown -R hdusr:hdusr java/

$ tar -zxvf /home/software/jdk-8u192-linux-x64.tar.gz -C /usr/local/java/

b.配置环境变量

$ vim ~/.bashrc

export JAVA_HOME=/usr/local/java/jdk1.8.0_192/

export PATH=${JAVA_HOME}/bin:$PATH

$ source ~/.bashrc

$ echo $PATH

$ java -version

java version "1.8.0_192"

Java(TM) SE Runtime Environment (build 1.8.0_192-b12)

Java HotSpot(TM) 64-Bit Server VM (build 25.192-b12, mixed mode)

$ javac -version

javac 1.8.0_192

7.安装并配置hadoop环境变量

a.解压到安装目录

$ sudo mkdir hadoop

$ sudo chown -R hdusr:hdusr hadoop/

$ tar -zxvf /home/software/hadoop-2.8.5.tar.gz -C /usr/local/hadoop/

b.配置环境变量

$ vim ~/.bashrc

export HADOOP_HOME=/usr/local/hadoop/hadoop-2.8.5

export PATH=$PATH:$HADOOP_HOME/bin

export PATH=$PATH:$HADOOP_HOME/sbin

export HADOOP_MAPRED_HOME=$HADOOP_HOME

export HADOOP_COMMON_HOME=$HADOOP_HOME

export HADOOP_HDFS_HOME=$HADOOP_HOME

export YARN_HOME=$HADOOP_HOME

export HADOOP_COMMON_HOME=$HADOOP_HOME

export HADOOP_HDFS_HOME=$HADOOP_HOME

export YARN_HOME=$HADOOP_HOME

export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native

export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"

export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native:$JAVA_LIBRARY_PATH

$ source ~/.bashrc

$ hadoop version

8.hadoop配置

Step1 修改hadoop-env.sh

$ vim hadoop/hadoop-2.8.5/etc/hadoop/hadoop-env.sh

export JAVA_HOME=/usr/local/java/jdk1.8.0_192/

Step2 修改core-site.xml

$ vim hadoop/hadoop-2.8.5/etc/hadoop/core-site.xml

<configuration>

        <property>

                <name>fs.default.name</name>

                <value>hdfs://localhost:9000</value>

        </property>

</configuration>

Step3 修改yarn-site.xml

$ vim hadoop/hadoop-2.8.5/etc/hadoop/yarn-site.xml

<configuration>

        <property>

                <name>yarn.nodemanager.aux-services</name>

                <value>mapreduce_shuffle</value>

        </property>

        <property>

                <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>

                <value>org.apache.hadoop.mapred.ShuffleHandler</value>

        </property>

        <!-- Site specific YARN configuration properties -->

</configuration>

Step4 修改mapred-site.xml

$ cp hadoop/hadoop-2.8.5/etc/hadoop/mapred-site.xml.template hadoop/hadoop-2.8.5/etc/hadoop/mapred-site.xml

$ vim hadoop/hadoop-2.8.5/etc/hadoop/mapred-site.xml

<configuration>

        <property>

                <name>mapreduce.framework.name</name>

                <value>yarn</value>

        </property>

</configuration>

Step5 修改hdfs-site.xml

$ vim hadoop/hadoop-2.8.5/etc/hadoop/hdfs-site.xml

<configuration>

        <property>

                <name>dfs.replication</name>

                <value>3</value>

        </property>

        <property>

                <name>dfs.namenode.name.dir</name>

                <value>file:/usr/local/hadoop/hadoop-2.8.5/hddata/hdfs/namenode</value>

        </property>

        <property>

                <name>dfs.datanode.data.dir</name>

                <value>file:/usr/local/hadoop/hadoop-2.8.5/hddata/hdfs/datanode</value>

        </property>

</configuration>

9.建立与格式化HDFS目录

$ mkdir -p hadoop/hadoop-2.8.5/hddata/hdfs/namenode

$ mkdir -p hadoop/hadoop-2.8.5/hddata/hdfs/datanode

$ hadoop namenode -format

10.启动hadoop

有a和b两种启动方式

a.启动start-dfs.sh，再启动start-yarn.sh

$ start-dfs.sh

$ start-yarn.sh

b.启动全部

$ start-all.sh

查看目前所执行的行程

$ jps

7441 DataNode

7655 SecondaryNameNode

7929 NodeManager

7340 NameNode

7823 ResourceManager

9903 Jps

11.访问管理界面

Hadoop Resource­Manager Web界面地址

http://10.13.9.9:8088/

NameNode HDFS Web界面地址

http://10.13.9.9:50070/