Hadoop安装流程&使用CDH版的Hadoop搭建HDFS环境

本文链接：https://blog.csdn.net/m0_45899013/article/details/107938184

hadoop：大数据处理框架
hdfs：分布式文件系统

hadoop的操作网址/参考文档：
http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-common/SingleCluster.html

软件下载

cdh-hadoop2.6.0
http://archive.cloudera.com/cdh5/cdh/5/
jdk1.8
https://www.oracle.com/java/technologies/javase-downloads.html

实验环境

系统：centOS7
用户名：hadoop
密码：hadoop
homename:hadoop000

前置工作：新建hadoop用户并修改主机名

注意：此处的#代表在root用户下操作，在其他用户下显示的为$。

[root@192 ~]# useradd hadoop
[root@192 ~]# passwd hadoop
更改用户 hadoop 的密码 。
新的 密码：
无效的密码： 密码少于 8 个字符
重新输入新的 密码：
passwd：所有的身份验证令牌已经成功更新。
[root@192 ~]# hostnamectl set-hostname hadoop000
[root@192 ~]# hostname
hadoop000

# 更改文件授权，在第100行将hadoop用户添加进去
[root@hadoop000 hadoop]# chmod 777 /etc/sudoers
[root@hadoop000 hadoop]# vim /etc/sudoers

hadoop	ALL(ALL)	ALL

[root@192 ~]# reboot

切换到hadoop用户下，同时主机切换为hadoop000：
su - hadoop

只是切换用户到hadoop：
su hadoop

目录结构

software:软件安装包存放目录
data:存放所有的测试数据
app:所有软件的安装目录
source:存放所有源码的目录

[hadoop@hadoop000 ~]$ ll /mnt/hgfs/big-data-package/
总用量 494843
-rwxrwxrwx. 1 root root 311585484 8月  10 14:51 hadoop-2.6.0-cdh5.7.0.tar.gz
-rwxrwxrwx. 1 root root 195132576 5月  22 18:12 jdk-8u251-linux-x64.tar.gz

[hadoop@hadoop000 ~]$ mkdir software
[hadoop@hadoop000 ~]$ cp /mnt/hgfs/big-data-package/hadoop-2.6.0-cdh5.7.0.tar.gz ./software/
[hadoop@hadoop000 ~]$ cp /mnt/hgfs/big-data-package/jdk-8u251-linux-x64.tar.gz ./software/
[hadoop@hadoop000 ~]$ ll software/
总用量 494844
-rwxrwxr-x. 1 hadoop hadoop 311585484 8月  10 15:28 hadoop-2.6.0-cdh5.7.0.tar.gz
-rwxrwxr-x. 1 hadoop hadoop 195132576 8月  10 15:28 jdk-8u251-linux-x64.tar.gz

[hadoop@hadoop000 ~]$ mkdir app source data
[hadoop@hadoop000 ~]$ ll
总用量 0
drwxrwxr-x. 2 hadoop hadoop  6 8月  10 15:32 app
drwxrwxr-x. 2 hadoop hadoop  6 8月  10 15:32 data
drwxrwxr-x. 2 hadoop hadoop 76 8月  10 15:28 software
drwxrwxr-x. 2 hadoop hadoop  6 8月  10 15:32 source

安装jdk

[hadoop@hadoop000 ~]$ su root
密码：
[root@hadoop000 hadoop]# rpm -e --nodeps $(rpm -qa| grep jdk)
[root@hadoop000 hadoop]# rpm -qa| grep jdk

[root@hadoop000 software]# cd /usr/local/jdk1.8.0_251/
[root@hadoop000 jdk1.8.0_251]# pwd
/usr/local/jdk1.8.0_251
[root@hadoop000 jdk1.8.0_251]# vim /etc/profile
export java_home=/usr/local/jdk1.8.0_251
export PATH=$PATH:$java_home/bin

[root@hadoop000 jdk1.8.0_251]# source /etc/profile
[root@hadoop000 jdk1.8.0_251]# echo $java_home
/usr/local/jdk1.8.0_251
[root@hadoop000 jdk1.8.0_251]# java -version

搭建步骤

1.Linux主机配置

[root@hadoop000 ~]# vim /etc/sysconfig/network

# Created by anaconda
HOSTNAME=hadoop000

2.添加IP和hostname的映射关系

[root@hadoop000 ~]# vim /etc/hosts
192.168.1.13    hadoop000
192.168.1.13    localhost

3.ssh免密码登录

[root@hadoop000 ~]# su - hadoop
上一次登录：二 8月 11 14:35:59 CST 2020从 192.168.1.13pts/3 上

[hadoop@hadoop000 ~]$ ssh-copy-id -i ./.ssh/id_rsa.pub hadoop@hadoop000
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "./.ssh/id_rsa.pub"
The authenticity of host 'hadoop000 (192.168.1.13)' can't be established.
ECDSA key fingerprint is SHA256:HvOGhhTQosLMRjX8aroVfwSARoJF2CuWN6orFw+8alQ.
ECDSA key fingerprint is MD5:5f:ea:47:c6:2d:6a:80:37:9c:c6:f4:3e:d3:01:5b:a7.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
hadoop@hadoop000's password:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'hadoop@hadoop000'"
and check to make sure that only the key(s) you wanted were added.

[hadoop@hadoop000 ~]$ ll ./.ssh
总用量 12
-rw-------. 1 hadoop hadoop  398 8月  11 14:44 authorized_keys
-rw-------. 1 hadoop hadoop 1679 8月  11 14:43 id_rsa
-rw-r--r--. 1 hadoop hadoop  398 8月  11 14:43 id_rsa.pub

[hadoop@hadoop000 ~]$ ssh 'hadoop@hadoop000'
The authenticity of host 'hadoop000 (192.168.1.13)' can't be established.
ECDSA key fingerprint is SHA256:HvOGhhTQosLMRjX8aroVfwSARoJF2CuWN6orFw+8alQ.
ECDSA key fingerprint is MD5:5f:ea:47:c6:2d:6a:80:37:9c:c6:f4:3e:d3:01:5b:a7.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'hadoop000,192.168.1.13' (ECDSA) to the list of known hosts.
Last login: Tue Aug 11 14:42:25 2020 from 192.168.1.13

[hadoop@hadoop000 ~]$ exit
登出
Connection to hadoop000 closed.

4.HDFS文件参数设置
（1）解压hadoop

[hadoop@hadoop000 software]$ tar zxvf hadoop-2.6.0-cdh5.7.0.tar.gz -C ../app

（2）hadoop目录结构介绍
（3）配置文件
etc/hadoop/hadoop-env.sh

# 添加jdk的目录
export JAVA_HOME=/usr/local/jdk1.8.0_251
# 添加hadoop的目录
export HADOOP_PREFIX=/home/hadoop/app/hadoop-2.6.0-cdh5.7.0

etc/hadoop/core-site.xml

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/home/hadoop/app/tmp</value>
    </property>
</configuration>

etc/hadoop/hdfs-site.xml

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
</configuration>

（4）系统环境变量

[hadoop@hadoop000 ~]$ su root
密码：

[root@hadoop000 ~]$ vim /etc/profile
export java_home=/usr/local/jdk1.8.0_251
export hadoop_home=/home/hadoop/app/hadoop-2.6.0-cdh5.7.0
export PATH=$PATH:$java_home/bin:$hadoop_home/bin:$hadoop_home/sbin

[root@hadoop000 ~]$ source /etc/profile

[root@hadoop000 hadoop]# su - hadoop
上一次登录：二 8月 11 15:03:39 CST 2020从 192.168.1.13pts/6 上

[hadoop@hadoop000 ~]$ echo $hadoop_home
/home/hadoop/app/hadoop-2.6.0-cdh5.7.0

HDFS格式化及启停

格式化nomanode

[hadoop@hadoop000 ~]$ hdfs namenode -format
20/08/11 15:52:52 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = hadoop000/192.168.1.13
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 2.6.0-cdh5.7.0
STARTUP_MSG:   classpath = /home/hadoop/app/hadoop-2.6.0-
# 省略部分输出

20/08/11 15:52:56 INFO common.Storage: Storage directory /home/hadoop/app/tmp/dfs/name has been successfully formatted.
20/08/11 15:52:57 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
20/08/11 15:52:57 INFO util.ExitUtil: Exiting with status 0
20/08/11 15:52:57 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at hadoop000/192.168.1.13
************************************************************/
# 查看hadoop工作目录
[hadoop@hadoop000 ~]$ ll  /home/hadoop/app/tmp
总用量 0
drwxrwxr-x. 3 hadoop hadoop 18 8月  11 15:52 dfs
[hadoop@hadoop000 ~]$ ll  /home/hadoop/app/tmp/dfs/name/current/
总用量 16
-rw-rw-r--. 1 hadoop hadoop 353 8月  11 15:52 fsimage_0000000000000000000
-rw-rw-r--. 1 hadoop hadoop  62 8月  11 15:52 fsimage_0000000000000000000.md5
-rw-rw-r--. 1 hadoop hadoop   2 8月  11 15:52 seen_txid
-rw-rw-r--. 1 hadoop hadoop 205 8月  11 15:52 VERSION