Hadoop的伪分布式部署

Hadoop介绍
1.定义
广义：以Apache Hadoop软件为主的生态圈
狭义：Apache Hadoop软件

2.Hadoop软件的组件

• 存储：hdfs 分布式文件系统，需要部署
• 计算：mapreduce 分布式计算，在企业中很少用，不需要部署
• 资源（memory内存、CPU处理器）和作业调度：yarn，需要部署

安装条件及软件说明

• Linux环境配置ssh服务
• Hadoop版本：hadoop-2.6.0-cdh5.7.0
• JDK版本：jdk-8u45-linux-x64

Hadoop的伪分布式部署
1.下载Hadoop-2.6.0-cdh5.7.0
wget http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.7.0.tar.gz
或 rz上传已下载好的软件

2.创建专门维护的用户 hadoop

[root@hadoop001 ~]# useradd hadoop
[root@hadoop001 ~]# su - hadoop
[hadoop@hadoop001 ~]$ pwd
/home/hadoop

3.JDK部署
1）jdk目录务必在/usr/java/下，若没有该目录则创建一个
因为cdh的一些脚本默认在该路径下，如果jdk不在这个路径，cdh无法启动
2）用户用户组修正
3）要配置全局环境变量，因为各个大数据组件都会用到Java。
环境变量配置的置前原则：新来的放在path开头

4.创建app，data，log，software

[hadoop@hadoop001 ~]$ mkdir app software log data
[hadoop@hadoop001 ~]$ ll
total 16
drwxrwxr-x 2 hadoop hadoop 4096 Jun 18 07:10 app
drwxrwxr-x 2 hadoop hadoop 4096 Jun 18 07:10 data
drwxrwxr-x 2 hadoop hadoop 4096 Jun 18 07:10 log
drwxrwxr-x 2 hadoop hadoop 4096 Jun 18 07:10 software

5.移动安装包到/home/hadoop/software/目录下

[root@hadoop001 ~]# mv hadoop-2.6.0-cdh5.7.0.tar.gz /home/hadoop/software/
[root@hadoop001 ~]# ll /home/hadoop/software/
total 304284
-rw-r--r-- 1 root root 311585484 Jul  3  2019 hadoop-2.6.0-cdh5.7.0.tar.gz

6.变更所属用户

[root@hadoop001 ~]# chown hadoop:hadoop /home/hadoop/software/*
[root@hadoop001 ~]# ll /home/hadoop/software/
total 304284
-rw-r--r-- 1 hadoop hadoop 311585484 Jul  3  2019 hadoop-2.6.0-cdh5.7.0.tar.gz

7.进入software并解压，做软连接放到app目录下

[hadoop@hadoop001 ~]$ cd software
[hadoop@hadoop001 software]$ ll
total 304284
-rw-r--r-- 1 hadoop hadoop 311585484 Jul  3  2019 hadoop-2.6.0-cdh5.7.0.tar.gz
[hadoop@hadoop001 software]$ tar xzvf hadoop-2.6.0-cdh5.7.0.tar.gz
[hadoop@hadoop001 software]$ ll
total 304288
drwxr-xr-x 14 hadoop hadoop      4096 Mar 24  2016 hadoop-2.6.0-cdh5.7.0
-rw-r--r--  1 hadoop hadoop 311585484 Jul  3  2019 hadoop-2.6.0-cdh5.7.0.tar.gz
[hadoop@hadoop001 software]$ ln -s /home/hadoop/software/hadoop-2.6.0-cdh5.7.0 /home/hadoop/app/hadoop
[hadoop@hadoop001 software]$ cd ../app
[hadoop@hadoop001 app]$ ll
total 0
lrwxrwxrwx 1 hadoop hadoop 43 Jun 18 07:18 hadoop -> /home/hadoop/software/hadoop-2.6.0-cdh5.7.0

如果原始文件和快捷键的文件夹在同一个目录，在做软连接时用相对路径即可，如果不在同一个目录，就要用绝对路径，否则软连接会不生效。

8.修改配置文件

• 修改改etc/hadoop/hadoop-env.sh

[hadoop@hadoop001 hadoop]$ vi hadoop-env.sh
# set to the root of your Java installation
export JAVA_HOME=/usr/java/jdk1.8.0_45

• 修改etc/hadoop/core-site.xml和etc/hadoop/hdfs-site.xml

[hadoop@hadoop001 hadoop]$ pwd
/home/hadoop/app/hadoop/etc/hadoop
[hadoop@hadoop001 hadoop]$ vi core-site.xml

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>
</configuration>

[hadoop@hadoop001 hadoop]$ vi hdfs-site.xml

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
</configuration>

在修改core-site.xml时，不要随意删除xml文件的第一句话和第二句话

9.配置ssh免密登陆

[hadoop@hadoop001 ~]$ rm -rf .ssh
[hadoop@hadoop001 ~]$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa): 
Created directory '/home/hadoop/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
44:3c:e9:e7:09:51:c1:34:47:e8:b7:12:3b:c2:e8:04 hadoop@hadoop001
The key's randomart image is:
+--[ RSA 2048]----+
|       ..==+o    |
|       .= oo     |
|       ..+       |
|    E  .o + .    |
|     . oS+ = .   |
|      o o * .    |
|     o   . o     |
|      .          |
|                 |
+-----------------+
[hadoop@hadoop001 ~]$ cd .ssh
[hadoop@hadoop001 .ssh]$ cat id_rsa.pub >> authorized_keys
[hadoop@hadoop001 .ssh]$ ll
total 16
-rw-rw-r-- 1 hadoop hadoop  398 Jun 18 08:06 authorized_keys
-rw------- 1 hadoop hadoop 1675 Jun 18 08:04 id_rsa
-rw-r--r-- 1 hadoop hadoop  398 Jun 18 08:04 id_rsa.pub
-rw-r--r-- 1 hadoop hadoop  391 Jun 18 08:07 known_hosts
[hadoop@hadoop001 .ssh]$ chmod 600 authorized_keys
[hadoop@hadoop001 .ssh]$ ssh localhost        ssh登录localhost机器
[hadoop@hadoop001 ~]$       
[hadoop@hadoop001 .ssh]$ ssh localhost date   
Sun Jun 30 21:36:58 CST 2019
#ssh登录localhost机器执行date命令（一瞬间） 返回结果 不会做机器切换

10.格式化namenode

[hadoop@hadoop001 hadoop]$ pwd
/home/hadoop/app/hadoop
[hadoop@hadoop001 hadoop] bin/hdfs namenode -format

11.启动

[hadoop@hadoop001 hadoop]$ sbin/start-dfs.sh 
19/07/01 23:39:35 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [localhost]
localhost: starting namenode, logging to /home/hadoop/software/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-namenode-hadoop001.out
localhost: starting datanode, logging to /home/hadoop/software/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-datanode-hadoop001.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /home/hadoop/software/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-secondarynamenode-hadoop001.out
19/07/01 23:39:50 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

12.jps验证是否启动成功

[hadoop@hadoop001 sbin]$ jps
4887 DataNode
5208 Jps
4795 NameNode
5100 SecondaryNameNode