Hadoop伪分布式部署

Hadoop三个组件

# HDFS 分布式文件存储系统(底层、生产)
# MapReduce 分布式计算框架(基于磁盘，速度慢，shuffle)
# Yarn 资源(内存+core)+作业(job)调度管理系统(生产)

企业一般不选择原生Apace Hadoop部署，而是选用CDH,Ambari,Hdp部署。
CDH：

cloudera公司 将Apache hadoop-2.6.0源代码
修复bug,新功能，编译为自己的版本cdh5.7.0
Apache hadoop-2.6.0 --》hadoop-2.6.0-cdh5.7.0

部署步骤
1、安装配置JDK

[root@hadoop001 java]# ll
-rw-r--r-- 1 root root 173271626 Sep 18 21:09 jdk-8u45-linux-x64.gz
[root@hadoop001 java]# tar -zxvf jdk-8u45-linux-x64.gz
[root@hadoop001 java]# pwd
/usr/java
[root@hadoop001 java]#  vi /etc/profile
添加
	export JAVA_HOME=/usr/java/jdk1.8.0_45
	export PATH=$JAVA_HOME/bin:$PATH
[root@hadoop001 java]# source /etc/profile
[root@hadoop001 java]# java -version
	java version "1.8.0_45"
	Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
	Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)

2、添加sudo权限的无密码访问的hadoop用户

[root@hadoop001 ~]# useradd hadoop
[root@hadoop001 ~]# vi /etc/sudoers
 # 添加 hadoop      ALL=(ALL)       NOPASSWD:ALL
[root@hadoop001 ~]# su - hadoop
[hadoop@hadoop001 ~]$

3、下载hadoop-2.6.0-cdh5.7.0.tar.gz

[hadoop@hadoop001 ~]$ mkdir app
[hadoop@hadoop001 ~]$ cd app
[hadoop@hadoop001 app]$ wget http://archive-primary.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.7.0.tar.gz
[hadoop@hadoop001 app]$ tar -xzvf hadoop-2.6.0-cdh5.7.0.tar.gz
[hadoop@hadoop001 app]$cd hadoop-2.6.0-cdh5.7.0
[hadoop@hadoop001 hadoop-2.6.0-cdh5.7.0]$ 

4、配置环境变量

[hadoop@hadoop001 app]$ cd hadoop-2.6.0-cdh5.7.0
[hadoop@hadoop001 hadoop-2.6.0-cdh5.7.0]$ cd etc/hadoop
[hadoop@hadoop001 hadoop]$ vi hadoop-env.sh
	export JAVA_HOME=/usr/java/jdk1.8.0_45
	export HADOOP_PREFIX=/home/hadoop/app/hadoop-2.6.0-cdh5.7.0

5、配置文件

[hadoop@hadoop001 hadoop]$  vi core-site.xml
	--添加
	<configuration>
     <property>
        <name>fs.defaultFS</name>
        <value>hdfs://hadoop002:9000</value>
     </property>
	</configuration>
[hadoop@hadoop001 hadoop]$ vi hdfs-site.xml
	--添加
	<configuration>
      <property>
        <name>dfs.replication</name>
        <value>1</value>
      </property>
	</configuration>

6、设置ssh免密码登录

[hadoop@hadoop001 ~]$ rm -rf .ssh
[hadoop@hadoop001 ~]$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
Generating public/private dsa key pair.
Created directory '/home/hadoop/.ssh'.
Your identification has been saved in /home/hadoop/.ssh/id_dsa.
Your public key has been saved in /home/hadoop/.ssh/id_dsa.pub.
The key fingerprint is:
a3:c7:ba:e9:2e:77:ff:6f:50:bd:bc:f7:1b:1d:a6:e1 hadoop@hadoop002
The key's randomart image is:
+--[ DSA 1024]----+
|                 |
|                 |
|               . |
|              . .|
|        S    o.o.|
|       o .  o +oo|
|      . o    E .o|
|    . .+.     ..o|
|     =*o ....o..=|
+-----------------+
[hadoop@hadoop001 ~]$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
[hadoop@hadoop001 ~]$ cd .ssh
[hadoop@hadoop001 .ssh]$ ll
-rw------- 1 hadoop hadoop  606 Sep 27 10:28 authorized_keys
-rw------- 1 hadoop hadoop  668 Sep 27 10:28 id_dsa
-rw-r--r-- 1 hadoop hadoop  606 Sep 27 10:28 id_dsa.pub
[hadoop@hadoop001 .ssh]$ chmod 600 authorized_keys
[hadoop@hadoop001 .ssh]$ ssh hadoop001

7、配置hadoop用户环境变量

[hadoop@hadoop001 ~]$ vi .bash_profile
--添加
export MVN_HOME=/home/hadoop/app/apache-maven-3.3.9
export PROTOC_HOME=/home/hadoop/app/protobuf
export FINDBUGS_HOME=/home/hadoop/app/findbugs-1.3.9
export JAVA_HOME=/usr/java/jdk1.8.0_45
export HADOOP_PREFIX=/home/hadoop/app/hadoop-2.6.0-cdh5.7.0

8、启动HDFS

[hadoop@hadoop001 ~]$ which hdfs
~/app/hadoop-2.6.0-cdh5.7.0/bin/hdfs
[hadoop@hadoop001 ~]$ hdfs namenode -format
[hadoop@hadoop001 hadoop-2.6.0-cdh5.7.0]$ pwd
/home/hadoop/app/hadoop-2.6.0-cdh5.7.0
[hadoop@hadoop001 hadoop-2.6.0-cdh5.7.0]$ sbin/start-dfs.sh
[hadoop@hadoop001 hadoop-2.6.0-cdh5.7.0]$  jps
27707 SecondaryNameNode
27820 Jps
27432 NameNode

9、启动Yarn

[hadoop@hadoop001 hadoop]$ pwd
/home/hadoop/app/hadoop-2.6.0-cdh5.7.0/etc/hadoop
[hadoop@hadoop001 hadoop]$ cp mapred-site.xml.template mapred-site.xml
[hadoop@hadoop001 hadoop]$ vi mapred-site.xml
--添加
<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
</configuration>
[hadoop@hadoop001 hadoop]$  vi yarn-site.xml
--添加
<configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
</configuration>
[hadoop@hadoop001 hadoop]$ sbin/start-yarn.sh
[hadoop@hadoop001 hadoop]$ jps
27712 ResourceManager
27809 NodeManager
9267 Jps
26789 DataNode
26949 SecondaryNameNode
26666 NameNode
在浏览器打开http://ip:8088/，查看是否部署成功