hadoop概况与hdfs部署

一、hadoop概况

广义的Hadoop是指以hadoop软件为主的生态圈
狭义d的hadoop是指hadoop软件
大数据软件官网：
hadoop: hadoop.apache.org
hive: hive.apache.org
spark: spark.apache.org
flink: flink.apache.org
hadoop软件版本：
1.x
2.x 生产上多用2.x版本
3.x 在更新迭代

hadoop2.x组件:
（1）hdfs: 存储大数据量的数据| 属于分布式文件系统 ，底层生产必不可少，生产上高级数据存储使用hive/hbase等
（2）mapreduce: 分布式计算系统 | 生产上不常用，因其开发难度高、计算慢(shuffle 磁盘)；生产上常用hive sql或者spark来进行分布式计算
（3）yarn: 资源(内存+core)+作业(job)调度管理系统 ，生产上必不可少
但是：
一般不选择部署apache官网上的提供的hadoop，企业一般选择CDH、Ambari、hdp部署
最常用的就是CDH，关于CDH的起源，以hadoop-2.6.0版本为例，是cloudera公司将Apache hadoop-2.6.0源代码，修复bug，增加了一些新功能新特性，编译为自己的版本hadoop-2.6.0-cdh5.7.0，成为企业版本获取收益！！
因此接下来的hadoop hdfs的部署是以cloudera公司编译的hadoop-2.6.0-cdh5.7.0版本部署，也是在cloudera公司官网进行学习。

二、hdfs部署

我们直击官网：http://archive-primary.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.7.0/
hadoop软件下载地址：http://archive-primary.cloudera.com/cdh5/cdh/5/
下载hadoop-2.6.0-cdh5.7.0 .tar.gz包

（1）添加sudo权限的无密码访问的hadoop用户
[root@hadoop001 ~]# useradd hadoop
[root@hadoop001 ~]# vi /etc/sudoers （修改hadoop用户权限）
添加：hadoop ALL=(ALL) NOPASSWD:ALL

[root@hadoop001 ~]# cat /etc/sudoers | grep hadoop （查看hadoop 用户权限）
[root@hadoop001 ~]# su - hadoop （切换hadoop用户）

（2）在hadoop 用户的家目录下创建环境文件：source源代码文件夹，software软件文件夹,app安装文件夹
[hadoop@hadoop001 ~]$ mkdir source software app
[hadoop@hadoop001 ~]$ cd app （切换到app文件夹）
[hadoop@hadoop001 app]$ wget http://archive-primary.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.7.0.tar.gz （下载hadoop-2.6.0-cdh5.7.0 .tar.gz包）
[hadoop@hadoop001 app]$ tar -xzvf hadoop-2.6.0-cdh5.7.0.tar.gz （解压）

（3）java环境部署：使用java jdk 1.7版本
我们直击官网：http://archive-primary.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.7.0/
学习的话先用单节点安装：Single Node Setup

官网要求：
Required Software
Required software for Linux include:
Java™ must be installed. Recommended Java versions are described at HadoopJavaVersions.
ssh must be installed and sshd must be running to use the Hadoop scripts that manage remote Hadoop daemons.

Hadoop可以在Linux或 Windows中部署，但是一般不会用Windows部署，另外需要安装一定版本的Java环境并且ssh必须要与hadoop连通
步骤之前已经在安装MySQL时候演示过了，这里不再详解：
先上传jdk-7u80-linux-x64.tar.gz包（rz命令），然后解压，修改用户和用户组，配置全局环境变量，生效全局环境变量文件
[hadoop@hadoop001 hadoop-2.6.0-cdh5.7.0]$ chown -R roor:root jdk1.7.0_80

[hadoop@hadoop001 hadoop-2.6.0-cdh5.7.0]$ vi /etc/proflie
添加以下内容：

[hadoop@hadoop001 hadoop-2.6.0-cdh5.7.0]$ source /etc/proflie
[hadoop@hadoop001 hadoop-2.6.0-cdh5.7.0]$ which java
[hadoop@hadoop001 hadoop-2.6.0-cdh5.7.0]$ java -version

（4）准备工作：修改变量文件

官网要求：
Prepare to Start the Hadoop Cluster
Unpack the downloaded Hadoop distribution. In the distribution, edit the file etc/hadoop/hadoop-env.sh to define some parameters as follows:
#set to the root of your Java installation
export JAVA_HOME=/usr/java/latest
Assuming your installation directory is /usr/local/hadoop
export HADOOP_PREFIX=/usr/local/hadoop
Try the following command:
$ bin/hadoop
This will display the usage documentation for the hadoop script.
Now you are ready to start your Hadoop cluster in one of the three supported modes:
Local (Standalone) Mode
Pseudo-Distributed Mode
Fully-Distributed Mode

[hadoop@hadoop001 hadoop-2.6.0-cdh5.7.0]$ cd etc/hadoop （按照要求进入etc/hadoop文件夹修改hadoop-env.sh文件）
[hadoop@hadoop001 hadoop]$ vi hadoop-env.sh
添加以下内容：
export JAVA_HOME=/usr/java/jdk1.7.0_80
export HADOOP_PREFIX=/home/hadoop/app/hadoop-2.6.0-cdh5.7.0

检查：
[hadoop@hadoop001 hadoop-2.6.0-cdh5.7.0]$ bin/hadoop

启动三种模式
Local (Standalone) Mode: 单机 没有进程 不用
Pseudo-Distributed Mode: 伪分布式 1台机器 每个服务只有一个进程 学习用
Fully-Distributed Mode: 分布式 有多个进程 生产用

（5）配置文件

官网要求更改core-site.xml和hdfs-site.xml文件：

[hadoop@hadoop001 hadoop-2.6.0-cdh5.7.0]$ cd etc/hadoop
[hadoop@hadoop001 hadoop]$ vi core-site.xml
替换成以下内容，并将localhost改写成当前机器名称：

[hadoop@hadoop001 hadoop]$ vi hdfs-site.xml
替换成以下内容：

（6）配置无密码ssh

官网要求：
Setup passphraseless ssh
Now check that you can ssh to the localhost without a passphrase:
$ ssh localhost
If you cannot ssh to localhost without a passphrase, execute the following commands:
$ ssh-keygen -t dsa -P ‘’ -f ~/.ssh/id_dsa
$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

执行之前，先删除.ssh隐藏文件（在hadoop家目录下）

[hadoop@hadoop001 ~]$ rm -rf .ssh （重要）
[hadoop@hadoop001 ~]$ ssh-keygen -t dsa -P ‘’ -f ~/.ssh/id_dsa
[hadoop@hadoop001 ~]$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

修改authorized_keys文件的权限：

[hadoop@hadoop001 .ssh]$ chmod 600 authorized_keys （重要）

检测是否能无密码远程连接：
[hadoop@hadoop001 .ssh]$ ssh hadoop001

（7）改个人环境变量：
[hadoop@hadoop001 ~]$ vi .bash_profile
添加以下内容：
export JAVA_HOME=/usr/java/jdk1.7.0_80
export HADOOP_PREFIX=/home/hadoop/app/hadoop-2.6.0-cdh5.7.0

export PATH=$ HADOOP_PREFIX/bin:$ JAVA_HOME/bin:$PATH

之后生效个人环境变量：
[hadoop@hadoop001 ~]$ source .bash_profile
检测是否能无密码ssh远程连接：
[hadoop@hadoop001 ~]$ ssh hadoop001
[hadoop@hadoop001 ~]$ which hdfs

（8）配置slaves
[hadoop@hadoop001 hadoop-2.6.0-cdh5.7.0]$ cd etc/hadoop
[hadoop@hadoop001 hadoop]$ vi slaves
将localhost改成hadoop001

（9）格式化

官网要求：
Execution
The following instructions are to run a MapReduce job locally. If you want to execute a job on YARN, see YARN on Single Node.
1、Format the filesystem:
$ bin/hdfs namenode -format
2、Start NameNode daemon and DataNode daemon:
$ sbin/start-dfs.sh
The hadoop daemon log output is written to the $HADOOP_LOG_DIR directory (defaults to $HADOOP_HOME/logs).
3、Browse the web interface for the NameNode; by default it is available at:
NameNode - http://localhost:50070/

[hadoop@hadoop001 hadoop-2.6.0-cdh5.7.0]$ bin/hdfs namenode -format

看到successfully formatted 和 with status 0说明格式化成功，否则会抛错

[hadoop@hadoop001 hadoop-2.6.0-cdh5.7.0]$ sbin/start-dfs.sh
18/10/04 11:39:08 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
Starting namenodes on [hadoop001]
hadoop001: starting namenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-namenode-hadoop001.out
hadoop001: starting datanode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-datanode-hadoop001.out
Starting secondary namenodes [0.0.0.0]
The authenticity of host ‘0.0.0.0 (0.0.0.0)’ can’t be established.
RSA key fingerprint is e2:06:2f:8d:c0:a9:34:18:29?d3:24:fd:81:7a:40.
Are you sure you want to continue connecting (yes/no)? yes
0.0.0.0: Warning: Permanently added ‘0.0.0.0’ (RSA) to the list of known hosts.
0.0.0.0: starting secondarynamenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-secondarynamenode-hadoop001.out
18/10/04 11:39:52 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable

（10）查看启动是否成功：jps命令
[hadoop@hadoop001 hadoop-2.6.0-cdh5.7.0]$ jps
13490 DataNode
13774 Jps
13366 NameNode
13651 SecondaryNameNode

DateNode 和NameNode 和SecondaryNameNode都启动了！！！

官网要求使用地址和端口号50070来看看网址是否ping通，查看我们部署的hadoop的概况：
来看看：47.75.243.240：50070

是ok的~~~

------------------------------------------------------未完待续------------------------------------------------