大数据＋零基础＋大数据入门（一）

搜藏网站：
1.阿帕奇官网：
www.apache.org
2.cdh hadoop:
http://archive.cloudera.com/cdh5/cdh/5/
CDH：每个组件都会对应一个用户
组件 用户
hdfs hdfs
yarn yarn
zookeeper zookeeper
hbase hbase

一.安装
①创建hadoop用户：

[root@hadoop002 ~]# useradd hadoop
[root@hadoop002 ~]# su - hadoop
[hadoop@hadoop002 ~]#mkdir app
[hadoop@hadoop002 ~]#cd app/
[hadoop@hadoop002 app]#wget http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.7.0.tar.gz
或者
[hadoop@hadoop002 app]#rz上传本地软件包
[hadoop@hadoop002 app]#tar -xzvf hadoop-2.6.0-cdh5.7.0.tar.gz
[hadoop@hadoop002 app]#cd hadoop-2.6.0-cdh5.7.0
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]#ll  查看
drwxr-xr-x  2 hadoop hadoop  4096 Mar 24  2016 bin  可执行脚本
drwxr-xr-x  2 hadoop hadoop  4096 Mar 24  2016 bin-mapreduce1
drwxr-xr-x  3 hadoop hadoop  4096 Mar 24  2016 cloudera
drwxr-xr-x  6 hadoop hadoop  4096 Mar 24  2016 etc  配置目录(conf)
drwxr-xr-x  5 hadoop hadoop  4096 Mar 24  2016 examples
drwxr-xr-x  3 hadoop hadoop  4096 Mar 24  2016 examples-mapreduce1
drwxr-xr-x  2 hadoop hadoop  4096 Mar 24  2016 include
drwxr-xr-x  3 hadoop hadoop  4096 Mar 24  2016 lib  jar包目录
drwxr-xr-x  2 hadoop hadoop  4096 Mar 24  2016 libexec
drwxr-xr-x  3 hadoop hadoop  4096 Mar 24  2016 sbin hadoop组件的启动 停止脚本
drwxr-xr-x  4 hadoop hadoop  4096 Mar 24  2016 share
drwxr-xr-x 17 hadoop hadoop  4096 Mar 24  2016 src
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]#vi etc /hadoop/croe-site.xml   配置为以下标准：
<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>
</configuration>
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]#vi etc/hadoop/hdfs-site.xml   配置为以下标准：
<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
</configuration>

②配置ssh localhost 无密码信任关系：

[hadoop@hadoop002 ~]$ ssh - keygen
#连续三下回车键#   以下内容不需要查看更改：
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa): 
Created directory '/home/hadoop/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
ba:48:3d:ff:af:4d:da:74:67:31:d6:98:ad:a0:b3:76 hadoop@hadoop002
The key's randomart image is:
+--[ RSA 2048]----+
|                 |
|                 |
|                 |
|               +.|
|        S   . o+o|
|     . .   . ...o|
|    . +   o o o o|
|   . . +  .OE. o |
|    . . .o=++    |
+-----------------+
[hadoop@hadoop002 ~]$cd. ssh
[hadoop@hadoop002 ssh]$ll
total 8
-rw------- 1 hadoop hadoop 1675 Feb 13 22:36 id_rsa     私钥
-rw-r--r-- 1 hadoop hadoop  398 Feb 13 22:36 id_rsa.pub 公钥
[hadoop@hadoop002 .ssh]$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
[hadoop@hadoop002 .ssh]$ ll
total 12
-rw-rw-r-- 1 hadoop hadoop  398 Feb 13 22:37 authorized_keys
-rw------- 1 hadoop hadoop 1675 Feb 13 22:36 id_rsa
-rw-r--r-- 1 hadoop hadoop  398 Feb 13 22:36 id_rsa.pub
-rw-r--r-- 1 hadoop hadoop    0 Feb 13 22:39 known_hosts

注意：ssh localhost date 是需要输入密码，但是这个用户是没有配置密码。
我们应该在没有配置密码情况下去完成无密码信任呢？
③修改权限，改为600

[hadoop@hadoop002 .ssh]$ chmod 600 authorized_keys
[hadoop@hadoop002 .ssh]$ ssh localhost date
The authenticity of host 'localhost (127.0.0.1)' can't be established.
RSA key fingerprint is b1:94:33:ec:95:89:bf:06:3b:ef:30:2f:d7:8e:d2:4c.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
Wed Feb 13 22:41:17 CST 2019
[hadoop@hadoop002 .ssh]$ ssh localhost date
Wed Feb 13 22:41:22 CST 2019

④格式化

[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ bin/hdfs namenode -format	

⑤启动

[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ sbin/start-dfs.sh

⑥ssh 信任关系 是配置localhost

[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ ssh localhost date
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ sbin/stop-dfs.sh
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ sbin/start-dfs.sh
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ jps
15059 Jps
14948 SecondaryNameNode 第二名称节点 老二
14783 DataNode  数据节点  小弟
14655 NameNode  名称节点  老大 读写

[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ netstat -nle | grep 14655
00.0.0.0：50070 代表外部可以访问

⑦配置环境变量

[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ 	cat ~/.bash_profile   配置为以下标准：
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

# User specific environment and startup programs
export HADOOP_PREFIX=/home/hadoop/app/hadoop-2.6.0-cdh5.7.0
export PATH=$HADOOP_PREFIX/bin:$PATH

⑧命令: hdfs dfs操作命令和Linux命令极其相似，可以通过命令帮助进行查询

二.部署JDK
①创建/usr/java 和 /usr/share/java 文件

[root@hadoop002 ~]#mkdir /usr/java
JDK部署   
[root@hadoop002 ~]#mkdir /usr/share/java
部署CDH需要mysql jdbc jar包

②上传安装包，并进行解压，更改软件权限

[root@hadoop002 java]#rz 上传jdk-8u45-linux-x64.gz
[root@hadoop002 java]# tar -xzvf jdk-8u45-linux-x64.gz 
[root@hadoop002 java]# ll
会发现解压后权限发生改变:
[root@hadoop002 java]# ll
total 319156
drwxr-xr-x 8 uucp 143      4096 Apr 11  2015 jdk1.8.0_45
需要将权限变更一下:
[root@hadoop002 java]# chown -R root:root jdk1.8.0_45
再次 ll 查看,变更成功：
[root@hadoop002 java]# ll
total 319156
drwxr-xr-x 8 root root      4096 Apr 11  2015 jdk1.8.0_45

③配置java环境变量：

[root@hadoop002 java]# vi /etc/profile  
添加以下内容：
#env
export JAVA_HOME=/usr/java/jdk1.8.0_45
export JRE_HOME=$JAVA_HOME/jre
export CLASSPATH=.:$JAVA_HOME/lib:$JER_HOME/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$JER_HOME/bin:$PATH
[root@hadoop002 java]# source /etc/profile  
[root@hadoop002 java]#which java
/usr/java/jdk1.8.0_45/bin/java

注意：知识点

bin/bash   用户可以登录
sbin/nologin  用户不可以登录
登录办法：
[root@hadoop002 ~]#cat /etc/password
找到想要登录的用户
将尾部的 sbin/nologin  改为  bin/bash