Hadoop学习一(Hadoop安装)

环境：Oracle VM VirtualBox 安装的虚拟机系统 Ubuntu 16.0.4

第一步、安装JDK  请移步

第二步、安装SSH

                sudo apt-get install ssh

                sudo /etc/init.d/ssh start

第三步、创建用户

                sudo useradd -m hadoop -s /bin/bash  #这条命令创建了可以登陆的 hadoop 用户，并使用 /bin/bash 作为 shell

                sudo passwd hadoop # 设置密码

                sudo adduser hadoop sudo #增加管理员权限

                注销当前用户，在登陆界面使用刚创建的 hadoop 用户进行登陆

第四步、安装SSH Server

                 sudo apt-get install openssh-server

                 ssh localhost  #测试

                 设置SHH无密登录：

                首先退出刚才的 ssh，就回到了我们原先的终端窗口，然后利用 ssh-keygen 生成密钥，并将密钥加入到授权中

                exit

                cd ~/.ssh

                ssh-keygen -t rsa

                cat ./id_rsa.pub >> ./authorized_keys

                ssh localhost  #测试是否设置成功

第五步、hadoop 安装

               下载地址：http://hadoop.apache.org/releases.html

                sudo tar -zxvf ~/下载/hadoop-2.7.1.tar.gz -C /usr/local/   #解压到/usr/local目录下

                cd /usr/local/

                sudo mv ./hadoop-2.7.1/ ./hadoop   #文件夹改名hadoop

                sudo chown -R hadoop ./hadoop    #修改文件夹权限

                #进入目录下，检查命令是否可用

                cd /usr/local/hadoop

                ./bin/hadoop version

                设置环境变量

                sudo vim ~/.bashrc

                输入如下内容

                export JAVA_HOME=/usr/lib/jvm/java-8-oracle
                export HADOOP_HOME=/usr/local/hadoop
                export PATH=$PATH:$HADOOP_HOME/bin

                保存后执行命令  source ~/.bashrc

                测试是否配置生效:  hadoop version

第六步、hadoop配置

                （1）单机

                          Hadoop 默认模式为非分布式模式，无需进行其他配置即可运行。非分布式即单 Java 进程，方便进行调试。

                          ./bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar  

                         #可看到所有列子,包括 wordcount、terasort、join、grep 等。

                （2）伪分布式

                          主要是配置Hadoop的几个配置文件对应目录：$HADOOP_HOME/etc/hadoop/     (注：$HADOOP_HOME 是hadoop目录)

                          几个文件：hadoop-env.sh ，core-site.xml，hdfs-site.xml，yarn-site.xml，mapred-site.xml

                          1.hadoop-env.sh   该文件中设置java_home   

                             # export JAVA_HOME=${JAVA_HOME}
                             export JAVA_HOME=/usr/lib/jvm/java-8-oracle

                          2.core-site.xml     该文件中设置hdfs目录(hadoop.tmp.dir)及地址端口

                             <configuration>
                                    <property>
                                          <name>fs.defaultFS</name>
                                          <value>hdfs://192.168.56.101:9000</value>   <!-- 192.168.56.101是我虚拟机的IP-->
                                    </property>
                                    <property>
                                          <name>hadoop.tmp.dir</name>
                                          <value>/home/hadoop/hadoop/tmp</value>
                                    </property>
                             </configuration>
                          3.hdfs-site.xml  该文件中设置副本冗余数，我测试设置为1

                             <configuration>
                                   <property>
                                       <name>dfs.replication</name>
                                       <value>1</value>
                                  </property>
                             </configuration>

                           4.yarn-site.xml   yarn配置

                               <configuration>

                               <!-- Site specific YARN configuration properties -->
                                     <property>
                                            <name>yarn.nodemanager.aux-services</name>
                                            <value>mapreduce_shuffle</value>
                                     </property>
                                     <property>
                                            <name>yarn.resourcemanager.hostname</name>
                                            <value>localhost</value>
                                      </property>
                                 </configuration>

                             5.mapred-site.xml   yarn相关配置  （目录下没有这个文件，只有个mapred-site.xml.template，重命名为mapred-site.xml）

                               <configuration>
                                    <property>
                                         <name>mapreduce.framework.name</name>
                                         <value>yarn</value>
                                    </property>
                               </configuration>

                              都配置好后，进入$HADOOP_HOME/sbin/  目录  运行./start-dfs.sh启动dfs

                              启动好后，输入jps，看到如下内容，启动成功

                               hadoop@xintangli-VirtualBox:/usr/local/hadoop/sbin$ jps
                               26514 NameNode
                               26818 SecondaryNameNode
                               26661 DataNode
                               27710 Jps

                               浏览器里访问：http://localhost:50070

                              

                                                      启动成功！

                                                      命令行输入：hdfs dfs -ls /  可以查看目录