使用Docker搭建Hadoop

在Docker中下载镜像

docker pull ubuntu 

设置挂载点

如果需要共享文件的话在创建容器的时候设置挂载点，否则就要重新搭一遍环境，血淋淋的教训啊

docker run --name ContainerName -it -v SharePath:/data ubuntu /bin/bash  

ContainerName是容器名，SharePath是共享的挂载点，/data是容器里共享文件的路径， /bin/bash用命令行的形式创建容器

更新软件源

在命令行输入

apt-get update  
apt-get install vim  
vim /etc/apt/sources.list  

在文件的最后追加：

deb http://mirrors.aliyun.com/ubuntu/ bionic main restricted universe multiverse  
deb http://mirrors.aliyun.com/ubuntu/ bionic-security main restricted universe multiverse  
deb http://mirrors.aliyun.com/ubuntu/ bionic-updates main restricted universe multiverse  
deb http://mirrors.aliyun.com/ubuntu/ bionic-proposed main restricted universe multiverse  
deb http://mirrors.aliyun.com/ubuntu/ bionic-backports main restricted universe multiverse  
deb-src http://mirrors.aliyun.com/ubuntu/ bionic main restricted universe multiverse  
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-security main restricted universe multiverse  
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-updates main restricted universe multiverse  
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-proposed main restricted universe multiverse  
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-backports main restricted universe multiverse  

在命令行输入：

apt-get update  
apt-get upgrade 

安装SSH

在命令行输入

apt-get install openssh-server

此时SSH服务还未开启

service ssh start  

输入ssh localhost,若是无法登陆修改sshd_config文件

vim /etc/ssh/sshd_config

在文件中找到

#Authentication:  
#LoginGraceTime 120  
#PermitRootLogin without-password  
#StrictModes yes  

修改为下面内容

#Authentication:  
#LoginGraceTime 120  
#PermitRootLogin without-password
PermitRootLogin yes   
#StrictModes yes     

设置无密码登录

ssh-keygen -t rsa  

一直按回车就可以了，然后把公钥加入到授权中：

cat ./id_rsa.pub >> ./authorized_keys  

使用以下命令就可以无密码登录到主机：

ssh localhost  

安装JDK

方法一：

安装OpenJDK8，可能会出现找不到环境变量的问题。而且OpenJDK是精简版的JDK，可能会出现问题。

apt-get install openjdk-8-jdk  

配置Java环境变量

vim /etc/profile  

在文件尾追加

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64  
export JRE_HOME=$JAVA_HOME/jre  
export CLASSPATH=$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH  
export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH  

验证是否安装成功

java -version  
openjdk version "1.8.0_232"  
OpenJDK Runtime Environment (build 1.8.0_232-8u232-b09-0ubuntu1~18.04.1-b09)  
OpenJDK 64-Bit Server VM (build 25.232-b09, mixed mode)

方法二：

安装Orcal JDK

将JDK文件放在开头提及的文件夹里，在容器里解压到/usr/lib/jdk，并按照步骤一的方法配置环境变量，记得修改路径。另外lib文件下的路径需要自己创建文件夹

安装Hadoop

首先安装links2，方便在命令行打开网页

apt-get install links2

打开Hadoop网站,下载需要的版本并安装

links https://mirrors.cnnic.cn/apache/hadoop/common/  
tar -zxf /hadoop-3.2.1.tar.gz -C /usr/local  
mv ./hadoop-3.2.1/ ./hadoop

验证安装是否成功

/usr/local/hadoop/bin/hadoop version  
Hadoop 3.2.1  
Source code repository https://gitbox.apache.org/repos/asf/hadoop.git -r   b3cbbb467e22ea829b3808f4b7b01d07e0bf3842  
Compiled by rohithsharmaks on 2019-09-10T15:56Z  
Compiled with protoc 2.5.0  
From source with checksum 776eaf9eee9c0ffc370bcbc1888737  
This command was run using /usr/local/hadoop/share/hadoop/common/  hadoop-common-3.2.1.jar  

以伪分布式配置Hadoop

伪分布式需要修改2个配置文件core-site.xml和hdfs-site.xml

cd /usr/local/hadoop/etc/hadoop  
vim core-site.xml 
**加入下列信息：**
<configuration>  
    <property>  
            <name>hadoop.tmp.dir</name>  
            <value>/usr/local/hadoop/tmp</value>  
            <description>Abase for other temporary directories.</description>  
    </property>
    <property>
            <name>fs.defaultFS</name>
            <value>hdfs://localhost:9000</value>
    </property>
</configuration>
	    ```
```shell	    
vim hdfs-site.xml

     <property>
                <name>dfs.replication</name>
                <value>1</value>
        </property>
        <property>
                <name>dfs.namenode.name.dir</name>
                <value>/usr/local/hadoop/tmp/dfs/name</value>
        </property>
        <property>
                <name>dfs.datanode.data.dir</name>
                <value>/usr/local/hadoop/tmp/dfs/data</value>
        </property>

切换回hadoop主目录：/usr/local/hadoop

NameNode的格式化:

./bin/hdfs namenode -format  

出现以下信息即格式化成功:

    2020-01-02 12:30:07,439 INFO namenode.NameNode: SHUTDOWN_MSG:  
    /************************************************************
    SHUTDOWN_MSG: Shutting down NameNode at 08c0610a5e3d/172.17.0.3
    ************************************************************/

开启NameNode和DataNode守护进程

vim ./sbin/start-dfs.sh  

加入如下信息：  

   #!/usr/bin/env bash  
    HDFS_DATANODE_USER=root  
    HADOOP_SECURE_DN_USER=hdfs  
    HDFS_NAMENODE_USER=root  
    HDFS_SECONDARYNAMENODE_USER=root  

vim ./sbin/stop-dfs.sh  
加入如下信息：  
    \#!/usr/bin/env bash  
    YARN_RESOURCEMANAGER_USER=root  
    HADOOP_SECURE_DN_USER=yarn  
    YARN_NODEMANAGER_USER=root   

./sbin/start-dfs.sh

若是报错找不到JAVA_HOME，则再手动配置环境变量

    # Technically, the only required environment variable is JAVA_HOME.
    # All others are optional.  However, the defaults are probably not
    # preferred.  Many sites configure these options outside of Hadoop,
    # such as in /etc/profile.d
    # The java implementation to use. By default, this environment
    # variable is REQUIRED on ALL platforms except OS X!
    # export JAVA_HOME=
    export JAVA_HOME=/usr/lib/jdk/jdk1.8.0_231 

出现如下则成功：

Starting namenodes on [localhost]  
Starting datanodes  
Starting secondary namenodes [08c0610a5e3d]  

将Hadoop加入到环境变量

vim /etc/bash.bashrc   
export HADOOP_HOME=/usr/local/hadoop  
export PATH=${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin:$PATH
source /etc/bash.bashrc

Hadoop的使用

创建输入文件夹

./bin/hdfs dfs -mkdir -p /user/hadoop  
./bin/hdfs dfs -mkdir /user/hadoop/input
./bin/hdfs dfs -put /home/BigData/mag_papers_0/mag_papers_0.txt /user/hadoop/input  

执行

./bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.1.jar wordcont /user/hadoop/input output 

查看输出结果

./bin/hdfs dfs -cat output/*`