Ubuntu18.04搭建Hadoop环境

一、安装JDK

• 更新源

sudo apt-get update

sudo apt install openjdk-8-jdk-headless

请一定注意JDK不要安装最新版此前因为安装版本太高走了很多弯路

• 查看JDK版本

java -version

• 查看JDK安装目录

update-alternatives --display java

二、设在SSH免密码登录

• 安装SSH

sudo apt-get install ssh

• 安装rsync

sudo apt-get install rsync

• 产生SSH KEY（密钥）进行后续验证

hduser@master:~$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
Generating public/private dsa key pair.
Your identification has been saved in /home/hduser/.ssh/id_dsa.
Your public key has been saved in /home/hduser/.ssh/id_dsa.pub.
The key fingerprint is:
SHA256:OSfzo5m9QUvq6nsb4a2HbRxh8Y7wc1KFSrJgRRS6i4o hduser@master
The key's randomart image is:
+---[DSA 1024]----+
|       +=.   .   |
|      o.. o . .  |
|     ... + + .   |
|       .o.+ o    |
|      . S++=     |
|     . o %*.o    |
|    . . ++*=     |
| . .   .oB=o     |
|E .  .++B+o.     |
+----[SHA256]-----+

• 查看产生的SSH KEY

ll /home/hduser/.ssh

• 将产生的KEY 放到许可文件当中

hduser@master:~$ ll /home/hduser/.ssh
总用量 16
drwx------  2 hduser hduser 4096 8月   1 09:54 ./
drwxr-xr-x 42 hduser hduser 4096 8月   1 08:57 ../
-rw-------  1 hduser hduser  668 8月   1 09:54 id_dsa
-rw-r--r--  1 hduser hduser  603 8月   1 09:54 id_dsa.pub
hduser@master:~$ ll ~/.ssh
总用量 16
drwx------  2 hduser hduser 4096 8月   1 09:54 ./
drwxr-xr-x 42 hduser hduser 4096 8月   1 08:57 ../
-rw-------  1 hduser hduser  668 8月   1 09:54 id_dsa
-rw-r--r--  1 hduser hduser  603 8月   1 09:54 id_dsa.pub
hduser@master:~$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

三、下载安装Hadoop

wget http://ftp.twaren.net/Unix/Web/apache/hadoop/common/hadoop-2.7.6/hadoop-2.7.6.tar.gz

考虑到现在的企业大多用2.7版本自里安装2.7

• 解压Hadoop-2.7.6.tar.gz

sudo tar -zxvf hadoop-2.7.6.tar.gz 

• 移动Hadoop到目录usr/loacl/hadoop

 sudo mv hadoop-2.7.6 /usr/local/hadoop

• 查看目录

hduser@master:~$ ll /usr/local/hadoop/
总用量 144
drwxr-xr-x  9 20415 systemd-journal  4096 4月  18  2018 ./
drwxr-xr-x 12 root  root             4096 8月   1 11:18 ../
drwxr-xr-x  2 20415 systemd-journal  4096 4月  18  2018 bin/
drwxr-xr-x  3 20415 systemd-journal  4096 4月  18  2018 etc/
drwxr-xr-x  2 20415 systemd-journal  4096 4月  18  2018 include/
drwxr-xr-x  3 20415 systemd-journal  4096 4月  18  2018 lib/
drwxr-xr-x  2 20415 systemd-journal  4096 4月  18  2018 libexec/
-rw-r--r--  1 20415 systemd-journal 86424 4月  18  2018 LICENSE.txt
-rw-r--r--  1 20415 systemd-journal 14978 4月  18  2018 NOTICE.txt
-rw-r--r--  1 20415 systemd-journal  1366 4月  18  2018 README.txt
drwxr-xr-x  2 20415 systemd-journal  4096 4月  18  2018 sbin/
drwxr-xr-x  4 20415 systemd-journal  4096 4月  18  2018 share/

• 常用目录说明

目录	说明
bin	各项运行文件包括Hadoop,HDFS,YARS等
sbin	各项shall运行文件包括start-all.sh和stop-all.sh
etc	etc/Hadoop子目录包含Hadoop配置文件 ,例如Hadoop-env.sh,core-site.xml,yars-site.xml,mapred-site.xml,hdfs-site.xml
bli	Hadoop函数
logs	系统日志，可以查看但系统运行状况，运行有问题是可以从文件中找出问题

四、Hadoop环境设置

sudo gedit ~/.bashrc

• 文件末尾添加

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 # JDK安装目录
export HADOOP_HOME=/usr/local/hadoop 
export PATH=$PATH:$HADOOP_HOME/bin 
export PATH=$PATH:$HADOOP_HOME/sbin 
export HADOOP_MAPRED_HOME=$HADOOP_HOME 
export HADOOP_COMMON_HOME=$HADOOP_HOME 
export HADOOP_HDFS_HOME=$HADOOP_HOME 
export YARN_HOME=$HADOOP_HOME 
export HADOOP_COMMON_HOME=$HADOOP_HOME 
export HADOOP_HDFS_HOME=$HADOOP_HOME 
export YARN_HOME=$HADOOP_HOME 
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native 
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib" 
export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native:$JAVA_LIBRARY_PATH

• 激活环境

hduser@master:~$ source ~/.bashrc
hduser@master:~$ hadoop
Usage: hadoop [--config confdir] [COMMAND | CLASSNAME]
  CLASSNAME            run the class named CLASSNAME
 or
  where COMMAND is one of:
  fs                   run a generic filesystem user client
  version              print the version
  jar <jar>            run a jar file
                       note: please use "yarn jar" to launch
                             YARN applications, not this command.
  checknative [-a|-h]  check native hadoop and compression libraries availability
  distcp <srcurl> <desturl> copy file or directories recursively
  archive -archiveName NAME -p <parent path> <src>* <dest> create a hadoop archive
  classpath            prints the class path needed to get the
  credential           interact with credential providers
                       Hadoop jar and the required libraries
  daemonlog            get/set the log level for each daemon
  trace                view and modify Hadoop tracing settings

Most commands print help when invoked w/o parameters.

五、修改Hadoop配置文件

• 修改hadoop-env.sh

sudo gedit /usr/local/hadoop/etc/hadoop/hadoop-env.sh

修改之前： export JAVA_HOME=${JAVA_HOME}
修改之后：export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

• 修改core-site.xml

sudo gedit /usr/local/hadoop/etc/hadoop/core-site.xml

<configuration>
	<property>
		<name>fs.default.name</name>	
		<value>hdfs://localhost:9000</value>
	</property>
</configuration>

• 修改yarn-site.xml

sudo gedit /usr/local/hadoop/etc/hadoop/yarn-site.xml

<configuration>
<!-- Site specific YARN configuration properties -->
<property>
	<name>yarn.nodemanager.aux-services</name>
	<value>mapreduce_shuffle</value>
</property>
	<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
	<value>org.apache.hadoop.mapred.ShuffleHandler</value>
<property>
</property>
</configuration>

• 设置mapred-site.xml

sudo cp /usr/local/hadoop/etc/hadoop/mapred-site.xml.template /usr/local/hadoop/etc/hadoop/mapred-site.xml

• 编辑mapred-site.xml

<configuration>
	<property>
		<name>mapreduce.framework.name</name>
		<value>yarn</value>
	</property>
</configuration>

• 编辑hdfs-site.xml

sudo gedit /usr/local/hadoop/etc/hadoop/hdfs-site.xml

<configuration>
	<property>
		<name>dfs.replication</name>
		<value>3</value>
	</property>
	<property>
		<name>dfs.namenode.name.dir</name>
		<value>file:/usr/local/hadoop/hadoop_data/hdfs/namenode</value>
	</property>
	<property>
		<name>dfs.datanode.data.dir</name>
		<value>file:/usr/local/hadoop/hadoop_data/hdfs/datanode</value>
	</property>
</configuration>

• 创建并格式化HDFS目录

sudo mkdir -p /usr/local/hadoop/hadoop_data/hdfs/namenode
sudo mkdir -p /usr/local/hadoop/hadoop_data/hdfs/datanode
sudo chown hduser:hduser -R /usr/local/hadoop

• 将HDFS格式化

六、启动Hadoop

hduser@master:~$ start-dfs.sh
19/08/01 13:10:26 WARN conf.Configuration: Unexpected tag in conf file yarn-site.xml: expected <property> but found <name>
19/08/01 13:10:26 WARN conf.Configuration: Unexpected tag in conf file yarn-site.xml: expected <property> but found <value>
19/08/01 13:10:26 WARN conf.Configuration: Unexpected tag in conf file yarn-site.xml: expected <property> but found <name>
19/08/01 13:10:26 WARN conf.Configuration: Unexpected tag in conf file yarn-site.xml: expected <property> but found <value>
Starting namenodes on [localhost]
The authenticity of host 'localhost (127.0.0.1)' can't be established.
ECDSA key fingerprint is SHA256:4TVFf1D79N8Df9Qg9JQLyX/1Lj/1h1reMBN6/W5sQ88.
Are you sure you want to continue connecting (yes/no)? yes
localhost: Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
hduser@localhost's password: 
localhost: starting namenode, logging to /usr/local/hadoop/logs/hadoop-hduser-namenode-master.out
hduser@localhost's password: 
localhost: starting datanode, logging to /usr/local/hadoop/logs/hadoop-hduser-datanode-master.out
Starting secondary namenodes [0.0.0.0]
The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established.
ECDSA key fingerprint is SHA256:4TVFf1D79N8Df9Qg9JQLyX/1Lj/1h1reMBN6/W5sQ88.
Are you sure you want to continue connecting (yes/no)? yes
0.0.0.0: Warning: Permanently added '0.0.0.0' (ECDSA) to the list of known hosts.
hduser@0.0.0.0's password: 
0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop/logs/hadoop-hduser-secondarynamenode-master.out
19/08/01 13:11:15 WARN conf.Configuration: Unexpected tag in conf file yarn-site.xml: expected <property> but found <name>
19/08/01 13:11:15 WARN conf.Configuration: Unexpected tag in conf file yarn-site.xml: expected <property> but found <value>
19/08/01 13:11:15 WARN conf.Configuration: Unexpected tag in conf file yarn-site.xml: expected <property> but found <name>
19/08/01 13:11:15 WARN conf.Configuration: Unexpected tag in conf file yarn-site.xml: expected <property> but found <value>
hduser@master:~$ start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop/logs/yarn-hduser-resourcemanager-master.out
hduser@localhost's password: 
localhost: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-hduser-nodemanager-master.out

浏览器打开输入：localhost:8088