ubuntu 18.04LTS hadoop 3.0.3安装

ubuntu18.04 LTS安装后，更新hadoop系列安装。这次针对hadoop，系统和安装环境如下:ubuntu18.04 LTS和hadoop3.0.3、jdk1.8.0_172。

一、安装ssh

sudo apt-get install ssh openssh-server

二、使用ssh无密登陆

ssh-keygen -t rsa

cd ~/.ssh

cat id_rsa.pub >> authorized_keys

ssh localhost

三、hadoop解压缩

tar -zxf /opt/software/hadoop-3.0.3.tar.gz -C /opt/modules/

四、配置hadoop文件

配置文件包括hadoop-env.sh、core-site.xml、mapred-site.xml、hdfs-site.xml，存放在etc/hadoop下面。

1. hadoop-env.sh

exportJAVA_HOME=/opt/modules/jdk1.8.0_172

2. core-site.xml

fs.defaultFS是hdfs集群访问入口

<configuration>

<property>

<name>hadoop.tmp.dir</name>

<value>/opt/modules/hadoop-3.0.3/tmp</value>

<description>A base for other temporary directories</description>

</property>

<property>

<name>fs.defaultFS</name>

<value>hdfs://lee:8020</value>

</property>

</configuration>

3. hdfs-site.xml

dfs.replicationblock副本数，web打不开可设置dfs.http.address

<configuration>

<property>

<name>dfs.replication</name>

<value>1</value>

<description>replication of block</description>

</property>

<property>

<name>dfs.namenode.name.dir</name>

<value>/opt/modules/hadoop-3.0.3/tmp/dfs/name</value>

</property>

<property>

<name>dfs.datanode.data.dir</name>

<value>/opt/modules/hadoop-3.0.3/tmp/dfs/data</value>

</property>

<property>

<name>dfs.http.address</name>

<value>lee:50070</value>

</property>

</configuration>

4. 启动hdfs

(1)格式化namenode

bin/hdfs namenode -format

(2)启动namenode和datanode

sbin/hadoop-daemon.sh start namenode

sbin/hadoop-daemon.sh start datanode

或

sbin/start-dfs.sh

(3)web验证：http://lee:50070/

(4)上传文件

bin/hdfs dfs -mkdir /input

bin/hdfs dfs -put /opt/data/git.txt /input

bin/hdfs dfs -cat /input/git.txt

5. yarn-site.xml

yarn-env.sh增加JAVA_HOME配置。

<configuration>

<property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

<property>

<name>yarn.resourcemanager.hostname</name>

<value>lee</value>

</property>

</configuration>

6. mapred-site.xml

mapred-env.sh增加JAVA_HOME配置。

<configuration>

<property>

<name>mapreduce.framework.name</name>

<value>yarn</value>

</property>

</configuration>

若报错如下：

则增加配置

<property>

<name>yarn.app.mapreduce.am.env</name>

<value>HADOOP_MAPRED_HOME=/opt/modules/hadoop-3.0.3</value>

</property>

<property>

<name>mapreduce.map.env</name>

<value>HADOOP_MAPRED_HOME=/opt/modules/hadoop-3.0.3</value>

</property>

<property>

<name>mapreduce.reduce.env</name>

<value>HADOOP_MAPRED_HOME=/opt/modules/hadoop-3.0.3</value>

</property>

7. 启动yarn

（1）启动resourcemanager和nodemanager

sbin/start-yarn.sh

（2）web验证

http://lee:8088

(3) 运行wordcount

bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.3.jar wordcount /input /output

查看结果

bin/hdfs dfs -cat /output/part*

8. 配置日志服务器

yarn-site.xml新增配置如下：

<property>

<name>yarn.log-aggregation-enable</name>

<value>true</value>

<description>启用日志聚合功能</description>

</property>

<property>

<name>yarn.log-aggregation.retain-seconds</name>

<value>86400</value>

<description>日志保存时间</description>

</property>

mapred-site.xml新增配置如下：

<property>

<name>mapreduce.jobhistory.address</name>

<value>lee:10020</value>

<description>进程通信</description>

</property>

<property>

<name>mapreduce.jobhistory.webapp.address</name>

<value>lee:19888</value>

<description>客户端访问入口</description>

</property>

启动historyserver服务

sbin/mr-jobhistory-daemon.sh start historyserver