ubuntu18.04 LTS安装后,更新hadoop系列安装。这次针对hadoop,系统和安装环境如下:ubuntu18.04 LTS和hadoop3.0.3、jdk1.8.0_172。
一、安装ssh
sudo apt-get install ssh openssh-server
二、使用ssh无密登陆
ssh-keygen -t rsa
cd ~/.ssh
cat id_rsa.pub >> authorized_keys
ssh localhost
三、hadoop解压缩
tar -zxf /opt/software/hadoop-3.0.3.tar.gz -C /opt/modules/
四、配置hadoop文件
配置文件包括hadoop-env.sh、core-site.xml、mapred-site.xml、hdfs-site.xml,存放在etc/hadoop下面。
1. hadoop-env.sh
exportJAVA_HOME=/opt/modules/jdk1.8.0_172
2. core-site.xml
fs.defaultFS是hdfs集群访问入口
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/opt/modules/hadoop-3.0.3/tmp</value>
<description>A base for other temporary directories</description>
</property>
<property>
<name>fs.defaultFS</name>
<value>hdfs://lee:8020</value>
</property>
</configuration>
3. hdfs-site.xml
dfs.replicationblock副本数,web打不开可设置dfs.http.address
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
<description>replication of block</description>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/opt/modules/hadoop-3.0.3/tmp/dfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/opt/modules/hadoop-3.0.3/tmp/dfs/data</value>
</property>
<property>
<name>dfs.http.address</name>
<value>lee:50070</value>
</property>
</configuration>
4. 启动hdfs
(1)格式化namenode
bin/hdfs namenode -format
(2)启动namenode和datanode
sbin/hadoop-daemon.sh start namenode
sbin/hadoop-daemon.sh start datanode
或
sbin/start-dfs.sh
(3)web验证:http://lee:50070/
(4)上传文件
bin/hdfs dfs -mkdir /input
bin/hdfs dfs -put /opt/data/git.txt /input
bin/hdfs dfs -cat /input/git.txt
5. yarn-site.xml
yarn-env.sh增加JAVA_HOME配置。
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>lee</value>
</property>
</configuration>
6. mapred-site.xml
mapred-env.sh增加JAVA_HOME配置。
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
若报错如下:
则增加配置
<property>
<name>yarn.app.mapreduce.am.env</name>
<value>HADOOP_MAPRED_HOME=/opt/modules/hadoop-3.0.3</value>
</property>
<property>
<name>mapreduce.map.env</name>
<value>HADOOP_MAPRED_HOME=/opt/modules/hadoop-3.0.3</value>
</property>
<property>
<name>mapreduce.reduce.env</name>
<value>HADOOP_MAPRED_HOME=/opt/modules/hadoop-3.0.3</value>
</property>
7. 启动yarn
(1)启动resourcemanager和nodemanager
sbin/start-yarn.sh
(2)web验证
(3) 运行wordcount
bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.3.jar wordcount /input /output
查看结果
bin/hdfs dfs -cat /output/part*
8. 配置日志服务器
yarn-site.xml新增配置如下:
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
<description>启用日志聚合功能</description>
</property>
<property>
<name>yarn.log-aggregation.retain-seconds</name>
<value>86400</value>
<description>日志保存时间</description>
</property>
mapred-site.xml新增配置如下:
<property>
<name>mapreduce.jobhistory.address</name>
<value>lee:10020</value>
<description>进程通信</description>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>lee:19888</value>
<description>客户端访问入口</description>
</property>
启动historyserver服务
sbin/mr-jobhistory-daemon.sh start historyserver