伪分布式
1 修改 vim etc/hadoop/core-site.xml 文件
添加 //hadoop 是主机名称
<configuration>
<!-- 指定HDFS中nameNode地址 -->
<property>
<name>fs.defaultFS</name>
<value>hdfs://hadoop:9000</value>
</property>
<!-- 指定hadoop运行时产生的文件存储路径 -->
<property>
<name>hadoop.tmp.dir</name>
<value>/opt/module/hadoop-3.3.0/data/tmp</value>
</property>
</configuration>
2 修改vim etc/hadoop/hadoop-env.sh 文件
修改 export JAVA_HOME=/opt/module/jdk-15.0.1
3 修改 vim etc/hadoop/hdfs-site.xml 文件
添加
<!-- 指定HDFS副本个数(默认 3个) -->
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
4 格式化HDFS 文件系统
bin/hdfs namenode -format
启动 namenode
sbin/hadoop-daemon.sh start namenode 或者
bin/hdfs --daemon start namenode(推荐)
启动 datanode
sbin/hadoop-daemon.sh start datanode 或者
bin/hdfs --daemon start datanode(推荐)
查看是否启动成功
jps
http://192.168.159.100:9870 hadoop3 之前的端口是50070,
// 如果外部无法连接 可以先关闭防火墙则能联通
1,查看防火墙状态
systemctl status firewalld.service
2,开启防火墙
systemctl start firewalld.service
3,关闭防火墙
systemctl stop firewalld.service
4,禁用防火墙
systemctl disable firewalld.service
HDFS
// 创建目录
bin/hdfs dfs -mkdir -p /pys/vmtest
//将linux文件上传到HDFS
bin/hdfs dfs -put wcoutput/part-r-00000 /pys/vmtest
启动YAN
修改 vim etc/hadoop/yarn-site.xml 文件
添加
<!-- Reducer 获取数据方式 -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<!--指定yarn的resourcemanager地址 -->
<property>
<name>yarn.resourcemanager.hostname</name>
<value>hadoop</value>
</property>
修改 vim etc/hadoop/mapred-site.xml 文件
添加
<!-- 指定MR运作在yarn上 -->
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<!-- hadoop3要配置 -->
<property>
<name>yarn.app.mapreduce.am.env</name>
<value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
</property>
<property>
<name>mapreduce.map.env</name>
<value>HADOOP_MAPRED_HOME=${HADOP_HOME}</value>
</property>
<property>
<name>mapreduce.reduce.env</name>
<value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
</property>
<property>
<name>mapreduce.application.classpath</name>
<value>
${HADOOP_HOME}/etc/hadoop,
${HADOOP_HOME}/share/hadoop/common/*,
${HADOOP_HOME}/share/hadoop/common/lib/*,
${HADOOP_HOME}/share/hadoop/hdfs/*,
${HADOOP_HOME}/share/hadoop/hdfs/lib/*,
${HADOOP_HOME}/share/hadoop/mapreduce/*,
${HADOOP_HOME}/share/hadoop/mapreduce/lib/*,
${HADOOP_HOME}/share/hadoop/yarn/*,
${HADOOP_HOME}/share/hadoop/yarn/lib/*
</value>
</property>
启动 resourcemanager
sbin/yarn-daemon.sh start resourcemanager
yarn --daemon start resourcemanager (推荐)
启动 nodemanager
sbin/yarn-daemon.sh start nodemanager
yarn --daemon start nodemanager(推荐)
运行wordcount
hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.0.jar wordcount /pys/wcinput /pys/wcoutput
配置历史服务器
修改 vim etc/hadoop/mapred-site.xml 文件
添加
<!-- 历史服务器地址 -->
<property>
<name>mapreduce.jobhistory.address</name>
<value>hadoop:10020</value>
</property>
<!-- 历史服务器web地址 -->
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>hadoop:19888</value>
</property>
启动历史服务器
sbin/mr-jobhistory-daemon.sh start historyserver
mapred --daemon start historyserver (推荐)
配置日志聚集
修改 vim etc/hadoop/yarn-site.xml 文件
添加
<!-- 日志聚集功能使用 -->
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<!-- 日志保留时间设置 -->
<property>
<name>yarn.log-aggregation.retain-seconds</name>
<value>604800</value>
</property>
完全分布式配置
1 创建3台虚拟机
修改IP地址
vim /etc/sysconfig/network-scripts/ifcfg-ens33
IPADDR=192.168.159.101
GATEWAY=192.168.159.2
DNS1=192.168.159.2
BOOTPROTO=static //修改为 static
主机名称
vim /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=hadoop01
修改映射关系
vim /etc/hosts
192.168.159.100 hadoop
192.168.159.101 hadoop01
192.168.159.102 hadoop02
192.168.159.103 hadoop03
2 安装JDK
[root@hadoop01 ~]# vim .bashrc
#JAVA_HOME
export JAVA_HOME=/opt/module/jdk-15.0.1
export PATH=
J
A
V
A
H
O
M
E
/
b
i
n
:
JAVA_HOME/bin:
JAVAHOME/bin:PATH:
#HADOOP
export HADOOP_INSTALL=/opt/module/hadoop-3.3.0
export PATH=
P
A
T
H
:
PATH:
PATH:HADOOP_INSTALL/bin
export PATH=
P
A
T
H
:
PATH:
PATH:HADOOP_INSTALL/sbin
// 拷贝安装包数据
从hadoop拷贝数据到 hadoop01上
scp -r module root@hadoop01:/opt/module
从hadoop02上将hadoop数据拉取过来
scp -r root@hadoop:/opt/module /opt/module
在hadoop02上将hadoop数据拷贝到hadoop03
scp -r root@hadoop:/opt/module root@hadoop03:/opt/module
// 拷贝 ~/.bashrc
scp -r root@hadoop:~/.bashrc ~/.bashrc
生效文件
source ~/.bashrc
// rync拷贝 差异化文件
rsync -rvl /opt/module/a.txt root@hadoop03:/opt/module/a.txt
-r 递归 可以拷贝目录
-v 显示拷贝过程
-l 拷贝符号链接
3 集群脚本编写
mkdir bin // 创建bin目录
touch xsync // 创建xsync文件
chmod 777 xsync //设置其他用户可操作
#!/bin/bash
# 获得输入参数 没有参数直接退出
pcount=$#
if((pcount==0));then
echo no args;
exit;
fi
#获得文件名称
p1=$1
fname=`basename $p1`
echo fname=$fname
#获取上级目录得到绝对路径
pdir=`cd -P $(dirname $p1); pwd`
echo pdir=$pdir
#获得当前用户
user=`whoami`
#循环跑
for((host=2;host<4;host++));do
echo ---------------hadoop$host------------
rsync -rvl $pdir/$fname $user@hadoop0$host:$pdir
done
执行 xsync xsync 将文件分发到hadoop02和03
4 集群部署规划
namenode hadoop01
resourcemanager hadoop02
secondarynamenode hadoop03
5 格式化namenode
rm -rf data/ logs/
bin/hdfs namenode -format
启动 bin/hdfs --daemon start namenode
启动 bin/hdfs --daemon start datanode
6 配置SSH免密登录
hadoop01上配置 nameNode 需要访问集群的datanode
hadoop02上配置 resourcemanager 需要访问集群的datanode
cd
cd .ssh/
ssh-keygen -t rsa
ssh-copy-id hadoop01
ssh-copy-id hadoop02
ssh-copy-id hadoop03
7 启动集群
修改vim etc/hadoop/workers文件 hadoop3开始workers文件,之前的是slaves 文件
hadoop01
hadoop02
hadoop03
注意不能有空格,空行
启动集群 sbin/start-dfs.sh
报错
Starting namenodes on [master]
ERROR: Attempting to operate on hdfs namenode as root
ERROR: but there is no HDFS_NAMENODE_USER defined. Aborting operation.
Starting datanodes
ERROR: Attempting to operate on hdfs datanode as root
ERROR: but there is no HDFS_DATANODE_USER defined. Aborting operation.
Starting secondary namenodes [master]
ERROR: Attempting to operate on hdfs secondarynamenode as root
ERROR: but there is no HDFS_SECONDARYNAMENODE_USER defined. Aborting operation.
解决方法:
对于start-dfs.sh和stop-dfs.sh文件,文件头部添加下列参数:
HDFS_DATANODE_USER=root
HADOOP_SECURE_DN_USER=hdfs
HDFS_NAMENODE_USER=root
HDFS_SECONDARYNAMENODE_USER=root
对于start-yarn.sh和stop-yarn.sh文件,添加下列参数:
YARN_RESOURCEMANAGER_USER=root
HADOOP_SECURE_DN_USER=yarn
YARN_NODEMANAGER_USER=root
启动yarn 必须在部署yarn的机器上启动
sbin/start-yarn.sh
集群时间同步
查看是否安装 chrony
rpm -q chrony
服务端修改 vim etc/chrony.conf 文件
allow 192.168.0.0/16 //修改为自己集群网段
客户端修改 vim etc/chrony.conf 文件
pool hadoop01 iburst // 同步时间的机器地址
启动chrony
systemctl start chronyd
设置开机自启动 (客户端也设置开机自动会自动同步)
systemctl enable chronyd
手动同步时间
chronyc tracking