## 一、基础配置:
1.查看主机名: cat /etc/hostname
2.防火墙状态: firewall-cmd --state
3.关闭防火墙:systemctl stop firewalld.service 停止firewalld服务
systemctl disable firewalld.service 禁用friewalld服务
4. SS面免密登录:(node1执行->node1|node2|node3) ssh-keygen #4个回车 生成 公钥、私钥 ssh-copy-id node1、ssh-copy-id node2、ssh-copy-id node3
5. 时间同步: ntpdate ntp5.aliyun.com
6. 创建统一工作目录(3台机器)
mkdir -p /export/server/ #软件安装路径
mkdir -p /export/data/ #数据存储路径
mkdir -p /export/software/ #安装包存放路径
编译环境软件安装目录
mkdir -p /export/server
## 安装配置JDK 1.8 (三台机器都要做)
解压安装包
tar zxvf jdk-8u65-linux-x64.tar.gz
#配置环境变量
vim /etc/profile
添加到文件
export JAVA_HOME=/export/server/jdk1.8.0_241
export PATH=$PATH:$JAVA_HOME/bin
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
加载生效:
source /etc/profile
#验证是否安装成功
java -version
java version "1.8.0_241"
Java(TM) SE Runtime Environment (build 1.8.0_241-b07)
Java HotSpot(TM) 64-Bit Server VM (build 25.241-b07, mixed mode)
其他两台机器直接使用命令拷贝
scp -r /export/server/jdk1.8.0_241/ root@node2:/export/server/
scp -r /export/server/jdk1.8.0_241/ root@node3:/export/server/
拷贝配置文件:scp /etc/profile root@node1:/etc/
scp /etc/profile root@node2:/etc/
scp /etc/profile root@node3:/etc/
三台机器统一验证:source /etc/profile
java -version
- 集群规划
| 主机 | 角色 |
| ----- | -------------------- |
| node1 | NN DN RM NM |
| node2 | SNN DN NM |
| node3 | DN NM |
> 3台机器都需要操作
----
hosts映射
vim /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.88.151 node1.itcast.cn node1
192.168.88.152 node2.itcast.cn node2
192.168.88.153 node3.itcast.cn node3
## - 上传Hadoop安装包到node1 /export/server
hadoop-3.3.0-Centos7-64-with-snappy.tar.gz
tar zxvf hadoop-3.3.0-Centos7-64-with-snappy.tar.gz
cd /export/server/hadoop-3.3.0/etc/hadoop下面进行配置
- 修改配置文件(配置文件路径 hadoop-3.3.0/etc/hadoop)
hadoop-env.sh
#文件最后添加
export JAVA_HOME=/export/server/jdk1.8.0_241
export HDFS_NAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root
core-site.xml
fs.defaultFS hdfs://node1:8020 hadoop.tmp.dir /export/data/hadoop-3.3.0 hadoop.http.staticuser.user root hadoop.proxyuser.root.hosts * hadoop.proxyuser.root.groups * fs.trash.interval 1440hdfs-site.xml
dfs.namenode.secondary.http-address node2:9868mapred-site.xml
mapreduce.framework.name yarn mapreduce.jobhistory.address node1:10020 mapreduce.jobhistory.webapp.address node1:19888 yarn.app.mapreduce.am.env HADOOP_MAPRED_HOME=${HADOOP_HOME} mapreduce.map.env HADOOP_MAPRED_HOME=${HADOOP_HOME} mapreduce.reduce.env HADOOP_MAPRED_HOME=${HADOOP_HOME}yarn-site.xml
yarn.resourcemanager.hostname node1 yarn.nodemanager.aux-services mapreduce_shuffle yarn.nodemanager.pmem-check-enabled false yarn.nodemanager.vmem-check-enabled false yarn.log-aggregation-enable true yarn.log.server.url http://node1:19888/jobhistory/logs yarn.log-aggregation.retain-seconds 604800workers
node1.itcast.cn
node2.itcast.cn
node3.itcast.cn
分发同步hadoop安装包,在其他两个机器上
cd /export/server
scp -r hadoop-3.3.0 root@node2: P W D s c p − r h a d o o p − 3.3.0 r o o t @ n o d e 3 : PWD scp -r hadoop-3.3.0 root@node3: PWDscp−rhadoop−3.3.0root@node3:PWD
将hadoop添加到环境变量(3台机器)
vim /etc/profile
export HADOOP_HOME=/export/server/hadoop-3.3.0
export PATH=
P
A
T
H
:
PATH:
PATH:HADOOP_HOME/bin:$HADOOP_HOME/sbin
拷贝其他两台:scp /etc/profile node2:/etc/
scp /etc/profile node3:/etc/
三台都要执行:source /etc/profile
测试:hadoop
#别忘了scp给其他两台机器哦
- Hadoop集群启动 - format只能进行一次 后续不再需要; 3. 如果多次format除了造成数据丢失外,还会导致 hdfs集群主从角色之间互不识别。通过删除所有机 器hadoop.tmp.dir目录重新format解决 - (首次启动)格式化namenode
hdfs namenode -format
启动
HDFS集群
start-dfs.sh
stop-dfs.sh
YARN集群
start-yarn.sh
stop-yarn.sh
Hadoop集群
start-all.sh
stop-all.sh
查看进程:jps
查看是否启动成功,查看日志:
cd /export/server/hadoop-3.3.0/logs
- Web UI页面
- HDFS集群:http://node1:9870/
- YARN集群:http://node1:8088/