Hadoop、yarn、zookeeper、kafka、Elastic、mysql、spark、flume、logstash、tomcat、flink等开源组件运维手册

LogManager 开源组件运维手册

Hadoop 日常运维操作

hdfs

生产环境hadoop为三台服务器组成的集群，统一安装配置，版本号为2.7.7
部署路径：/opt/hadoop

启动用户：hadoop

配置文件：

• /opt/hadoop/config/hdfs-site.xml
• /opt/hadoop/config/core-site.xml

hadoopy运行环境变量配置文件：

• hadoop-env.sh
• journalnode.env
• datanode.env
• namenode.env

hadoop系统服务配置文件：

• zkfc.service
• journalnode.service
• namenode.service
• datanode.service

存储快照文件snapshot的目录：/data/hadoop/data
运行日志输出目录：/data/hadoop/logs

Hadoop运行正常时会有下列端口

• 50010 HDFS datanode 服务端口，用于数据传输
• 50075 HDFS namenode http服务的端口
• 50020 HDFS namenode ipc服务的端口
• 50070 HDFS namenode http服务的端口,active namenode中启动
• 8020 HDFS namenode 接收Client连接的RPC端口，用于获取文件系统metadata信息。

[hadoop@hostname-2 ~]$ netstat -ln|egrep "(50010|50075|50475|50020|50070|50470|8020|8019)"
tcp        0      0 172.0.0.2:50070      0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:50010           0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:50075           0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:50020           0.0.0.0:*               LISTEN

Hadoop官方参考文档

hadoop组件启动与停止命令

# 启动
sudo systemctl start namenode.service
sudo systemctl start datanode.service
sudo systemctl start journalnode.service
# 停止
sudo systemctl stop namenode.servicec
sudo systemctl stop datanode.servicec
sudo systemctl stop journalnode.service
# 查看启动状态
sudo systemctl status namenode.service
sudo systemctl status datanode.service
sudo systemctl status journalnode.service
# 开机时自动自动启动
sudo systemctl enable namenode.service
sudo systemctl enable datanode.service
sudo systemctl enable journalnode.service

查看hadoop组件运行状态参数

# 查看当前namenode节点
[hadoop@hostname-2 ~]$ hdfs getconf -namenodes
hostname-3 hostname-2
# 查看集群datanode节点配置文件
[hadoop@hostname-2 ~]$ hdfs getconf -includeFile
/opt/hadoop/config/slaves
# 查看datanode rpc端口
[hadoop@hostname-2 ~]$ hdfs getconf -nnRpcAddresses
hostname-3:9000
hostname-2:9000

hdfs getconf -confKey [key]
# dfsadmin
[hadoop@hostname-2 ~]$  hdfs dfsadmin -report -live
Configured Capacity: 422346469376 (393.34 GB)
Present Capacity: 317439557632 (295.64 GB)
DFS Remaining: 315510235136 (293.84 GB)
...

-------------------------------------------------
Live datanodes (3):

Name: 172.0.0.3:50010 (hostname-3)
Hostname: iZ8vbacq1jxnabyu7992d1Z
Decommission Status : Normal
...


Name: 172.0.0.1:50010 (hostname-1)
Hostname: iZ8vb2s7y1j8fqmqbmufz9Z
Decommission Status : Normal
...


Name: 172.0.0.2:50010 (iZ8vbacq1jxnabyu7992d2Z)
Hostname: iZ8vbacq1jxnabyu7992d2Z
Decommission Status : Normal
...


# haadmin 查看namenode主节点
[hadoop@hostname-2 ~]$ hdfs haadmin -getServiceState hostname-2
active

yarn

启动用户： hadoop

配置文件：

• /opt/hadoop/config/yarn-site.xml

环境变量文件：

• yarn.env
• zkfc.env

系统服务配置文件：

• yarn-nm.service
• yarn-rm.service
• zkfc.service

hadoop Yarn组件运行正常时会有下列端口

• 8030 YARN ResourceManager scheduler组件的IPC端口
• 8031 YARN ResourceManager RPC
• 8032 YARN ResourceManager RM的applications manager(ASM)端口
• 8033 YARN ResourceManager IPC
• 8088 YARN ResourceManager http服务