大数据集群搭建（连载版）--Hadoop高可用部署

最新推荐文章于 2024-05-13 16:27:29 发布

.Feng_

最新推荐文章于 2024-05-13 16:27:29 发布

阅读量139

点赞数

文章标签： hadoop 大数据 hdfs

本文链接：https://blog.csdn.net/weixin_40657218/article/details/130393701

版权

结合上篇文章【大数据集群搭建（连载版）–集群规划】继续来搭建Hadoop，其中包含HDFS、Yarn的部署

一、HDFS集群部署

1.1 下载并解压安装包到/opt/soft目录下

cd /opt/soft 
wget $HADOOP_DOWNLOAD_URL
tar -zxvf hadoop-2.8.5.tar.gz
mv hadoop-2.8.5 hadoop
# ---------------------------------
# 添加环境变量
cat > /etc/profile.d/hadoop.sh << EOF
export HADOOP_HOME=/opt/soft/hadoop
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
EOF

更改配置文件，默认配置文件使用/opt/soft/hadoop/etc/hadoop

`core-site.xml`

<configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://ns1</value>
  </property>
  <property>
    <name>dfs.journalnode.edits.dir</name>
    <value>/data/bigdata/hadoop/journalnode</value>
  </property>
  <property>
    <name>hadoop.tmp.dir</name>
    <value>/data/bigdata/hadoop/tmp</value>
  </property>
  <property>
    <name>fs.trash.interval</name>
    <value>1440</value>
  </property>
  <property>
    <name>io.file.buffer.size</name>
    <value>65536</value>
  </property>
  <property>
    <name>ha.zookeeper.quorum</name>
    <value>mapper-node1:2181,mapper-node2:2181,mapper-node3:2181</value>
  </property>
<property>
    <name>hadoop.proxyuser.hdfs.hosts</name>
    <value>*</value>
</property>
<property>
    <name>hadoop.proxyuser.hdfs.groups</name>
    <value>*</value>
</property>
<property>
    <name>hadoop.proxyuser.root.hosts</name>
    <value>*</value>
</property>
<property>
    <name>hadoop.proxyuser.root.groups</name>
    <value>*</value>
</property>
<property>
    <name>hadoop.proxyuser.hive.hosts</name>
    <value>*</value>
</property>
<property>
    <name>hadoop.proxyuser.hive.groups</name>
    <value>*</value>
</property>
</configuration>

`hdfs-site.xml`

<configuration>
  <property>
    <name>dfs.replication</name>
    <value>3</value>
  </property>
  <property>
    <name>dfs.namenode.name.dir</name>
    <value>/data/bigdata/hadoop/name</value>
  </property>
  <property>
    <name>dfs.blocksize</name>
    <value>67108864</value>
  </property>
  <property>
    <name>dfs.datanode.data.dir</name>
    <value>/data/bigdata/hadoop/data</value>
  </property>
  <property>
    <name>dfs.namenode.checkpoint.dir</name>
    <value>/data/bigdata/hadoop/checkpoint</value>
  </property>
  <property>
    <name>dfs.namenode.handler.count</name>
    <value>10</value>
  </property>
  <property>
    <name>dfs.datanode.handler.count</name>
    <value>10</value>
  </property>
  <property>
    <name>dfs.nameservices</name>
    <value>ns1</value>
  </property>
  <property>
    <name>dfs.ha.namenodes.ns1</name>
    <value>nn1,nn2</value>
  </property>
  <property>
    <name>dfs.namenode.rpc-address.ns1.nn1</name>
    <value>mapper-node1:9000</value>
  </property>
  <property>
    <name>dfs.namenode.rpc-address.ns1.nn2</name>
    <value>mapper-node2:9000</value>
  </property>
  <property>
    <name>dfs.namenode.http-address.ns1.nn1</name>
    <value>mapper-node1:50070</value>
  </property>
  <property>
    <name>dfs.namenode.http-address.ns1.nn2</name>
    <value>mapper-node2:50070</value>
  </property>
  <property>
    <name>dfs.namenode.shared.edits.dir</name>
    <value>qjournal://mapper-node1:8485;mapper-node2:8485;mapper-node3:8485;/ns1</value>
  </property>
  <property>
    <name>dfs.client.failover.proxy.provider.ns1</name>
    <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
  </property>
  <property>
    <name>dfs.ha.automatic-failover.enabled</name>
    <value>true</value>
  </property>
  <property>
    <name>dfs.ha.fencing.methods</name>
    <value>shell(/bin/true)</value>
  </property>
  <property>
    <name>dfs.permissions.enabled</name>
    <value>true</value>
  </property>
  <property>
    <name>dfs.permissions.superusergroup</name>
    <value>hadoop</value>
  </property>
</configuration>

`hadoop-env.sh`

export JAVA_HOME=${JAVA_HOME}
export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/etc/hadoop"}
for f in $HADOOP_HOME/contrib/capacity-scheduler/*.jar; do
  if [ "$HADOOP_CLASSPATH" ]; then
    export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$f
  else
    export HADOOP_CLASSPATH=$f
  fi
done
export HADOOP_HEAPSIZE=2048
# export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true -Dsun.security.krb5.debug=true -Dsun.security.spnego.debug"
export HADOOP_OPTS="$HADOOP_OPTS -Djava.net.preferIPv4Stack=true"
export HADOOP_NAMENODE_OPTS="-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_NAMENODE_OPTS"
export HADOOP_DATANODE_OPTS="-Dhadoop.security.logger=ERROR,RFAS $HADOOP_DATANODE_OPTS"
export HADOOP_SECONDARYNAMENODE_OPTS="-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_SECONDARYNAMENODE_OPTS"
export HADOOP_NFS3_OPTS="$HADOOP_NFS3_OPTS"
export HADOOP_PORTMAP_OPTS="-Xmx512m $HADOOP_PORTMAP_OPTS"

# The following applies to multiple commands (fs, dfs, fsck, distcp etc)
export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS"
# set heap args when HADOOP_HEAPSIZE is empty
if [ "$HADOOP_HEAPSIZE" = "" ]; then
  export HADOOP_CLIENT_OPTS="-Xmx1024m $HADOOP_CLIENT_OPTS"
fi
export HADOOP_SECURE_DN_USER=${HADOOP_SECURE_DN_USER}
export HADOOP_LOG_DIR=/data/bigdata/logs/hadoop/hdfs
export HADOOP_PID_DIR=${HADOOP_PID_DIR}
export HADOOP_SECURE_DN_PID_DIR=${HADOOP_PID_DIR}
export HADOOP_IDENT_STRING=$USER

1.2 上述配置更改完成后，拷贝部署目录到其他节点

scp /opt/soft/hadoop root@mapper-node2:/opt/soft
scp /opt/soft/hadoop root@mapper-node3:/opt/soft

格式化zkfc、namende，并启动对应角色服务

hdfs zkfc -formatZK
hdfs namenode -format # mapper-node1节点上执行
#启动mapper-node1节点服务
hadoop-daemon.sh start zkfc
hadoop-daemon.sh start journalnode
hadoop-daemon.sh start datanode
hadoop-daemon.sh start namenode
# mapper-node2节点操作
hdfs namenode -bootstrapStandby # mapper-node2节点执行，引导standby节点同步active节点数据

依次在mapper-node2、mapper-node3使用hadoop-daemon.sh启动对应节点的角色服务

启动完成后进行验证，可以通过访问mapper-node1:50070端口查看是否已正常启动

二、验证HDFS服务

# 操作文件是否正常
hdfs dfs -mkdir -p /tmp/test
hdfs dfs -put a.txt /tmp/test
hdfs dfs -get /tmp/test/a.txt
# 查看hdfs集群信息
hdfs dfsadmin -report
# 查看active、standby节点是否正常
hdfs haadmin -getAllServiceState

在这里插入图片描述

三、验证HDFS高可用

获取到active状态的节点

hdfs haadmin -getAllServiceState # 获取到mapper-node2
# 停止active状态的节点namenode服务，停止过程中对zkfc服务日志进行监控
hadoop-daemon.sh stop namenode
# 停止active状态的namenode后，zkfc会检测到active namenode退出并尝试连接，最终将standby状态namenode转换为active，日志输出如下
# Successfully transitioned NameNode at mapper-node1/192.168.30.183:9000 to active state

# 再次进行上述的文件操作，验证是否能够正常使用
hdfs dfs -mkdir -p /tmp/test
hdfs dfs -put a.txt /tmp/test
hdfs dfs -get /tmp/test/a.txt

四、Yarn集群部署

配置文件

`yarn-site.xml`

<configuration>
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
  <property>
    <name>yarn.resourcemanager.ha.enabled</name>
    <value>true</value>
  </property>
  <property>
    <name>yarn.resourcemanager.cluster-id</name>
    <value>cluster1</value>
  </property>
  <property>
    <name>yarn.resourcemanager.ha.rm-ids</name>
    <value>rm1,rm2</value>
  </property>
  <property>
    <name>yarn.resourcemanager.hostname.rm1</name>
    <value>mapper-node1</value>
  </property>
  <property>
    <name>yarn.resourcemanager.hostname.rm2</name>
    <value>mapper-node2</value>
  </property>
  <property>
    <name>yarn.resourcemanager.webapp.address.rm1</name>
    <value>mapper-node1:8088</value>
  </property>
  <property>
    <name>yarn.resourcemanager.webapp.address.rm2</name>
    <value>mapper-node2:8088</value>
  </property>
  <property>
    <name>yarn.resourcemanager.zk-address</name>
    <value>mapper-node1:2181,mapper-node2:2181,mapper-node3:2181</value>
  </property>
</configuration>

`yarn-env.sh`

export HADOOP_YARN_USER=${HADOOP_YARN_USER:-yarn}
export YARN_CONF_DIR="${YARN_CONF_DIR:-$HADOOP_YARN_HOME/conf}"
if [ "$JAVA_HOME" != "" ]; then
  JAVA_HOME=$JAVA_HOME
fi
  
if [ "$JAVA_HOME" = "" ]; then
  echo "Error: JAVA_HOME is not set."
  exit 1
fi

JAVA=$JAVA_HOME/bin/java
JAVA_HEAP_MAX=-Xmx1000m 
YARN_HEAPSIZE=2048
if [ "$YARN_HEAPSIZE" != "" ]; then
  JAVA_HEAP_MAX="-Xmx""$YARN_HEAPSIZE""m"
fi
export YARN_RESOURCEMANAGER_HEAPSIZE=2048
export YARN_NODEMANAGER_HEAPSIZE=10240
IFS=
if [ "$YARN_LOG_DIR" = "" ]; then
  YARN_LOG_DIR="/data/bigdata/logs/hadoop/yarn"
fi
if [ "$YARN_LOGFILE" = "" ]; then
  YARN_LOGFILE='yarn.log'
fi
if [ "$YARN_POLICYFILE" = "" ]; then
  YARN_POLICYFILE="hadoop-policy.xml"
fi
unset IFS
YARN_OPTS="$YARN_OPTS -Dhadoop.log.dir=$YARN_LOG_DIR"
YARN_OPTS="$YARN_OPTS -Dyarn.log.dir=$YARN_LOG_DIR"
YARN_OPTS="$YARN_OPTS -Dhadoop.log.file=$YARN_LOGFILE"
YARN_OPTS="$YARN_OPTS -Dyarn.log.file=$YARN_LOGFILE"
YARN_OPTS="$YARN_OPTS -Dyarn.home.dir=$YARN_COMMON_HOME"
YARN_OPTS="$YARN_OPTS -Dyarn.id.str=$YARN_IDENT_STRING"
YARN_OPTS="$YARN_OPTS -Dhadoop.root.logger=${YARN_ROOT_LOGGER:-INFO,console}"
YARN_OPTS="$YARN_OPTS -Dyarn.root.logger=${YARN_ROOT_LOGGER:-INFO,console}"
if [ "x$JAVA_LIBRARY_PATH" != "x" ]; then
  YARN_OPTS="$YARN_OPTS -Djava.library.path=$JAVA_LIBRARY_PATH"
fi  
YARN_OPTS="$YARN_OPTS -Dyarn.policy.file=$YARN_POLICYFILE"

`mapred-site.xml`

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
</configuration>

`mapred-env.sh`

export HADOOP_JOB_HISTORYSERVER_HEAPSIZE=1024
export HADOOP_MAPRED_ROOT_LOGGER=INFO,RFA
export HADOOP_MAPRED_LOG_DIR="/data/bigdata/logs/hadoop/mapred"

每个节点的yarn相关配置文件更改完成后，启动yarn

yarn-daemon.sh start resourcemanager
yarn-daemon.sh start nodemanager
mr-jobhistory-daemon.sh start historyserver

验证yarn服务是否正常使用

# 提交Pi mapreduce任务
hadoop jar /opt/soft/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.5.jar pi 1 2

在这里插入图片描述

界面显示任务执行正常，并在终端显示Estimated value of Pi is 4.00000000000000000000，即表示yanr服务使用正常

五、验证yarn集群高可用

# 停止yarn处于active状态的resourcemanager
yarn rmadmin -getAllServiceState
# mapper-node1:8033                                  standby   
# mapper-node2:8033                                  active  
ssh mapper-node2 'yarn-daemon.sh stop resourcemanager'

停止后，standby节点会检测到active的节点宕机后，切换为active状态，并将nodemanager节点注册到新的resourcemanager