大数据平台配置目录分离与Hadoop升级3.3.1版本

最新推荐文章于 2023-07-14 20:00:33 发布

若苗瞬

最新推荐文章于 2023-07-14 20:00:33 发布

阅读量2.2k

点赞数

分类专栏：大数据 Linux 文章标签： hadoop big data hdfs spark 升级

本文链接：https://blog.csdn.net/ddrfan/article/details/121990270

版权

Linux 同时被 2 个专栏收录

34 篇文章 2 订阅

订阅专栏

大数据

14 篇文章 1 订阅

订阅专栏

文章目录

（一）配置目录分离
（二）滚动升级Hadoop版本

（一）配置目录分离

之前在写：🔗《从零开始学习大数据平台（Episode 1）》时，搭建了测试的大数据平台。
但是按照那时的配置，升级Hadoop，Spark等软件时容易弄丢配置，稍显不便。
所以最好将配置目录（Hadoop还包括HDFS数据目录）和软件目录分开。

1.1 Hadoop

指定：HADOOP_CONF_DIR，到软件目录以外。

$ vim .bashrc 
......
#hadoop
export HADOOP_HOME=/home/ac/hadoop_bin
export HADOOP_CONF_DIR=/home/ac/hadoop_conf #就是这句指定配置所在目录
export PATH=$PATH:${HADOOP_HOME}/sbin
export PATH=$PATH:${HADOOP_HOME}/bin
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"
export LD_LIBRARY_PATH=$HADOOP_HOME/lib/native
......

因为有HDFS存储数据，所以数据目录也分离。
指定：dfs.namenode.name.dir，dfs.namenode.name.dir，到软件目录以外。

$ vim $HADOOP_CONF_DIR/hdfs-site.xml

<configuration>
	......
	<property> <!--目录无需预先存在，会自动创建-->
		<name>dfs.namenode.name.dir</name>
		<value>/home/ac/hadoop_data/dfs/name</value><!---->
	</property>
	<property>  <!--目录无需预先存在，会自动创建-->
		<name>dfs.datanode.data.dir</name>
		<value>/home/ac/hadoop_data/dfs/data</value>
	</property>
	......
</configuration>

1.2 Spark

指定：SPARK_CONF_DIR，到软件目录以外。

$ vim .bashrc 
......
#Spark
export SPARK_HOME=/home/ac/spark
export SPARK_CONF_DIR=/home/ac/spark_conf
export PATH=$PATH:${SPARK_HOME}/bin
......

1.3 其他软件

比如：HBASE_CONF_DIR，等等类似……

（二）滚动升级Hadoop版本

2.1 停机升级

将配置目录和数据目录都分离后，更新软件版本只需要删除原有软件目录，将新版本解压到原来的目录名就可以了（直接替换）。

当然这样只适合小版本升级，如果升级跨度太大，还是仔细看官方文档吧。

升级完以后可以用指令看看版本号：

$ hadoop version

Hadoop 3.3.1
Source code repository https://github.com/apache/hadoop.git -r a3b9c37a397ad4188041dd80621bdeefc46885f2
Compiled by ubuntu on 2021-06-15T05:13Z
Compiled with protoc 3.7.1
From source with checksum 88a4ddb2299aca054416d6b7f81ca55
This command was run using /home/ac/hadoop_bin/share/hadoop/common/hadoop-common-3.3.1.jar

我这里之前是3.3.0。
如果软件目录替换后直接启动dfs则会发现NameNode无法启动。

$ start-dfs.sh
......

$ jps
1111 SecondaryNameNode
2222 Jps
3333 DataNode

需要先停下集群，启动NameNode带上这个参数 “-rollingUpgrade started” .

$ hdfs --daemon start namenode -rollingUpgrade started

执行一次这个以后就可以正常启动了。

2.1 不停机的情况下升级

官方文档：

In an HA cluster, there are two or more NameNodes (NNs), many DataNodes (DNs), a few JournalNodes (JNs) and a few ZooKeeperNodes (ZKNs). JNs is relatively stable and does not require upgrade when upgrading HDFS in most of the cases. In the rolling upgrade procedure described here, only NNs and DNs are considered but JNs and ZKNs are not. Upgrading JNs and ZKNs may incur cluster downtime.

Upgrading Non-Federated Clusters
Suppose there are two namenodes NN1 and NN2, where NN1 and NN2 are respectively in active and standby states. The following are the steps for upgrading an HA cluster:

Prepare Rolling Upgrade
Run “hdfs dfsadmin -rollingUpgrade prepare” to create a fsimage for rollback.
Run “hdfs dfsadmin -rollingUpgrade query” to check the status of the rollback image. Wait and re-run the command until the “Proceed with rolling upgrade” message is shown.
Upgrade Active and Standby NNs
Shutdown and upgrade NN2.
Start NN2 as standby with the “-rollingUpgrade started” option.
Failover from NN1 to NN2 so that NN2 becomes active and NN1 becomes standby.
Shutdown and upgrade NN1.
Start NN1 as standby with the “-rollingUpgrade started” option.
Upgrade DNs
Choose a small subset of datanodes (e.g. all datanodes under a particular rack).
Run “hdfs dfsadmin -shutdownDatanode <DATANODE_HOST:IPC_PORT> upgrade” to shutdown one of the chosen datanodes.
Run “hdfs dfsadmin -getDatanodeInfo <DATANODE_HOST:IPC_PORT>” to check and wait for the datanode to shutdown.
Upgrade and restart the datanode.
Perform the above steps for all the chosen datanodes in the subset in parallel.
Repeat the above steps until all datanodes in the cluster are upgraded.
Finalize Rolling Upgrade
Run “hdfs dfsadmin -rollingUpgrade finalize” to finalize the rolling upgrade.

Upgrading Federated Clusters
In a federated cluster, there are multiple namespaces and a pair of active and standby NNs for each namespace. The procedure for upgrading a federated cluster is similar to upgrading a non-federated cluster except that Step 1 and Step 4 are performed on each namespace and Step 2 is performed on each pair of active and standby NNs, i.e.

Prepare Rolling Upgrade for Each Namespace
Upgrade Active and Standby NN pairs for Each Namespace
Upgrade DNs
Finalize Rolling Upgrade for Each Namespace

文档翻译：

‎在高可用性集群中，有两个或多个‎‎NameNodes （NN）、‎‎许多‎‎DataNode （DN）、‎‎几个‎‎JournalNode （JN）‎‎和几个‎‎ZooKeeperNodes （ZKN）。‎‎JNs‎‎相对稳定，在大多数情况下，升级 HDFS 时不需要升级。在此处介绍的滚动升级过程中，仅考虑‎‎NN‎‎和‎‎DN，‎‎但不考虑‎‎NN‎‎和‎‎ZKN。‎‎升级‎‎JN‎‎和‎‎ZKN‎‎可能会导致群集停机。‎

‎升级非联合群集‎
‎假设有两个名称节点‎‎NN1‎‎和‎‎NN2，‎‎其中‎‎NN1‎‎和‎‎NN2‎‎分别处于活动和待机状态。以下是升级 HA 集群的步骤：‎

‎准备滚动升级‎
‎运行 “‎‎hdfs dfsadmin -rollingUpgrade prepare‎‎” 来创建一个用于回滚的文件系统镜像。‎
‎运行"‎‎hdfs dfsadmin -rollingUpgrade query"‎‎来检查回滚映像的状态。等待并重新运行该命令，直到显示"‎‎继续滚动升级‎‎"消息。‎
‎升级活动和备用‎‎NN‎
‎关机并升级‎‎NN2‎‎。‎
‎使用‎‎"-rollingUpgrade started"‎‎选项启动‎‎NN2‎‎作为备用数据库。‎
‎从‎‎NN1‎‎故障转移到‎‎NN2，‎‎以便‎‎NN2‎‎变为活动状态‎‎，NN1‎‎变为备用。‎
‎关机并升级‎‎NN1‎‎。‎
‎使用‎‎"-rollingUpgrade started"选项将‎‎NN1‎‎作为备用启动。‎
‎升级‎‎DN们
‎选择一小部分数据节点（例如，特定机架下的所有数据节点）。‎
‎运行"‎‎hdfs dfsadmin -shutdownDatanode <DATANODE_HOST：IPC_PORT> upgrade"‎‎以关闭其中一个选定的数据节点。‎
‎运行 “‎‎hdfs dfsadmin -getDatanodeInfo <DATANODE_HOST：IPC_PORT>‎‎” 进行检查并等待数据节点关闭。‎
‎升级并重新启动数据节点。‎
‎对子集中所有选定的数据节点并行执行上述步骤。‎
‎重复上述步骤，直到集群中的所有数据节点都升级。‎
‎完成滚动升级‎
‎运行"‎‎hdfs dfsadmin -rollingUpgrade finalize"‎‎以完成滚动升级。‎