NetApp Ontap系统降级（重新安装）流程

1. 为何需要降级Ontap

需要降级Ontap的场景99%以上无外乎以下几种，下面我结合这几年来碰到的真实案例分享下：
✔新版本有恶性Bug并影响生产或管理需要回退：
版本问题说个题外话，据内部消息，Ontap9的奇数版本比偶数版本稳定，无论是升降级Ontap优先考虑3,5,7,9的版本，另外确实9.7过后的新WebUI极其难用，还有多个显示上的bug，直接导致了版本回退的必要性
✔源端新Ontap需要Snapmirror至老NetApp平台：
Snapmirror的基本要求之一是目的端的Ontap版本必须大于等于源端，而有一个很常见的场景就是用户新购NetApp，源NetApp利旧作为Snapmirror的目的端，利旧NetApp往往硬件平台较老，能支持到的最高版本也低于新NetApp的出厂版本，因此必须降级
✔多台NetApp组建Cluster集群：
从Ontap9.3开始，NetApp支持不同版本的Ontap平台组建cluster

cluster join -allow-mixed-version-join

cluster add-node -allow-mixed-version-join

但同时跨版本可能造成的新功能上的差异，对于用到新Ontap功能的SVM等可能无法被老版本Node接管，因此除非特殊情况，官方也不建议使用混合版本做Cluster，以下摘自官方KB：

ONTAP is designed to remain operational and serving data during a mixed version state, however, it is not recommended to remain in mixed version state for longer than the time it takes to upgrade the entire cluster. It is also highly discouraged to make any configuration changes to the cluster while the cluster is in a mixed version state.
The cluster can also enter mixed version state when a storage controller with a newer version of ONTAP is joined to a cluster that is older version. If this occurs, then upgrade the rest of the cluster to the newer ONTAP version.

本案例正好为原AFF8020（Ontap9.3）与新购FAS2750（Ontap9.9）组成集群，由于8020最高支持到Ontap9.7，因此需要将Ontap9.9降级回9.7

以下重装流程不光适用于控制器有Ontap系统的情况下降级，同时也适用于无系统的空控制器新安装Ontap，安装流程不复杂，关键是评估安装的版本及必要性

2. 重装流程

通过Console链接至控制器，启动控制器并在如下界面按Ctrl+C进入LOADER模式

Boot Loader version 6.0.12 
Copyright (C) 2000-2003 Broadcom Corporation.
Portions Copyright (C) 2002-2021 NetApp, Inc. All Rights Reserved.



Starting AUTOBOOT press Ctrl-C to abort...

Autoboot of PRIMARY image aborted by user.

LOADER-A> 

ifconfig配置e0M临时IP，完成后printenv确认配置

LOADER-A> ifconfig e0M -addr=10.254.254.200 -mask=255.255.255.0 -gw=10.254.254.254
Network interface setup for IPv4
Device e0M: 
    hwaddr D0-39-EA-37-E3-2D, ipaddr 10.254.254.200, mask 255.255.255.0
    gateway 10.254.254.254, nameserver not set
    speed 1000 full duplex, loopback not supported
LOADER-A> 
LOADER-A> printenv
NET_DEVICE           e0M
NET_GATEWAY          10.254.254.254
NET_IPADDR           10.254.254.200
NET_NAMESERVER       0.0.0.0
NET_NETMASK          255.255.255.0

电脑链接至e0M，并将准备好的Ontap镜像拖入http服务器（推荐HFS），复制镜像路径
此处需要注意：
1.多网络适配器的情况下选择对应的正确网卡（Menu→IP address）
2.关闭防火墙，确保网络通信正常

使用Netboot启动镜像（若已有Ontap此步可忽略）

LOADER-A> netboot http://10.254.254.100/97p15_q_nodar_image.tgz
Downloading 10.254.254.100/97p15_q_nodar_image.tgz ..
Extracting archive 10.254.254.100/97p15_q_nodar_image.tgz ..
Loading netboot/kernel:0x200000/1092257 0x30b000/10941920 0xf7b000/3783864 0x1316cb8/13717000 0x200200/1016 Entry at 0xffffffff8030b000
Found netboot/SB_XXII/platform.rc
Loading netboot/platform.ko:0x202c000/4429408 0x2465660/596248 
Loading netboot/rootfs.img:0x24f7000/401075712 
Loading netboot/platfs3g.img:0x1a376000/232459264 
Closing network.
Starting program at 0xffffffff8030b000
NetApp Data ONTAP 9.7P15

在Ontap启动过程中Ctrl+C打断，并选择7重新安装Ontap系统

Copyright (C) 1992-2021 NetApp.
All rights reserved.
*******************************
*                             *
* Press Ctrl-C for Boot Menu. *
*                             *
*******************************
cryptomod_fips: Executing Crypto FIPS Self Tests.
cryptomod_fips: Crypto FIPS self-test: 'CPU COMPATIBILITY' passed.
cryptomod_fips: Crypto FIPS self-test: 'AES-128 ECB, AES-256 ECB' passed.
cryptomod_fips: Crypto FIPS self-test: 'AES-128 CBC, AES-256 CBC' passed.
cryptomod_fips: Crypto FIPS self-test: 'AES-128 GCM, AES-256 GCM' passed.
cryptomod_fips: Crypto FIPS self-test: 'AES-128 CCM' passed.
cryptomod_fips: Crypto FIPS self-test: 'CTR_DRBG' passed.
cryptomod_fips: Crypto FIPS self-test: 'KDF' passed.
cryptomod_fips: Crypto FIPS self-test: 'SHA1, SHA256, SHA512' passed.
cryptomod_fips: Crypto FIPS self-test: 'HMAC-SHA1, HMAC-SHA256, HMAC-SHA512' passed.
cryptomod_fips: Crypto FIPS self-test: 'PBKDF2' passed.
cryptomod_fips: Crypto FIPS self-test: 'Self-integrity' passed.
Thu Oct 14 22:54:38 2021 [nv2flash.restage.progress:NOTICE]: ReStage is not needed because the flash has no data.
open: Read-only file system


Oct 14 22:55:21 Power outage protection flash de-staging: 52 cycles
The Boot Menu will be presented because an alternate boot method was specified.

Please choose one of the following:

(1) Normal Boot.
(2) Boot without /etc/rc.
(3) Change password.
(4) Clean configuration and initialize all disks.
(5) Maintenance mode boot.
(6) Update flash from backup config.
(7) Install new software first.
(8) Reboot node.
(9) Configure Advanced Drive Partitioning.
Selection (1-9)? 7

This procedure will replace existing Data ONTAP software with new software.
Do you want to continue? {y|n} y

What is the URL for the package? http://10.254.254.100/97p15_q_nodar_image.tgz
What is the user name on "10.254.254.100", if any? 

Checking network link... -/|\-/|\-/|\-/|\-/|\-/|\-/|\-/|\-/|\-success.
Checking route to host "10.254.254.100"... success.
Attempting to reach 10.254.254.100... success.
Looking up URL "http://10.254.254.100/97p15_q_nodar_image.tgz"... success.
......
Installation complete.

安装完成后使用新Ontap系统启动

**************************************************************
*             Restore Backup Configuration                   *
*  This procedure only applies to storage controllers that   *
*  are configured as an HA pair.                             *
*                                                            *
*  Choose Yes to restore the 'varfs' backup configuration    *
*  from a SSH server.  Refer to the Boot Device Replacement  *
*  guide for more details.                                   *
*  Choose No to skip the back up recovery and return to the  *
*  boot menu.                                                *
**************************************************************

Do you want to restore the backup configuration
now? {y|n} n

The node must be rebooted to start using the newly installed software. Do you
want to reboot now? {y|n} y


Rebooting...

安装完默认导航到boot menu，第一次需要选4进行清配置及磁盘初始化

Please choose one of the following:

(1) Normal Boot.
(2) Boot without /etc/rc.
(3) Change password.
(4) Clean configuration and initialize all disks.
(5) Maintenance mode boot.
(6) Update flash from backup config.
(7) Install new software first.
(8) Reboot node.
(9) Configure Advanced Drive Partitioning.
Selection (1-9)? 4
Pensando Offload Driver, ver 1.0.1-E-31
Pensando Ethernet NIC Driver, ver: 1.0.1-E-39
ionic_rdma ver 1.0.1-E-39 : Pensando RoCE HCA driver
***OS2SP configured successfully***Oct 14 23:01:48 [localhost:cf.ic.sbb:notice]: HA interconnect: SBB Compatibility Event. Compatible partner node found. The interconnect device has been enabled.

Oct 14 23:01:57 [localhost:fal_nvme.partition.status:notice]: Partition 0-1 with capacity 894 GiB status: online.

Oct 14 23:02:02 [localhost:kern.version.change:notice]: Data ONTAP kernel version was changed from 9.9.1 to 9.7P15.

启动后简单完成初始化，确认下版本及License（建议提前跟400要好License）

SX-FAS2750::> cluster image show
Node                 Current Version         Installation Date
-------------------- ----------------------- -------------------
SX-FAS2750-01        9.7P15                  10/14/2021 22:58:18

SX-FAS2750::> system license show
Serial Number: 1-80-000011
Owner: SX-FAS2750
Package           Type     Description           Expiration
----------------- -------- --------------------- -------------------
Base              site     Cluster Base License  -