NVMe over TCP环境搭建和卸载

总结

本文讲述了如何在两个femu间搭建NVMe over TCP环境

前置知识

这里不过多解释何为NVMe over TCP，简单来说就是使用TCP协议下发NVMe命令给远端的NVMe设备。

NVMe over TCP环境搭建时，发送NVMe命令的叫做host端，接收NVMe命令的叫target端

femu启动时，默认用的是主机端的IP地址，为了两个femu之间能够通信，需要两个femu都有独立的IP，可以通过在主机端搭建虚拟网桥和虚拟网卡使两个femu拥有独立IP

1. 检查内核配置

在开始之前需要确定NVME_TCP、NVME_TARGET等内核配置是被set的 

$ cat /boot/config-`uname -r` | grep NVME

# NVME Support
CONFIG_NVME_CORE=m
CONFIG_BLK_DEV_NVME=m
# CONFIG_NVME_MULTIPATH is not set
# CONFIG_NVME_HWMON is not set
CONFIG_NVME_FABRICS=m
CONFIG_NVME_FC=m
CONFIG_NVME_TCP=m
CONFIG_NVME_TARGET=m
CONFIG_NVME_TARGET_LOOP=m
CONFIG_NVME_TARGET_FC=m
# CONFIG_NVME_TARGET_FCLOOP is not set
CONFIG_NVME_TARGET_TCP=m
# end of NVME Support

2. initiator端和target端安装nvme-cli

在host端和target端都安装上nvme-cli 

sudo modprobe nvmet
sudo modprobe nvmet-tcp

3. target端配置

cd /sys/kernel/config/nvmet/subsystems
sudo mkdir nvme-test-target
cd nvme-test-target/
echo 1 | sudo tee -a attr_allow_any_host > /dev/null
sudo mkdir namespaces/1
cd namespaces/1
echo -n /dev/nvme0n1 |sudo tee -a device_path > /dev/null
echo 1|sudo tee -a enable > /dev/null
sudo mkdir /sys/kernel/config/nvmet/ports/1
cd /sys/kernel/config/nvmet/ports/1

echo 192.168.64.239 |sudo tee -a addr_traddr > /dev/null 

echo tcp|sudo tee -a addr_trtype > /dev/null
echo 40000|sudo tee -a addr_trsvcid > /dev/null
echo ipv4|sudo tee -a addr_adrfam > /dev/null
sudo ln -s /sys/kernel/config/nvmet/subsystems/nvme-test-target/ /sys/kernel/config/nvmet/ports/1/subsystems/nvme-test-target

其中192.168.64.239是网卡的ip

4. intiator端配置

sudo modprobe nvmet
sudo modprobe nvmet-tcp
sudo nvme discover -t tcp -a 192.168.64.239 -s 40000
sudo nvme connect -t tcp -n nvme-test-target -a 192.168.64.239 -s 40000 --hostnqn=nqn.2014-08.org.nvmexpress:uuid:1b4e28ba-2fa1-11d2-883f-0016d3ccabcd

连接完成后，用命令nvme list可以看到盘，

kjay@kjay:~$ sudo nvme list
Node             SN                   Model                                    Namespace Usage                      Format           FW Rev
---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme1n1     713dcb7e1fef4c5d     Linux                                    1           4.29  GB /   4.29  GB    512   B +  0 B   5.4.0-77

5. 如何取消initiator端与target端的连接

sudo nvme disconnect /dev/nvme1n1 -n nvme-test-target
 

6.target端如何卸载nvme模块

sudo rm -rf /sys/kernel/config/nvmet/ports/1/subsystems/nvme-test-target
echo 0 > /sys/kernel/config/nvmet/subsystems/nvme-test-target/namespaces/1/enable
echo -n 0 > /sys/kernel/config/nvmet/subsystems/nvme-test-target/namespaces/1/device_path
sudo rmdir --ignore-fail-on-non-empty /sys/kernel/config/nvmet/subsystems/nvme-test-target/namespaces/1
sudo rmdir --ignore-fail-on-non-empty /sys/kernel/config/nvmet/subsystems/nvme-test-target
sudo rmdir --ignore-fail-on-non-empty /sys/kernel/config/nvmet/ports/1
sudo rmmod nvmet-tcp
sudo rmmod nvmet