总结
本文讲述了如何在两个femu间搭建NVMe over TCP环境
前置知识
这里不过多解释何为NVMe over TCP,简单来说就是使用TCP协议下发NVMe命令给远端的NVMe设备。
NVMe over TCP环境搭建时,发送NVMe命令的叫做host端,接收NVMe命令的叫target端
femu启动时,默认用的是主机端的IP地址,为了两个femu之间能够通信,需要两个femu都有独立的IP,可以通过在主机端搭建虚拟网桥和虚拟网卡使两个femu拥有独立IP
1. 检查内核配置
在开始之前需要确定NVME_TCP、NVME_TARGET等内核配置是被set的
$ cat /boot/config-`uname -r` | grep NVME
# NVME Support
CONFIG_NVME_CORE=m
CONFIG_BLK_DEV_NVME=m
# CONFIG_NVME_MULTIPATH is not set
# CONFIG_NVME_HWMON is not set
CONFIG_NVME_FABRICS=m
CONFIG_NVME_FC=m
CONFIG_NVME_TCP=m
CONFIG_NVME_TARGET=m
CONFIG_NVME_TARGET_LOOP=m
CONFIG_NVME_TARGET_FC=m
# CONFIG_NVME_TARGET_FCLOOP is not set
CONFIG_NVME_TARGET_TCP=m
# end of NVME Support
2. initiator端和target端安装nvme-cli
在host端和target端都安装上nvme-cli
sudo modprobe nvmet
sudo modprobe nvmet-tcp
3. target端配置
cd /sys/kernel/config/nvmet/subsystems
sudo mkdir nvme-test-target
cd nvme-test-target/
echo 1 | sudo tee -a attr_allow_any_host > /dev/null
sudo mkdir namespaces/1
cd namespaces/1
echo -n /dev/nvme0n1 |sudo tee -a device_path > /dev/null
echo 1|sudo tee -a enable > /dev/null
sudo mkdir /sys/kernel/config/nvmet/ports/1
cd /sys/kernel/config/nvmet/ports/1
echo 192.168.64.239 |sudo tee -a addr_traddr > /dev/null
echo tcp|sudo tee -a addr_trtype > /dev/null
echo 40000|sudo tee -a addr_trsvcid > /dev/null
echo ipv4|sudo tee -a addr_adrfam > /dev/null
sudo ln -s /sys/kernel/config/nvmet/subsystems/nvme-test-target/ /sys/kernel/config/nvmet/ports/1/subsystems/nvme-test-target
其中192.168.64.239是网卡的ip
4. intiator端配置
sudo modprobe nvmet
sudo modprobe nvmet-tcp
sudo nvme discover -t tcp -a 192.168.64.239 -s 40000
sudo nvme connect -t tcp -n nvme-test-target -a 192.168.64.239 -s 40000 --hostnqn=nqn.2014-08.org.nvmexpress:uuid:1b4e28ba-2fa1-11d2-883f-0016d3ccabcd
连接完成后,用命令nvme list可以看到盘,
kjay@kjay:~$ sudo nvme list
Node SN Model Namespace Usage Format FW Rev
---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme1n1 713dcb7e1fef4c5d Linux 1 4.29 GB / 4.29 GB 512 B + 0 B 5.4.0-77
5. 如何取消initiator端与target端的连接
sudo nvme disconnect /dev/nvme1n1 -n nvme-test-target
6.target端如何卸载nvme模块
sudo rm -rf /sys/kernel/config/nvmet/ports/1/subsystems/nvme-test-target
echo 0 > /sys/kernel/config/nvmet/subsystems/nvme-test-target/namespaces/1/enable
echo -n 0 > /sys/kernel/config/nvmet/subsystems/nvme-test-target/namespaces/1/device_path
sudo rmdir --ignore-fail-on-non-empty /sys/kernel/config/nvmet/subsystems/nvme-test-target/namespaces/1
sudo rmdir --ignore-fail-on-non-empty /sys/kernel/config/nvmet/subsystems/nvme-test-target
sudo rmdir --ignore-fail-on-non-empty /sys/kernel/config/nvmet/ports/1
sudo rmmod nvmet-tcp
sudo rmmod nvmet