DAOS-在docker中搭建daos开发调试环境

简介

本节介绍如何在 Docker 容器中构建和运行 DAOS 服务。 至少需要 4GB DRAM 和 16GB 磁盘空间。 在 Mac 上，请确保“Preferences/{Disk, Memory}”下的 Docker 设置已进行相应配置

步骤

拉取项目

git clone https://github.com/ssbandjl/daos.git

下载依赖包(缓存cache)

cd daos; wget http://47.92.214.21:8888/daos/cache_tgz

编译镜像或下载我这边已经编好的镜像

编译镜像: docker build  . -f utils/docker/Dockerfile.el.8 -t daos
# 已编好的镜像: wget http://47.92.214.21:8888/daos/daos_main.tgz; docker load -i daos_main.tgz

查看docker镜像:
[root@node3 daos]# docker images
REPOSITORY   TAG       IMAGE ID       CREATED       SIZE
daos         latest    8a64df225def   2 weeks ago   3.16GB

给宿主机配置大页(如8GB):
永久生效
tee<<EOF>>/etc/sysctl.conf
vm.nr_hugepages = 4096
EOF
sysctl -p

启动docker镜像:
docker run -it -d --privileged --cap-add=ALL --name daos_main1 -v /home/xb/project/stor/daos/main/daos:/home/xb/project/stor/daos/main/daos -v /dev:/dev -v /dev/hugepages:/dev/hugepages -v /sys/fs/cgroup:/sys/fs/cgroup:ro daos:latest

进入镜像: docker exec -u root -it daos_main1 bash -c 'cd /home/xb/project/stor/daos/main/daos;exec "${SHELL:-sh}"'

重新编译并安装DAOS:
./build.sh 或
scons --jobs 1 install PREFIX=/opt/daos BUILD_TYPE=debug TARGET_TYPE=debug --build-deps=yes

which dmg

拷贝证书和配置(启动时根据提示修改权限):
cp -r daosCA/certs /etc/daos/


参考配置文件:
daos_server_docker_main.yml
# For a single-server system

name: daos_server
access_points: ['172.17.0.2']
provider: ofi+sockets
control_log_mask: DEBUG
control_log_file: /tmp/daos_server.log
transport_config:
   allow_insecure: false
   client_cert_dir: /etc/daos/certs/clients
   ca_cert: /etc/daos/certs/daosCA.crt
   cert: /etc/daos/certs/server.crt
   key: /etc/daos/certs/server.key

engines:
-
  targets: 1
  first_core: 0
  nr_xs_helpers: 0
  fabric_iface: eth0
  fabric_iface_port: 31416
  log_mask: debug
  log_file: /tmp/daos_engine.0.log

  env_vars:
  - FI_LOG_LEVEL=warn
  - FI_SOCKETS_MAX_CONN_RETRY=1
  - FI_SOCKETS_CONN_TIMEOUT=2000
  - D_LOG_MASK=DEBUG
  - DD_SUBSYS=all
  - DD_MASK=all
  - HG_LOG_LEVEL=debug

  # Storage definitions

  # When scm_class is set to ram, tmpfs will be used to emulate SCM.
  # The size of ram is specified by scm_size in GB units.
  scm_mount: /mnt/daos  # map to -s /mnt/daos
  scm_class: ram
  scm_size: 4  #内存模拟scm

  # When bdev_class is set to file, Linux AIO will be used to emulate NVMe.
  # The size of file that will be created is specified by bdev_size in GB units.
  # The location of the files that will be created is specified in bdev_list.
  bdev_class: file
  bdev_size: 16   #文件模拟nvme
  bdev_list: [/tmp/daos-bdev]
  
启动集群:
./start.sh
或参考:
echo -e "start"

# source /root/.bashrc
# run_all "pkill daos_agent;pkill daos_server"

source /root/.bashrc
pkill daos_agent 
pkill daos_server
rm -f /tmp/daos*.log
rm -f /tmp/.daos_engine.0.log.swp
rm -f /tmp/daos_engine*
umount /mnt/daos

#export FI_LOG_LEVEL=debug
#export FI_LOG_LEVEL=warn

#export HG_LOG_LEVEL=debug
#export HG_LOG_LEVEL=warn

daos_agent &
# echo -e "dmg storage format"
daos_server start &

# count=0
# while true;do
#     joined_num=`dmg sys query -v|grep Joined|wc -l`
#     if [[ $joined_num -eq 3 ]];then
#         break
#     fi
#     echo -e "wait all rank join, $count times"
#     count=$((count+1))
#     sleep 1
# done

sleep 5
dmg storage format


创建池,容器, 挂载daos文件系统:
参考文件: daos_create_pool_cont_mount_dfuse
dmg pool create sxb -z 4g; dmg pool list --verbose
daos container create sxb --type POSIX sxb; daos container query sxb sxb --verbose; daos cont get-prop sxb sxb
mkdir -p /tmp/sxb; dfuse --mountpoint=/tmp/sxb --pool=sxb --cont=sxb; df -h
#cd /tmp/sxb

最终效果:

[root@8b90fa1190f1 daos]# dmg pool list --verbose
Label UUID                                 State SvcReps SCM Size SCM Used SCM Imbalance NVME Size NVME Used NVME Imbalance Disabled UpgradeNeeded? 
----- ----                                 ----- ------- -------- -------- ------------- --------- --------- -------------- -------- -------------- 
sxb   9b4157a8-7bab-493c-aa4d-5145144d92b3 Ready 0       240 MB   213 kB   0%            3.8 GB    52 MB     0%             0/1      None           

[root@8b90fa1190f1 daos]# daos container query sxb sxb --verbose; daos cont get-prop sxb sxb
  Container UUID              : ccc2f4f7-1e46-4709-8f28-09282a9f82ed                       
  Container Label             : sxb                                                        
  Container Type              : POSIX                                                      
  Pool UUID                   : 9b4157a8-7bab-493c-aa4d-5145144d92b3                       
  Container redundancy factor : 0                                                          
  Number of open handles      : 2                                                          
  Latest open time            : 0x126c7703f0840000 (2023-08-19 08:02:08.06070272 +0000 UTC)
  Latest close/modify time    : 0x126c76d527b80000 (2023-08-19 08:01:55.50212096 +0000 UTC)
  Number of snapshots         : 0                                                          
  Object Class                : UNKNOWN                                                    
  Dir Object Class            : UNKNOWN                                                    
  File Object Class           : UNKNOWN                                                    
  Chunk Size                  : 1.0 MiB                                                    

Properties for container sxb
Name                                             Value                               
----                                             -----                               
Highest Allocated OID (alloc_oid)                11                                  
Checksum (cksum)                                 off                                 
Checksum Chunk Size (cksum_size)                 32 KiB                              
Compression (compression)                        off                                 
Deduplication (dedup)                            off                                 
Dedupe Threshold (dedup_threshold)               4.0 KiB                             
EC Cell Size (ec_cell_sz)                        64 KiB                              
Performance domain affinity level of EC (ec_pda) 1                                   
Encryption (encryption)                          off                                 
Global Version (global_version)                  3                                   
Group (group)                                    root@                               
Label (label)                                    sxb                                 
Layout Type (layout_type)                        POSIX (1)                           
Layout Version (layout_version)                  1                                   
Max Snapshot (max_snapshot)                      0                                   
Object Version (obj_version)                     1                                   
Owner (owner)                                    root@                               
Performance domain level (perf_domain)           root (255)                          
Redundancy Factor (rd_fac)                       rd_fac0                             
Redundancy Level (rd_lvl)                        node (2)                            
Performance domain affinity level of RP (rp_pda) 4294967295                          
Server Checksumming (srv_cksum)                  off                                 
Health (status)                                  HEALTHY                             
Access Control List (acl)                        A::OWNER@:rwdtTaAo, A:G:GROUP@:rwtT 
[root@8b90fa1190f1 daos]# df -h
Filesystem                  Size  Used Avail Use% Mounted on
overlay                      50G   25G   26G  50% /
devtmpfs                     63G     0   63G   0% /dev
tmpfs                        63G     0   63G   0% /dev/shm
/dev/mapper/xxx-var    50G   25G   26G  50% /etc/hosts
tmpfs                        63G     0   63G   0% /sys/fs/cgroup
/dev/mapper/xxx-root  120G   56G   65G  47% /home/xb/project/stor/daos/main/daos
tmpfs                       4.0G  500M  3.6G  13% /mnt/daos
dfuse                       3.8G   51M  3.7G   2% /tmp/sxb
[root@8b90fa1190f1 daos]# 

参考

在docker中搭建开发环境: https://docs.daos.io/v2.0/dev/development/#daos-development-in-docker

DAOS IO全路径详解(视频)
DAOS 项目简介(视频)

晓兵

博客: https://logread.cn | https://blog.csdn.net/ssbandjl | https://cloud.tencent.com/developer/user/5060293/articles

weixin: ssbandjl

公众号: 云原生云

欢迎对DAOS, SPDK, RDMA等高性能技术感兴趣的朋友加入[DAOS技术交流(群)]