Ceph测试系列（一）：Ceph 应用方案的服务器硬件选型 (2 of 2)

一、前言

redhat官网给出不同应用场景时，其推荐的Ceph集群服务器硬件配置（包括CPU/Memory/Disk/Network）。仅作为服务器配置选择时参考，不做推荐使用。

场景包含如下：

场景：一侧重IOPS的（IOPS低延迟），例如实时性要求高的，但数据量不大的。比如订单的生成。

场景二：侧重Throughput的 （吞吐量优先），吞吐量大，但IOPS延时性要求适当。例如，实时流。

场景三：侧重容量和价格Cost/Capacity的（大存储量），例如大文件的存储。

二、硬件选型标准

1. 这里主要讲IOPS，Throughput, Cost/Capacity 三种不同场景的应用方案在硬件选型时的标准：

 

SOLUTIONS	CPU	RAM	Networking	OSD Media	OSDs	Journal Media	Controller/HBA	Vendor(i.e.)	Example Uses
IOPS-OPTIMIZED	10 cores per NVMe SSD, assuming a 2 GHz CPU.	16GB baseline, plus 2GB per OSD	10 Gigabit Ethernet (GbE) per 12 OSDs (each for client- and cluster-facing networks)	High-performance, high-endurance enterprise NVMe SSDs	Four per NVMe SSD	High-performance, high-endurance enterprise NVMe SSD, co-located with OSDs	Native PCIe bus	SuperMICRO(SYS-5038MR-OSD006P)	Database
THROUGHPUT-OPTIMIZED	0.5 cores per HDD, assuming a 2 GHz CPU	16GB baseline, plus 2GB per OSD	10 GbE per 12 OSDs (each for client- and cluster-facing networks)	7,200 RPM enterprise HDDs	One per HDD	High-endurance, high-performance enterprise serial-attached SCSI (SAS) or NVMe SSDs; OSD-to-journal ratio: 4-5:1 for an SSD journal, or 12-18:1 for an NVMe journal	Host bus adapter (HBA): just a bunch of disks (JBOD)	SuperMICRO(SSG-6028R-OSD072P)	Streaming media
COST/CAPACITY-OPTIMIZED	0.5 cores per HDD, assuming a 2 GHz CPU	16GB baseline, plus 2GB per OSD	10 GbE per 12 OSDs (each for client- and cluster-facing networks)	7,200 RPM enterprise HDDs	One per HDD	Co-located on the HDD	HBA:JBOD	SuperMICRO(SSG-6048R-OSD216P)	Archive
Reference: https://www.redhat.com/en/resources/resources-red-hat-ceph-storage-hardware-selection-guide-html

2. SuperMicro推荐的Ceph服务器机型：

块和文件存储类服务器：OSD 服务器 (CPU负载低)

Category	Block and File OSD Node
Server SKU	SSG-6019P-ACR12L+			SSG-5029P-E1CTR12L			SSG-6029P-E1CR12L
Profile	Density Optimized			Cost Optimized			Compute Optimized
Form Factor	1U			2U			2U
CPU	Dual Intel® Xeon® Scalable Processors (8C/16T each)			Dual Intel® Xeon® Scalable Processors (12C/24T each)			Dual Intel® Xeon® Scalable Processors (8C/16T each)
Memory	96GB DDR4 RDIMM
HBA	Broadcom 3216 (IT mode)			Broadcom 3008 (IT mode)			Broadcom 3008 (IT mode)
Power	600W Redundant Power Supplies			800W Redundant Power Supplies			1200W Redundant Power Supplies
NIC	Dual-Port 25G Mellanox (SFP+ compatibility) via AOC
Storage	Type	Qty	Form Factor	Type	Qty	Form Factor	Type	Qty	Form Factor
Caching	Optional NVMe	Up to 2	2.5" 7mm	Optional NVMe	1	AIC	Optional NVMe	1	AIC
Capacity	6-12 TB 7K RPM	12	3.5" SAS3 HDD	6-12 TB 7K RPM	12	3.5" SAS3 HDD	6-12 TB 7K RPM	12	3.5" SAS3 HDD
Drive (OS)	2x 2.5" 7mm SATA SSD			2x 2.5"  SATA SSD			2x 2.5"  SATA SSD

媒体类媒体流和媒体仓库服务器：OSD服务器 (CPU负载适中)

Category	Media OSD Node (File and Object)
Server SKU	SSG-6029P-E1CR24L			SSG-6049P-E1CR45L SSG-6049P-E1CR45L+
Profile	Media Streaming			Media Repository
Form Factor	1U			2U
CPU	Dual Intel® Xeon® Scalable Processors (12C/24T each)			Dual Intel® Xeon® Scalable Processors (12C/24T each)
Memory	192GB DDR4 RDIMM			384GB DDR4 RDIMM
HBA	Broadcom 3008 (IT mode)			Broadcom 3008 (IT mode)(L model) Broadcom 3216 (IT mode)(L+ model)
Power	1600W Redundant Power Supplies			1600W Redundant Power Supplies
NIC	SIOM-based Dual-Port 25G Mellanox (SFP+ compatibility) + Dual-Port 10G RJ45
Storage	Type	Qty	Form Factor	Type	Qty	Form Factor
Caching	Optional NVMe	Up to 2	AIC	Optional NVMe	Up to 6	2.5" U.2
Capacity	6-12 TB 7K RPM	24	3.5" SAS3 HDD	6-12 TB 7K RPM	45	3.5" SAS3 HDD
Drive (OS)	2x 2.5" SATA SSD			2x 2.5"  SATA SSD

媒体类存档服务器：OSD服务器(CPU负载稍高)

Category	Media OSD Node (File and Object)
Server SKU	SSG-6049P-E1CR60L SSG-6049P-E1CR60L+
Profile	Archive (minimum recommended deployment size of 10+ OSD Nodes)
Form Factor	4U
CPU	Dual Intel® Xeon® Scalable Processors (14C/28T each)
Memory	384GB DDR4 RDIMM
HBA	Broadcom 3008 (IT mode)(L model) Broadcom 3616 (IT mode)(L+ model)
Power	2000W Redundant Power Supplies
NIC	SIOM-based Dual-Port 25G Mellanox (SFP+ compatibility) + Dual-Port 10G RJ45
Storage	Type	Qty	Form Factor
Caching	Optional NVMe	Up to 6	2.5" U.2
Capacity	6-12 TB 7K RPM	60	3.5" SAS3 HDD
Drive (OS)	2x 2.5" SATA SSD

 Monitor/网关服务器 (CPU负载适中)

Category	Monitor Node / Gateway
Server SKU	SYS-1019P-WTR
Profile	Flash Performance (minimum recommended deployment size of 3 Monitor Nodes)
Form Factor	1U
CPU	Single Intel® Xeon® Scalable Processor (12C/24T each)
Memory	96GB DDR4 RDIMM
HBA	Broadcom 3008 (IT mode)(L model)
Power	500W Redundant Power Supplies
NIC	Dual-Port 25G Mellanox (SFP+ compatibility) via AOC
Storage	Type	Qty	Form Factor
Caching	N/A	-	-
Capacity	N/A	-	-
Drive (OS)	2x 2.5" SATA SSD

美光固态高速服务器：(CPU负载高)

Category	Micron Solid Ready
Reference BOM Check with PM for details	SYS-1028U-MON50M			SYS-1029U-OSD100M
Server SKU	SYS-1028U-TNRT+			SYS-1029U-TN10RT
Profile	Monitor Node			OSD Node
Form Factor	1U			1U
CPU	Dual Intel® Xeon® Scalable Processors (10C/20T each)			Dual Intel® Xeon® Scalable Processors (24C/48T each)
Memory	128GB DDR4 RDIMM			384GB DDR4 RDIMM
HBA	Broadcom 3008 (IT mode)			N/A
Power	750W Redundant Power Supplies			1000W Redundant Power Supplies
NIC	2x Single-Port 50GbE QSFP28 (Mellanox)			2x Dual-Port 100GbE QSFP (Mellanox)
Storage	Type	Qty	Form Factor	Type	Qty	Form Factor
Caching	N/A	-	-	Optional NVMe	-	-
Capacity	N/A	-	-	6.4 TB NVMe	10	U.2 NVMe
Drive (OS)	1x SATADOM			1x M.2 SATA SSD

三、结论：

可以看到高负载的CPU服务器需求集中在(美光)固态高速服务器 和 媒体类存档服务器(OSD服务器)。由于没有高负载计算过，存储服务器总体来说对CPU要求不高。所以如果你的CPU配置高，CPU负载不高，可以用如下服务器配置优化方案：

1. 配置CEPH的副本模式为 EC（Erase Code）来使用更多的CPU资源，减少磁盘IO，而不是配置Replicated。
2. 使用低延迟高吞吐率的NVME SSD 盘 和 增加 磁盘数
3. 使用Cache Tier 和 增加Cache size: 使用Cache Tier 和 增加 Cache size
4. 如果x86服务器功耗和价格不符合你的需求，可以考虑采购arm服务器，在存储领域，arm服务器的性价比很高。

Refer to: https://www.supermicro.org.cn/en/solutions/red-hat-ceph