目录
华为泰山 200 2.5 英寸 NVMe SSD 和 SATA/SAS 背板
华为泰山 200 HNS 4x 25GbE RDMA 适配器
在 STH 进行的一个更雄心勃勃的项目中，我们有一个华为泰山服务器向你展示。使得这个服务器比在美国被禁止更有趣的是，它使用的 CPU 是华为/海思的 Arm 服务器 CPU。具体来说，这是一台华为/海思的鲲鹏 920 服务器，它是首批 Arm v8 服务器 CPU 之一，最多有 64 个内核，支持 PCIe Gen4。就项目状况而言，这台服务器一直很不稳定，这一点我们在最近的 Solidigm D7-P5520 7.68TB PCIe Gen4 NVMe SSD Review on x86 Arm and Power9 这块的评论中有所涉及。因此，在这一点上，我们要做的不是评论，而是更多的记录我们所发现的东西，并讨论是什么让这个被禁止的服务器变得独特。当我们通过硬件配置时，很明显，这台服务器非常不同，光看硬件就有一大堆内容。
华为泰山 200 2280 服务器概述
我们将把我们的硬件概述分成两部分。我们将有一个外部和内部概述。在内部概述中，我们将对一些真正有趣的非 CPU 组件进行深入研究，并提供比平时更多的照片。还将有第二篇文章专门介绍鲲鹏 920。如果你喜欢通过视频听和看更多的镜头，我们有一个视频：
受限的 CPU 鲲鹏 920 和服务器（A Banned Server and CPU the US Forbids）_哔哩哔哩_bilibili
像往常一样，我们建议在自己的浏览器、标签或应用程序中打开它，以获得最佳观看体验。在视频中，有几个角度我们没有图片，只是由于视频的性质。
华为泰山 200 2280 外部服务器概述
从服务器的前部开始，这是一个 25 个 2.5 英寸硬盘的设计，但是我们只有24个硬盘托盘。这是一个相当标准的 2U 服务器的配置，至少在我们进入 EDSFF 时代之前是这样。
Huawei TaiShan 200 2280 Front
In this server, eight slots are NVMe while the other slots are SATA/ SAS.
Huawei TaiShan 200 2280 2.5in Drive Tray
Something that was surprising is that the drive trays were not tool-less. They still require screws. Huawei also uses drive blanks like Dell EMC, HPE, and Lenovo rather than just providing drive trays like Supermicro, Inspur, QCT, and others.
Huawei TaiShan 200 2280 2.5in Blank Tray
While this is a TaiShan 200 2280, this is the label for the part with K22R-02.
Huawei TaiShan 200 2280 Model Info Including FCC
On the left side, we get the Huawei logo, power, and status LEDs/ LCD.
Huawei TaiShan 200 2280 Front Left
On the right side, we get a VGA port and two USB 3.0 ports. Most servers we review have at most a single front USB port, so that is a little bit different.
Huawei TaiShan 200 2280 Front Right
Moving to the rear, things start to get a bit more exciting. One can see that there are eight full-height PCIe slots, but only one can be used for a card.
Huawei TaiShan 200 2280 Rear 2
Primary networking is provided via two OCP NIC 3.0 slots. We will deep-dive into this quad port 1GbE solution, but let us just say it is one of the most interesting solutions out there.
Huawei TaiShan 200 2280 Rear 4x 1GbE Realtek OCP
The main I/O block has management, RJ45 serial, VGA, and two USB ports. This is a standard stack except we rarely see the RJ45 form factor serial ports in servers. Those are more common in switches and firewalls. That section is connected to a PCB with a chip we cannot find information on via Google, and we will go into that in our internal deep-dive.
Huawei TaiShan 200 2280 Rear IO
The other OCP NIC 3.0 slot is to the right of the I/O block.
In Slot 8 we get a quad 25GbE port HNS card.
Huawei TaiShan 200 2280 HNS 4x 25GbE NIC Rear
One of the two power supplies got damaged during shipping. This one has many ratings and certifications, including the 80Plus Platinum certification. It is also a 200V+ power supply so it is not meant for lower power 110-120V North American racks.
Huawei TaiShan 200 2280 2kW PSU
As exciting as that is, the more exciting aspect is really what we see inside the server. From the outside, this is fairly standard. Once we get inside, things get interesting. Let us move on to that.
华为泰山 200 2280 内部服务器概述
在这篇稿件中，我们将展示比平时更多的角度和照片，介绍服务器内部的一些特定部分。我们意识到这是一个重要的行业平台，但也是我们的读者很少见过的。
华为泰山 200 2.5 英寸 NVMe SSD 和 SATA/SAS 背板
看一下机箱的正面，我们可以看到存储背板，一个用于电缆的大通道，然后是风扇分区。
Huawei TaiShan 200 2280 Storage Backplane 1
The storage backplane supports up to eight PCIe Gen4 x4 NVMe SSDs. It also has provisions for the other sixteen slots being SATA/ SAS.
Huawei TaiShan 200 2280 Storage Backplane 2
For those wondering from photos, the backplane is not bent wildly. Instead, the sheet metal above the backplane has a curve to it.
Huawei TaiShan 200 2280 Storage Backplane 3
这里有一点不同的是，我们看到很多行业都转向在2U 24-bay设计上将背板分割成三份。这使得配置更加灵活，并且在背板发生故障时减少了故障域。2.5英寸的8盘背板也更容易维修。不过，看看这台服务器是如何建造的，还是很有意思的。
华为泰山 200 冷却
机箱中间，存储背板后面是为冷却而设计的。有四个大风扇，然后是一个气流导向。在这里我们可以看到这个设置，还有一个小但很好的功能，就是在气流导板上标明DIMM插槽以及哪个CPU是哪个。这是一个小功能，但在DIMM发生故障需要维修的时候，这是一个很好的功能。
Huawei TaiShan 200 2280 Fan To Air Shroud Gap 1
The fans use six-pin custom connectors, but the fans only have five wires so not all six pins are being used. The fans are non-redundant and do not have counter-rotating designs. Still, it appears as though there are several unused fan connectors on the motherboard. There are also unpopulated pads labeled for 1U servers. As a result, we think there are 1U versions of this motherboard with those fan headers populated and perhaps even other fan configurations that we do not have.
Huawei TaiShan 200 2280 Fan Headers And Gap
Perhaps the most interesting part here is the giant gap. Between the fans and the airflow guide over the CPUs, there is a very large gap when most servers start the airflow guide directly at the exhaust of the fans to reduce airflow turbulence. Huawei does not do this. For those who are wondering if the airflow guide was in the wrong place, there are notches it fits into on the chassis edge to keep it situated. This is very different from most server designs we see from other vendors.
Huawei TaiShan 200 2280 Fan Headers Without Airflow Shroud
The fans themselves actually fit easily into the custom connectors. Given the design, we did not know how that hot-swap process would work. It was certainly an upside surprise.
Huawei TaiShan 200 2280 Fans
Something that was a bit different with the fans is that they have slats. These slats allow air to pass during normal operation. If a fan fails, these slats can close, thereby not allowing airflow leakage through the stopped fan.
Huawei TaiShan 200 2280 CPU And Memory With Shroud And Broadcom SAS
气流导板的顶部也有用于RAID控制器的BBU安装空间。我们没有这些电池，但这是在服务器中放置电池的一个常见位置。
华为泰山 200 鲲鹏 920 的 CPU 和 DIMMs
CPU 区域看起来非常现代。我们得到了两个带有大型散热片的CPU，每个CPU有16x DIMMs。这意味着8通道内存和总共32个DIMMS。这与DDR4时代的Ampere Altra（Max）、AMD EPYC和Intel Ice Lake Xeons类似。
Huawei TaiShan 200 2280 CPU And Memory With Airflow Guide And Risers
Here is a look at the area with the airflow guide removed.
Huawei TaiShan 200 2280 CPU Sockets And Memory Installed 1
Here it another view. One will notice our server came with 6 DDR4 DIMMs per CPU. The memory is DDR4 ECC and supports RDIMMs. We are using 8x 32GB per CPU for performance testing, but there are still some quirks to iron out. In our recent Solidigm D7-P5520 7.68TB PCIe Gen4 NVMe SSD Review on x86 Arm and Power9 it took some time just to get something as simple as that NVMe SSD working reasonably well on this system.
Huawei TaiShan 200 2280 CPU And Memory Area 2
The CPUs themselves are Huawei, HiSilicon Kunpeng 920 models. They are 48 core 2.6GHz CPUs. Kunpeng 920 models scaled to 64 cores, but this is what we can get. As we get the system sorted, we will have more formal performance figures, but our 48 core models are roughly equivalent to Cascade Lake 24 core Xeon models in integer workloads. There is more memory bandwidth available and PCIe in this platform.
Huawei TaiShan 200 2280 CPU And Memory Area 1
Perhaps one of the most interesting connectors we saw was this one. The “CPU1 Hydra” connector.
Huawei TaiShan 200 2280 Huawei Hydra CPU 1
Here is the CPU2 Hydra connector:
Huawei TaiShan 200 2280 Huawei Hydra CPU 2
Hydra是华为的CPU到CPU互连的名称，似乎与AMD EPYC的Infinity Fabric和英特尔至强UPI大致相似。我们通常不会看到UPI电缆。我们不时看到AMD EPYC的IF电缆，因为它们可以被重新用于PCIe通道，正如我们在 Dell EMC PowerEdge R7525 review 中看到的那样。我们将在本文的框图部分更多地展示Hydra。
接下来，我们将更详细地了解一下主板的情况。
华为泰山 200 主板
这台服务器真正有趣的地方在于，它采取了一种更现代的设计方法。主板没有延伸到机箱的后面。相反，BMC和网络（通过OCP NIC 3.0插槽）被用来到达后部并提供I/O空间。
Huawei TaiShan 200 2280 Internal Shot Without Risers From Empty OCP NIC 3.0
The motherboard also has a number of standard connectors. We see TPM connectors, along with PCIe riser connectors. There are also power headers for internal hard drives and accelerators.
Huawei TaiShan 200 2280 PCIe Gen4 X24 Riser And Power Without Broadcom SAS
In the middle of the system, we get more risers and the storage mezzanine connector.
Huawei TaiShan 200 2280 Motherboard Area With Broadcom SAS Removed
On the other side, we get power input directly to the motherboard and additional power headers for internal components.
Huawei TaiShan 200 2280 Internal HDD And PCIe Power
接下来让我们探讨一下这个系统中的立管和林立的线缆。
华为泰山 200 PCIe 立管
这个系统中的PCIe立管有点不同。这可能是一个配置选项，因为有插槽的立管会有很大的意义。但在这个系统中，我们有PCIe立管，有电缆的I/O。
Huawei TaiShan 200 2280 Risers With NVMe Middle
The risers are held in place by thumbscrews. These usually required us to use screwdrivers to disassemble.
Huawei TaiShan 200 2280 Riser Thumb Screw
Here is one of the risers. We did not take the heatsinks off to see, since if we break something getting replacement parts is effectively impossible, but our best guess is that these have retimers under them. Those would also be very large retimer heatsinks.
Huawei TaiShan 200 2280 PCIe Riser For NVMe Out 1
These risers are held in place mostly by pins to make them easy to service.
Huawei TaiShan 200 2280 PCIe Riser For NVMe Out 2
Here is a look at the middle riser:
Huawei TaiShan 200 2280 Risers With NVMe Middle 2
我们将对远处的立管进行更广泛的观察。接下来，我们将看一下该立管中的 HNS 25GbE 适配器和 Broadcom SAS 控制器。
华为泰山 200 HNS 4x 25GbE RDMA 适配器
在最后一个立管上，我们得到了一个网卡，在系统中显示为HNS网络适配器。
Huawei TaiShan 200 2280 HNS 4x 25GbE NIC Riser Installed 1
This adapter has four SFP28 ports for up to 25GbE operation.
Huawei TaiShan 200 2280 HNS 4x 25GbE NIC Riser Installed 2
The riser uses different cables than the other PCIe risers that go to the front of the system.
Huawei TaiShan 200 2280 HNS 4x 25GbE NIC Riser Installed 3
Despite there being eight full-height slots on the back of the server, this is the only one we had with a NIC.
Huawei TaiShan 200 2280 HNS 4x 25GbE NIC Riser Installed 4
Here is the NIC out of the chassis still in the riser:
Huawei TaiShan 200 2280 HNS 4x 25GbE NIC Riser Out Front 1
Here is the rear side. Huawei’s design of putting the bracket support more in the middle of the card is one that we do not see very often, even having reviewed dozens of low-profile NICs.
Huawei TaiShan 200 2280 HNS 4x 25GbE NIC Riser Out Back 2
Here is just another photo of that:
Huawei TaiShan 200 2280 HNS 4x 25GbE NIC Riser Out Back 1
Here is what we saw when we booted the system with the HNS adapter in it:
*-network:0
description: Ethernet interface
product: HNS GE/10GE/25GE RDMA Network Controller
vendor: Huawei Technologies Co., Ltd.
physical id: 0
bus info: pci@0000:7d:00.0
logical name: enp125s0f0
version: 21
capacity: 1Gbit/s
width: 64 bits
clock: 33MHz
capabilities: pciexpress msix pm bus_master cap_list ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation
configuration: autonegotiation=on broadcast=yes driver=hns3 driverversion=5.13.0-27-generic firmware=1.9.33.10 latency=0 link=no multicast=yes port=twisted pair
resources: iomemory:10-f iomemory:10-f irq:0 memory:121000000-12100ffff memory:120000000-1200fffff memory:121010000-12103ffff memory:120100000-1203fffff
我们在让光学器件在这些端口工作方面遇到了一些挑战，但我们已经订购了一些编码的光学器件，希望能够发挥作用。
华为泰山 200 博通 SAS3408 RAID 控制器
我们提到的服务器的前置端口可以是SAS或SATA。华为在中间的DIMMs后面使用了一个专有的SAS夹层卡插槽。我们在其他传统的服务器供应商那里看到过这种情况。
Huawei TaiShan 200 2280 CPU And Memory With Shroud And Broadcom SAS
The card itself is an 8-port model, but one can see that there are provisions for not just the SAS3408 we have here, but also a SAS3416 model with sixteen ports.
Huawei TaiShan 200 2280 Motherboard Area With Broadcom SAS 2
The Huawei-Broadcom SAS controller sits on a custom mezzanine connector that has bare vertical pins. We do not see this type of connector often in modern servers.
Huawei TaiShan 200 2280 Motherboard Area With Broadcom SAS Removed
Here is a look at that area with the Broadcom SAS adapter installed:
Huawei TaiShan 200 2280 Broadcom SAS Controller Mezz Area 1
We were slightly surprised to see the Broadcom SAS3408 here. Much of this server is focused on using non-Intel or other US parts. We thought Huawei would have swapped another controller in given the OCP NIC 3.0 card we will look at next.
Huawei TaiShan 200 2280 Broadcom SAS 1
Here is the bottom of the Huawei Broadcom SAS3408 card. We can see that there are pads for a second motherboard connector and pads for the extra ports for the SAS3416 version.
Huawei TaiShan 200 2280 Broadcom SAS 2 Bottom
Here is the LSI Broadcom SAS controller in Ubuntu:
*-raid
description: RAID bus controller
product: MegaRAID Tri-Mode SAS3408
vendor: Broadcom / LSI
physical id: 0
bus info: pci@0000:05:00.0
logical name: /dev/fb0
version: 01
width: 64 bits
clock: 33MHz
capabilities: raid pm msi pciexpress msix bus_master cap_list rom fb
configuration: depth=32 driver=megaraid_sas latency=0 mode=1024x768 visual=truecolor xres=1024 yres=768
resources: iomemory:8000-7fff iomemory:8000-7fff irq:26 memory:80000100000-800001fffff memory:80000000000-800000fffff memory:e6400000-e64fffff ioport:0(size=256) ioport:e6300000(size=1048576)
Here is another shot with the card installed and the middle riser removed:
Huawei TaiShan 200 2280 Broadcom SAS Controller Mezz 1
Here is that same setup from a slightly offset angle:
Huawei TaiShan 200 2280 Broadcom SAS Controller Mezz 2
我们这次的照片当然比平时多，但考虑到看到这台服务器的稀有性，这似乎是值得的。
接下来，我们将看看OCP网卡3.0插槽，包括独特的4x 1GbE适配器和BMC解决方案。
华为泰山 200 OCP NIC 3.0 插槽
主板后面有两个OCP NIC 3.0插槽。其中一个在我们的系统中未被占用。
Huawei TaiShan 200 2280 Empty OCP NIC 3.0 Slot
The second one holds what is perhaps the most interesting OCP NIC 3.0 card we have seen, at least in the quad 1GbE configuration.
Huawei TaiShan 200 2280 4x Realtek RTL8211 OCP NIC 3.0 Card In 1
This card does not use an Intel i350 NIC to provide four 1GbE ports as we see in most servers. We also do not have an array of Intel i210 NICs that we see on low-end servers. There are no Broadcom or Marvell equivalents either. Instead, we get four Realtek NICs.
Huawei TaiShan 200 2280 4x Realtek RTL8211 OCP NIC 3.0 Card Out 1
This card has four RTL8211 1GbE NICs. That is somewhat laughable as the card is also set up with an OCP NIC 3.0 x16 connector designed for many times that bandwidth. Normally we see RTL8211 NICs on low-end desktop PCs such as in the sub $200 segment. It is very uncommon to see these in dual-socket servers except sometimes as PHYs for a management port.
Huawei TaiShan 200 2280 4x Realtek RTL8211 OCP NIC 3.0 Card Out 2
关于管理的话题，这也同样耐人寻味。人们可以看到，管理板在两个OCP NIC 3.0插槽之间，这就是我们接下来要看的内容。
华为泰山 200 管理
管理板上有BMC以及提供诸如后I/O的管理端口、RJ45串行端口、USB端口和VGA端口等功能。它的设计不象OCP网卡3.0那样容易更换，但可以看到它使用了类似的边缘连接器。这更多的是为了在服务器的构建过程中进行定制。
Huawei TaiShan 200 2280 BMC 2
The card is most notable because of the chip that is on it. This is a HiSilicon Hi3052. Our best guess is that this is the BMC. We can see a Realtek NIC, storage, and DRAM next to it, similar to what we would see with an ASPEED series BMC.
Huawei TaiShan 200 2280 BMC 1
关于这个芯片有一些奇怪的事情。一个是在谷歌上搜索 "HiSilicon Hi3052 "并没有得到任何结果。仅仅是 "Hi3052 "就能得到HI-3052，一种用于注塑的己烯共聚物。这似乎是一个没有文件的芯片。
接下来，让我们看一下方框图和鲲鹏920的CPU。
华为泰山 200 2280 服务器框图
下面是服务器的框图：
Huawei TaiShan 2280 Block Diagram
Most of these components we have covered in our hardware overview. There were some challenges with the system. As an example, we could find the HNS adapter on the PCIe bus, but it seems very picky with optics.
Huawei Kunpeng 920 2x 48c Topology
The other one is that the Realtek OCP NIC 3.0 did not show up in the system. Here is a quick lscpu output from the Kunpeng 920, and the above is the topology output from when we initially powered on the system.
Huawei Kunpeng 920 2x 48c Lscpu Output
Here is the lshw output for the CPU that we found to be a Kunpeng 920-4826 or a 48 core 2.6GHz part:
*-cpu:0
description: CPU
product: ARM (To be filled by O.E.M.)
vendor: HiSilicon
physical id: 2b
bus info: cpu@0
version: Kunpeng 920-4826
slot: CPU01
size: 2600MHz
capacity: 2600MHz
clock: 100MHz
capabilities: lm
configuration: cores=48 enabledcores=48 threads=48
For those wondering, here is the HiSilicon PCIe Gen4 bridge in this system:
*-pci:0
description: PCI bridge
product: HiSilicon PCIe Root Port with Gen4
vendor: Huawei Technologies Co., Ltd.
physical id: 100
bus info: pci@0000:00:00.0
version: 21
width: 32 bits
clock: 33MHz
capabilities: pci pciexpress msi pm normal_decode bus_master cap_list
configuration: driver=pcieport
resources: irq:27 ioport:1000(size=4096) memory:e6500000-e66fffff ioport:80000200000(size=2097152)
如果朋友们对硬件设备感兴趣，我们可以发布更多的输出。
最后要说的是
这篇文章甚至没有深入介绍鲲鹏 920 Arm CPU，就变成了一篇大文章。我们将在接下来的几天里推出鲲鹏 920 这篇文章。虽然我们仍在努力使系统的性能提高，但固件方面的情况相当清楚。华为服务器的感觉更像是我们在 Cavium / Marvell ThunderX2 系统中体验到的，而不是更现代的 Ampere Arm 服务器。这是一个比正常情况下更多地与我们对抗的服务器。仅仅为了获得我们认为合适的 PCIe Gen4 NVMe SSD 性能，就花了几个小时的时间。我们的一般建议是获得安培 Altra（Max）而不是这个。中国的许多公司正在选择安培而不是这些华为鲲鹏 920 芯片，这似乎是有原因的。安培 Altra Max 每个插座的核心数是两倍，而且是插座式的。
Huawei HiSilicon Kunpeng 920 Next To Ampere Altra Max M128 30 1
In terms of surprises beyond the CPU, the Broadcom SAS controller is one that perhaps we should have expected, but without a Broadcom NIC onboard nor a standard ASPEED BMC, we thought we would get a different solution.
Huawei TaiShan 200 2280 Motherboard Area With Broadcom SAS 2
Aside from the HNS 25GbE RDMA adapter, these two cards were clearly the most interesting in the server. We have a quad 1GbE OCP NIC 3.0 card using less than $4 of NICs and a PCIe Gen4 x16 connector. There is also a BMC based on a chip we could not find information on via Google.
Huawei TaiShan 200 2280 BMC 2
就整体质量而言，华为泰山 200（2280）也令人惊讶。华为的设定是利润率与思科更接近。我们原以为质量会与思科相似。相反，从我们审查的中国国内服务器供应商中，浪潮服务器的质量更高。联想服务器可能比华为高一个档次。那么这个华为服务器将是第三名。泰山 200 有很多非常好的机械设计元素，但在服务器上工作，有相当明显的差异。
Huawei TaiShan 200 2280 Internal Shot Without Risers From Empty OCP NIC 3.0
总的来说，我们希望这次对华为服务器的考察能让大家觉得有用。我们将很快看到鲲鹏 920 的情况，然后随着部件的到达和服务器中更多的内容开始工作，我们将有更多的内容。不过，这里有很多内容，而且这也是一个非常雄心勃勃的项目，所以我们不得不把它分开。
我们意识到评论区可能会对这篇稿子进行热烈讨论，但请保持文明。我们想简单地分享一下，因为这些服务器的存在，然而关于它们的信息却不多。
3080
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
