ASRock 4U10G-ROME2/2T Review
September 27, 2022This time we have a variant from the usual strain of servers, the 12x bay ASRock 4U10G-ROME2/2T (SHOP HERE). Like the name would suggest it’s a 4U server offering support for 10 GPUs, with AMD EPYC ROME processors. Otherwise known as the second generation EPYC series processors. Their naming convention is surprisingly simple! So far… It also supports the 3rd generation AMD EPYC processors, but this system was initially launched prior to third gen CPUs so a little software update and you’re good to go! It’s a dual root system designed for applications in AI, AI Inference and training, plus high-performance computing applications.
We only had this server for about 3 days so we’ll try and give as comprehensive a hardware overview as possible. ASRock Rack, Inc. has been around since about 2013. They made an acronym out of the RACK part of their name, which just seems so 90s… not that we needed another acronym for this industry but here you go:
Reliable Innovation
Advanced Technology
Consistent Quality
Knowledgeable Professionals
We though the RACK part was more in reference to a server rack. Maybe it still is… The ASRock 4U10G-ROME2/2T GPU Server is a dual root server meaning you need both processors installed to power all of the GPUs, as each processor is in charge of 5 of the 10 GPUs. With a single root system, all GPUs would be routed through a single CPU with more PCIe switching happening. A dual root system does remove some of the latency associated with the need for more PCIe switches but does introduce other factors like the reliance for CPU-to-CPU communications through the Quick Path Interconnect.
The ASRock 4U10G-ROME2/2T has an unusual design on the back of the chassis, so we’ll start there. It’s not so much what you see, as what the top section does. It has 4x 1600W PSUs, with 3+1 redundancy, and the port area features dual 10GbE rJ45 ports, a dedicated management port, and the always welcome USB ports all on a separate hinged PSU tray. It’s designed to tilt upwards, past 90 degrees to give easy access to the 10 double-wide GPUs down below. You can see all the connections for additional power to the GPUs on the power distribution board mounted to the bottom of the tilt up section. We thought that was kind of cool. Not sure what our integrators thought of that feature. Just below the top tray are 5x large hot-swap fans to cool the GPUs outfitted with status LEDs and pull warm air out of the case.
There is a dedicated server management LAN port on the ASRock 4U10G-ROME2/2T both front and back. Oh, and those cables in front that give this system a decidedly Star Trek Borg look, allow you to use either the back accessible GbE management and 10GbE LAN Network ports or route them to front for easy access. The dedicated mLAN port provides access to the Server Management Utility, which is ASRock’s server management program suite. Like many AMD EPYC based servers it uses the AST2500 Baseboard management controller with intelligent platform management interface (IPMI), and DMTF’s Redfish API, to leverage common internet and web service standards. All of these utilities help administrators monitor system status and enables them to control servers remotely using a standard browser interface.
The front of the chassis features 12x 3.5-inch hot-swap NVMe or SATA storage bays, natively. It can also support 8x SAS drives, but you will need an 8-port RAID/HBA card. We will mention there is also a sister system to this one: the 4U8G-ROME2/2Ts. It supports only 8 GPUs but up to 24x 2.5-inch storage bays.
On our chassis, below the drive bays, PCIe slot 1 is an OCP 3.0 mezzanine card slot, with PCIe slots 2 and 3 providing a full-height half-length add-on card slot. The PCIe cards are installed on Riser Card Assembly trays that are easily removed from the front of the chassis with two screws. Below that on the right, a slot for an optional LCD module for at-chassis management of the system, but that feature is only available on the 8GPU version with 24X SFF drive support.
Next to that, what ASRock calls an I/O shield with the power Unit ID button, On/Off button, system reset button non-maskable interrupt button and System event button, all with integrated LED buttons. To the right of that panel are two 10Gb Ethernet ports, plus a dedicated 1Gb management port, and on the left 4x USB Type A ports.
Popping off the case cover on the ASRock 4U10G-ROME2/2T, you can see that just behind the drive trays, the system has a Server board which is separated from the GPU switchboard by a bank of 5x hot swap fans with the hinged PSU tray on top. Designed for compute-intensive workloads, the 4U10G-ROME2/2T supports 10 full-height full-width double slot GPUs in 10x PCIe 4.0 x16 slots or up to 20 single slot GPUs using the x8 slots between the x16 slots. 2x PCIe PEX Switchboards on either side of the GPU circuit board facilitate GPU to CPU traffic. As a dual root system half of the GPUs connect to CPU 1 and the other half to CPU2.
This platform is Nvidia-Qualified, meaning it has undergone thermal, mechanical, power, and signal integrity qualification to ensure that the tested GPU is fully functional in this specific server design. The ASRock 4U10G-ROME22T features the ROME2D32GM-2T motherboard and connects the CPUs to a PCIe switchboard housing the GPUs with GPU to CPU traffic regulated by a pair of PEX PCIe switches.
It supports not just the A100 40GB and 80GB GPU versions at up to 300W each, but also the A10, A30, A40, and the RTX A6000—which are all based on Nvidia’s latest Ampere architecture. It can be outfitted with 10 double wide GPUs, like the A100 80GB at up to 300W each, or up to 20 of the single width Nvidia A10 GPUs draining only 175W of juice per GPU.
CPUs supported on this system include the 2nd and 3rd generation AMD EPYC series processors. Also, AMD’s EPYC series with 3D v-Cache technology offering up to 768 MB of cache, which is about 3x more than the standard EPYC CPUs offering 256MB cache. The v-cache CPUs are designed to support certain workflows like electronics board design, fluid dynamics and things of that sort that can benefit from more data queuing at the processor. Thermal design power is at 280W which leaves the whole range of 2nd and 3rd generation EPYC processors available for installation. Standard and 3DS registered and load-reduced memory modules are supported on this system. RDIMMS are limited to module sizes of up to 64GB, while LRDIMMs are limited to 256GB modules. Memory speeds of up to 3200MT/s are supported. Tucked away on the system board under PCIe Slots 2 and 3 is a single PCIe 4.0 x4 for an M.2 drive. It offers fast reliable start-up for the operating system, although with only 1 drive it is not a redundant feature.
As a relatively new entry into the server market, ASRock is clearly producing some high-performance platforms with support for 8 and 10 GPUs in a 4U form factor. They also support the latest 3rd generation CPUs from AMD and Intel, not to mention AMD’s EPYC Processors with 3D vCache technology. Clearly ASRock is up to speed on technology, but the support documentation side was a little lean. Still, you get some high-quality features on this system with support for 3rd gen AMD, NVMe drives, and M.2 drive and support for the latest Nvidia GPUs with Ampere architecture.
By the way, IT Creations carries this server and many others. If you have any questions about our configuration post them in the comments section below. If you’re looking for other servers or professional workstations, check out our website here!
