Supermicro GPU SuperServer 420GP-TNAR+ Review

The Supermicro GPU SuperServer 420GP-TNAR+ with 8x SXM form factor NVIDIA A100 Tensor core GPUs is a powerful machine. That’s right, 8x high-performance GPUs in a 4U chassis. It’s more like a little cabinet you might find at IKEA with all these little drawers.

Now, what can this Supermicro GPU SuperServer 420GP-TNAR+ computational powerhouse be used for? Definitely AI, and Deep Learning, certainly you can dabble in some high-performance computing as well. Bitcoin mining? Yes. Scientific simulations? Most definitely.  There is actually a TNAR version without the + but there is only a very minor difference, which we will get to. Cliffhanger? Not really. Anyways it features dual 3^rd gen Intel Xeon scalable processors and is compatible with Intel Optane 200 series memory modules for up to 12TB of memory.

Now that IKEA reference. This system has 3x main drawers, two on the front housing the 1U compute node on top, and the 3U GPU node on the bottom with the 4x integrated 9 cm fans.

There’s also a switch tray for some low-profile PCIe slots. Back to the front, the lower drawer has locking screws that you will need to remove before you can pull down on the levers to either side of the chassis to release the safety locks on both sides. The top compute node has a locking screw and then you can pull the levers to either side out. Starting from the left top, a small control panel with tell-tale lights for general information for system status, Network Interface Controller, Unit ID, and the UID button at the bottom. Then a power on button, Baseboard Management Controller reset button, 2x USB 3.0 ports, BMC LAN port and a VGA port.

The 6x hot-swap drive bays are all hybrid bays that can be outfitted with Gen 4 NVMe, SAS, or SATA drive types depending on your workload. That said, if you do go SAS, you will need an HBA controller.

Next to the drive bays, 2x PCIe Gen 4.0 x16 PCIe slots, one to CPU 1 the other to CPU 2.  Below, 4x 9 cm fans.

Around back, the other removeable drawer is the switch tray for the PCIe slots. As this is a dual root system, the PCIe slots are divided between the two processors. Those 5x slots on the left map to CPU 1 and the other slots to the right map to CPU 2. We will mention that slot on the far left is for an Advanced I/O Module AKA, an OCP 3.0 mezzanine card slot.

For managing the system Supermicro offers a few software utilities including, Supermicro SuperCloud Composer, Supermicro Server Manager or SSM, and Supermicro SuperDoctor 5. There are a few others for updates, diagnostics, and automation. The Supermicro Server Manager is the all-in-one utility for managing multiple generations of supermicro servers from a single console. However, you will need licensing software for each target node you want to access, and that number can be up to 10,000 nodes, if you’re using Linux. That said, SSM is compatible with several generations of Windows Server too, including 2022. There is also a Supermicro IPMI Tools Suite for basic system configuration and updates using a command line interface or graphics user interface. A number of plugins are also available for use with VMware vCenter, Microsoft System Center Operations Manager, and NAGIOS. For local host monitoring there’s SuperDoctor 5, which is more of a one-to-one management utility.

Each processor has a separate bus so any expansion cards access one of those CPUs directly and the CPUs then communicate between each other via the Intel Ultra Path Interconnect or UPI. You might notice on the back of the system between the PCIe slots there is a panel of 12x Switchboard LEDs. Below all that, 4x PSUs. In this case with the TNAR+, the PSUs are redundant 3000W Titanium rated. On the TNAR without the + the PSUs are 2200W Platinum rated PSUs. At this point we were thinking perhaps the GPUs on the + are 80GB and those on the other are 40GB but no. Both platforms support the 80GB GPUs.

One we remove the 3U GPU node, you can see those large fans pulling fresh air into the chassis. Then a bank of 6x PCIe switches, and then, the 8x SXM form factor NVIDIA HGX A100 80GB GPUs in all their glory. Well at least the towering heat sinks on top of the cards. Air shrouds to either side, ensure air movement over and through the GPUs, then out the back of the chassis.

There are 6x PCIe switches controlling the traffic between the CPUs, GPUs, and potentially NVMe storage up front. The GPUs themselves are connected via NVSwitch and 3^rd generation NVLink at about 600GB/s, at least for this generation. NVLink speed has basically tripled with 5^th generation NVLink technology at up to 1,800GB/s. Lastly, we will say that no, NVSwitch is not the same as those PCIe PLX switches as it offers much higher bandwidth and lower latency. Also proprietary to NVIDIA.

The NVIDIA HGX A100 GPUs are designed to handle massive data sets and are powered by Ampere architecture. Designed for Data Center applications, NVIDIA says it provides up to 20x higher performance over the previous generation. Of course, it is now about 5 years old and 2x generations removed from Blackwell, which made its debut just a few short months ago at the end of 2024. That said, the A100 can still deliver a memory bandwidth of over 2TB/s using HBM2e memory, a 1.7x improvement over the previous generation. It was originally billed as the worlds’ fastest but not anymore. As a comparison, with Blackwell, memory bandwidth has increased to 4TB/s.

There are actually 2x form factors for this GPU; the SXM form factor supported on this system and a PCIe version. We won’t bore you with the details but suffice it to say the SXM version does offer slightly better performance on that memory bandwidth at 2,039GB/s compared to 1,935GB/s but also higher power consumption at up to 400W compared to 300W on the PCIe card. That’s about it! Well also, you can only connect 2x PCIe GPUs using an NVLink bridge. Both do still provide 7x MIGs or Multi-instance GPU at up to 10GB per instance. Each of those instances behaves like an independent GPU so you can run smaller workloads simultaneously. MIG provides a secure environment for multiple users or customers allowing the ability to scale resources as needed.

Now, that 1U Compute node. It also has the drive bays and two PCIe slots on the far right. Our unit was outfitted with 2x 1.6TB NVMe U.2 SFF drives and a single 1.92TB NVMe U.2 SFF drive. This unit connects to that GPU node via those PLX switches. The compute node has dual sockets that can be outfitted with 3^rd generation Intel Xeon Scalable processors. with 8 to 40 physical cores and 16 to 80 virtual threads, each. These CPUs feature 8-channel memory architecture with 16 memory module slot per CPU and 32x active memory module slots with both processors installed.

Supported memory includes ECC Registered or Load-Reduced DDR4 DIMM modules. Using RDIMMs or LRDIMM memory, the system can deliver up to 8TB of memory. Also compatible, are Intel Optane Persistent Memory 200-Series for up to 12TB when combined with D-RAM as Optane memory is installed 1x module per memory channel. The other memory module slots can be outfitted with Registered DIMMs. Memory speeds are variable depending on the CPU installed, the actual memory module, and the memory configuration, all of which can impact memory speed. CPUs in the Platinum and Gold categories offer the fastest memory speeds at up to 3200MT/s compared to Silver, which tops out at up to 2667MT/s. Aside from the memory speeds supported by the CPU and actual memory modules, peak memory speed is achieved with one DIMM per channel.

With Intel Optane Persistent memory, the system can provide more memory capacity in memory mode without persistence provided by the integrated flash storage. That flash storage can also be utilized with persistence in App Direct Mode and provides access to data closer to the CPU for improved performance on large data sets. We might as well cut that description short as Optane memory has already been discontinued. Do you think it was just too expensive or too complicated to implement? Seemed like a good idea at the time.

Next, the switch tray has all 8x of the other PCIe slots supported on this system and an AIOM module slot. It features a PCIe 4.0 switchboard for network communications. The only other difference between that TNAR+, and the TNAR without the + is the basic TNAR has a PCIe 3.0 bus for those 10x PCIe slots instead of PCIe 4.0. All in all, there are 10x PCIe slots when you count the 2x up front located on the Compute node. Those 2x also offer a PCIe 4.0 x16 low profile slot. Those PCIe slots can be outfitted with a number of options from Mellanox for ports and link throughput.

Supported Mellanox controllers offer high data throughput and extremely low latency for AI, high-performance computing, and data-intensive infrastructures. You will surely need that throughput to get the most out of that bank of 8x top-of-the-line NVIDIA A100 Tensor-core GPUs.

This system was configured with a single ConnectX-5 NIC at 100GbE with dual QSFP28 ports. ConnectX-6 cards are also supported and offer 200GbE with single or dual port QSFP56 ports. ConnectX-7 with 400G is perhaps a step too far as those require a PCIe 5.0 interface and this system tops out at PCIe 4.0. There are also a number of options for that Advanced I/O Module on the far right at the back of system for different ports and data throughput. Ours was configured with a 1GbE Advanced I/O Module with 2x RJ45 ports but it can also be outfitted with a Mellanox card offering 2x HDR 200GbE QSFP56 ports.

And there you have it! The Supermicro GPU SuperServer 420GP-TNAR+. If you are looking for a GPU server, there is a lot of wiggle room to support a number of workloads, depending on how it’s configured. And with MIG supported by those GPUs, you can run a number of workloads simultaneously. If you want more information on this platform, contact IT Creations! We have tons of servers, workstations, and components.