Nutanix XCP Deep-Dive – Part 2 – Hardware Architecture

vcdx133 This is part 2 of the Nutanix XCP Deep-Dive, covering the Nutanix platform hardware architecture.

This will be a multi-part series, describing how to design, install, configure and troubleshoot an advanced Nutanix XCP solution from start to finish for vSphere, AHV and Hyper-V deployments:

The complete breakdown of the Nutanix (SuperMicro OEM) platform is described in the Hardware Administration and Reference document (need a valid support contract to access)The current Nutanix Hardware Platform page describes the most current models.  The Dell website covers their offering of the Nutanix Dell OEM models as well.  I have taken the information from all of these sources and created a quick reference spreadsheet.  As my NPX design evolves and I get more information, I will add performance and usable capacity columns for each model.  Updated with corrections and Power/Cooling columns.  Note: NX-8150 datasheet does not list 10GBase-T, however the Hardware Admin and Ref. document does, so it has been included.

nx_dell_lenovo_model_list

Nutanix XCP Hardware Architecture

The Nutanix Xtreme Computing Platform has the following hardware architecture:

  • A single 2 Rack Unit chassis that is referred to as a Block
  • Each Block contains one to four Nutanix Nodes (depending upon model)
  • Each Node Mainboard contains CPU, RAM, Network and Storage resources that are completely separate from the rest of the nodes in the Block
  • The storage resources are a mix of InnoLite 64GB SATA DOM USB Flash (for Hypervisor boot and CVM ISO & config files), SSD drives and SATA drives (depending upon model)
  • The SSD and SATA drives plug into the Mid-Plane from the front of the chassis for easy access and replacement
  • Each Node Mainboard sled is connected to the dedicated SSD and SATA drives via a SCSI controller which plugs into the Mid-Plane
  • Each Node Mainboard has 10/100MbE IPMI, 2 to 4 x 1GbE and 1 to 6 10GbE (SFP+ or 10GBase-T depending upon model) network resources
  • Redundant power modules supply power to the Mid-Plane
  • The Mid-Plane distributes power to the fans and Node Mainboards via the same printed circuit board that contains the SCSI controller chipset
  • The performance fans are located between the Mid-Plane and Node sleds to suck cold air from the front of the Block and push heated air from the rear of the chassis, cooling each Node sled in the process

Nutanix_XCP_Hardware_Architecture

Hardware Design Analysis of the NX-1050 (SuperMicro OEM)

I have to say, the SuperMicro OEM design is very subtle and elegant, they had to do some creative thinking to fit that amount of hardware into such a small space.  I can just imagine the arguments within engineering as the design evolved.  Be aware that high density infrastructure has a plethora of power and cooling design issues that every vendor in this space must address.  Here are some of my observations that I would like to draw your attention to:

  • Notice how the length of the block is designed to allow up to 4 nodes plus associated hardware to fit into a 2 Rack Unit space and still fits into a standard server rack?
  • Notice how the ears of the block are also the Power and UID controls?
  • Notice how the mid-plane is also the power distribution and cooling control plane?
  • See how the mid-plane is fabricated as a single printed circuit board, but is actually 4 separate connection domains for storage (one for each node) and one power distribution plane?
  • See how the redundant power supplies are stacked in the center of the block with the shortest power cables to the mid-plane?
  • Notice how all of the internal cabling is neatly loomed and tied down?
  • See how the SCSI controller is vertically mounted and the end of the printed circuit board plugs directly into the mid-plane.  The CVM takes control of this SCSI Controller PCIe slot via Pass-Through mode.
  • Notice how the InnoLite SATA DOM 64GB Flash Drive is vertically mounted near the SCSI controller and plugged into a USB socket on the Node mainboard?
  • Notice how the SSD and SATA drives plug directly into the mid-plane?
  • Notice the perforation of the SSD/SATA drive bezels to allow air to be sucked into the front of the chassis, first cooling the SSD and SATA drives before continuing to the rear of the chassis.
  • Notice how the mid-plane has cut-outs to allow the air to continue to the rear of the chassis.
  • Notice how the side of the block has perforations for additional air to be sucked in.
  • See how the fans are strategically placed to allow the optimum airflow from the front to the rear.
  • Notice how the power supplies have their own fans for cooling?
  • Notice the PVC wave guide to optimise the cooling of the CPUs, which are directly inline for the cleanest airflow.  See the Venturi effect (think airplane wing), so that the greatest suction and airflow will occur through the CPUs, which are the hottest part of the chassis?
  • Notice how the 12 DIMM banks are equally split on either side of the CPU sockets as close as possible for lowest memory access times?

nutanix_1tb_sata_drive nutanix_400gb_ssd nutanix_block_front nutanix_block_midplane_node_a_drives nutanix_block_midplane nutanix_block_power_module nutanix_block_rear nutanix_block_side nutanix_node_satadom nutanix_node_sled_to_midplane_connector nutanix_node

Other Resources

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s