Nutanix XCP Deep-Dive – Part 5 – Installing ESXi Manually with Phoenix

This is Part 5 of the Nutanix XCP Deep-Dive, covering the manual installation of ESXi and CVM with Phoenix.

This will be a multi-part series, describing how to design, install, configure and troubleshoot an advanced Nutanix XCP solution from start to finish for vSphere, AHV and Hyper-V deployments:

You would normally use Foundation to deploy a Nutanix cluster, however, you sometimes need to do this manually when Foundation is having issues.

Use-Case

You have been given a Nutanix XCP block to bring online and you have tried to use Foundation to deploy the cluster.  No bueno.

So you have decided to follow the manual process of installing ESXi first and then customise each ESXi host with Phoenix to end up with three unconfigured Nutanix nodes (Node A, Node B and Node C).  Note: Until a Nutanix Cluster is created, you will not be able to access the Prism UI.

Prerequisites

Assumptions

Accessing the BIOS to set the IPMI IP Address

  1. Connect your VGA monitor and USB Keyboard to Node A.
  2. Power on Node A by pressing the Power-On button for Node A (located on bottom left mounting ear).
  3. Wait for the Nutanix logo to appear and press the “Delete” key to enter BIOS setup mode.
  4. Use the left/right arrow keys to navigate to the “IPMI” tab.
  5. Use the up/down arrow keys to navigate to the “BMC Network Configuration” object and press “Enter”.
  6. Select “Update IPMI LAN Configuration”, select “Yes” and press “Enter”.
  7. Select “Configuration Address Source”, select “Static” and press “Enter”.
  8. Select “Station IP Address”, “Subnet Mask” and “Router/Gateway IP Address” and configure the settings you want.
  9. Press “F4” or use the left/right arrow keys to navigate to the “Save & Exit” tab.
  10. From your Laptop, make sure you can ping the IP address you just configured and access the IPMI Login interface via your Web browser.
  11. Repeat steps 1 to 10 for Nodes B and C.
  12. You should now have three working IPMI IP addresses that you can access via your Web browser.  Proceed to the next section.
  13. IMPORTANT: Do not modify any other BIOS parameters unless instructed to do so by Nutanix Support.
  14. Note: within a functioning ESXi node, you can make these changes by using the “ipmitool” from the ESXi SSH Shell.

BIOS Screenshots:

Nutanix_XCP_BIOS_1 Nutanix_XCP_BIOS_2 Nutanix_XCP_BIOS_3 Nutanix_XCP_BIOS_4 Nutanix_XCP_BIOS_5 Nutanix_XCP_BIOS_6 Nutanix_XCP_BIOS_7 Nutanix_XCP_BIOS_8 Nutanix_XCP_BIOS_9

Connecting to IPMI

  1. From your Laptop open a Web browser and access the IPMI address of Node A (http:// <IPMI IP address>/).
  2. At the Nutanix IPMI login screen, use the credentials “ADMIN/ADMIN” and press “Login”.
  3. Select the “Remote Control” icon on the toolbar, press the “Console Redirection” object and then press the “Launch Console” button.
  4. Accept the Java security warnings and wait for the Console window to open.  You may have to try different browsers and Java versions to get this working – it can be painful.
  5. Press the “Virtual Media” button on the Console toolbar and select “Virtual Storage”.
  6. In the Virtual Storage window, select the “CDROM&ISO” tab, “Logical Drive Type” to “ISO File” and “Open Image”.
  7. Browse to the ESXi ISO image (VMware-VMvisor-Installer-201501001-2403361.x86_64.iso in this case) and press the “Open” button.
  8. In the Virtual Storage window, press “Plug in” and make sure the “Connection Status History” shows “Plug-In OK” and then press the “OK” button.
  9. You are now ready to reset the Node and start the ESXi installation process.
  10. Repeat steps 1 to 9 for Nodes B and C.
  11. You should now have three Console windows (to Nutanix nodes) with the ESXi ISO mounted and ready for installation.  Proceed to the next section.

IPMI Screenshots:

Nutanix_XCP_IPMI_1 Nutanix_XCP_IPMI_2 Nutanix_XCP_IPMI_3  Nutanix_XCP_IPMI_5Nutanix_XCP_ESXi_1Nutanix_XCP_IPMI_4

Installing ESXi 5.5

  1. From the previous section, you should have a Console window open with the ESXi ISO image mounted, ready to install.
  2. Press the “Virtual Media” button on the Console toolbar and select “Virtual Keyboard”.  Depending upon your Laptop OS and system configuration, this maybe required for pressing the function keys during the install.
  3. Select the “Power Cycle Server” option from the “Power Control” icon on the Console toolbar.
  4. Wait for the ESXi ISO image to boot and present the EULA screen.  Press “F11” to accept and continue.
  5. In the “Select a Disk to Install or Upgrade” screen, select the “InnoLite SATADOM” storage device and press “Enter”.
  6. If the “ESXi and VMFS Found” window appears, select “Install ESXi, overwrite VMFS datastore” and press “Enter”.
  7. In the “Keyboard layout” window, select “US Default” and press “Enter”.
  8. In the “Enter a root password” window, you must type “nutanix/4u” and press “Enter”.  Otherwise CVM will not be able to connect to ESXi.
  9. In the “Confirm Install” window, press “F11” to install.
  10. Press the “Virtual Media” button on the Console toolbar and select “Virtual Storage”.
  11. Wait for the “Installation Complete” window to appear.
  12. In the Virtual Storage window, press “Plug out” and make sure the “Connection Status History” shows “Plug-Out OK” to unmount the ESXi ISO image.
  13. In the “Installation Complete” window, press “Enter” to reboot.
  14. Repeat steps 2 to 13 for Nodes B and C.
  15. You should now have three Nutanix Nodes with ESXi successfully installed.  Proceed to the next section.

ESXi Install Screenshots:

Nutanix_XCP_ESXi_1 Nutanix_XCP_ESXi_2 Nutanix_XCP_ESXi_3 Nutanix_XCP_ESXi_4 Nutanix_XCP_ESXi_5 Nutanix_XCP_ESXi_6 Nutanix_XCP_ESXi_7

Using Phoenix to install the Controller VM and Customise ESXi

  1. From the previous section, you should have a Console window open with ESXi successfully installed.
  2. Press the “Virtual Media” button on the Console toolbar and select “Virtual Storage”.
  3. In the Virtual Storage window, select the “CDROM&ISO” tab, “Logical Drive Type” to “ISO File” and “Open Image”.
  4. Browse to the Phoenix ESXi ISO (phoenix-2.0_ESX_NOS-4.0.2.1.iso in this case) and press the “Open” button.
  5. In the Virtual Storage window, press “Plug in” and make sure the “Connection Status History” shows “Plug-In OK” and then press the “OK” button.
  6. Select the “Power Cycle Server” option from the “Power Control” icon on the Console toolbar.
  7. Wait for the “Nutanix Installer” screen to appear, then select “Configure Hypervisor” and “Clean SVM”  and then press the “Start” button.
  8. Wait for the Nutanix Installation process to complete (“reboot now” message will appear).
  9. In the Virtual Storage window, press “Plug out” and make sure the “Connection Status History” shows “Plug-Out OK” to unmount the Phoenix ESXi ISO image.
  10. In the Console window, press “Y” and then “Enter” to reboot.
  11. After ESXi boots, you will see the message “INSTALLING-PLEASE-BE-PATIENT” on the Console screen (ESXi DCUI).  This is a Nutanix VIB executing the first boot installation script, configuring ESXi and registering the CVM vmx file.
  12. Repeat steps 2 to 11 for Nodes B and C.
  13. You should now have three Nutanix Nodes with ESXi customised and CVM successfully installed.  Proceed to the next section.

Phoenix Install Screenshots:

Nutanix_XCP_IPMI_6Nutanix_XCP_Phoenix_1 Nutanix_XCP_Phoenix_2 Nutanix_XCP_Phoenix_3 Nutanix_XCP_Phoenix_4

Configure IP addresses

During this manual install process, everything is configured with DHCP.  Even if you configure a static IP for vmk0 initially after the ESXi installation, the Phoenix installation process will configure vmk0 with DHCP.  So you need to touch each node to configure the static IP addresses you require.

  1. From the previous section, you should have the consoles open to three Nutanix Nodes with ESXi customised and CVM successfully installed.
  2. Press “F2” on the IPMI Console and configure the “Management” IP to be a static IP address.  Then logout.
  3. Use the vSphere Client to connect to the static IP address of the ESXi host.
  4. From vSphere Client, open the console to the CVM and login with the credentials “nutanix/nutanix/4u”.
  5. Edit the file: “/etc/sysconfig/network-scripts/netconf/ifcfg-eth0” and modify/add “BOOTPROTO=”none”“, “NETMASK=”N.N.N.N”“, “IPADDR=”N.N.N.N”“, “GATEWAY=”N.N.N.N”” to the correct IP Address settings of the CVM.
  6. You now have an unconfigured Nutanix node with static IP addresses that is ready to be joined to a Nutanix cluster.
  7. Repeat steps 2 to 6 for Nodes B and C.
  8. Note: within a functioning cluster you can make these changes by using the URL http:// <IPv6 LinkLocal>:2100/cluster_init.html from your Web browser.

Nutanix CVM Screenshots:

Nutanix_XCP_CVM_IP_Change_Manual_1 Nutanix_XCP_CVM_IP_Change_Manual_2 Nutanix_XCP_CVM_IP_Change_Manual_3 Nutanix_XCP_CVM_IP_Change_URL

Configuring a Nutanix Cluster via NCLI

  1. From the vSphere Client, open the console to the CVM and login with the credentials “nutanix/nutanix/4u”.
  2. Run the command “cluster status” and verify that the cluster is unconfigured.
  3. Run the command “cluster -s <Node_A_CVM_eth0_IP>,<Node_B_CVM_eth0_IP>,<Node_C_CVM_eth0_IP> create” to create the cluster
  4. Run the command “ncli cluster add-to-name-servers servers=<DNS_IP>” to configure DNS.
  5. Run the command “ncli cluster add-to-ntp-servers servers=<NTP_IP>” to configure NTP.
  6. Run the command “ncli cluster set-external-ip-address external-ip-address=<CLUSTER_IP>” to configure the Cluster IP address.
  7. Run the command “cluster status” and verify that the cluster has been created.
  8. You now can access the Prism UI and continue configuring the Storage Pool and Container(s) for the cluster.

What Happened?

You have performed the following:

  1. Accessed the BIOS to statically set the IPMI network address.
  2. Accessed the IPMI URL to launch the Java Console and mounted the ISO images to install ESXi/Phoenix.
  3. Installed ESXi on the InnoLite SATADOM (64GB USB Flash Drive plugged into the Node motherboard).
  4. Installed Phoenix – which installed a VIB file (with first boot script) for ESXi on SATADOM and associated CVM files.
  5. The Phoenix installation launches via the script embedded in the VIB when ESXi is first booted and customises ESXi and installs the Controller VM.
  6. Coincidentally, William Lam has recently written a nice post about how VIBs can be used to run scripts within ESXi, which is what Nutanix are doing here.
  7. Configured static IP addresses for each ESXi vmk0 and each CVM eth0 interface – this is because the manual install process uses DHCP by default.
  8. Used NCLI to create a Nutanix cluster.

If you connect to ESXi using the vSphere Client, you can see the results of the VIB first time boot script:

  • vSS “vSwitchNutanix” with vSS Portgroup “svm-iscsi-pg” and VMkernel port “vmk-svm-iscsi-pg”
  • SSH enabled with SSH alert masked (UserVars.SuppressShellWarning)
  • Defunct iSCSI Software adapter
  • Advanced Software Settings for NFS (Net.TcpipHeapMax, Net.TcpipHeapSize, NFS.MaxVolumes, etc.)
  • Controller VM boot from ISO with LSI2008 SCSI Adapter in Passthrough mode and CPU/Memory reservations
  • NTP configured
  • Virtual Machine Startup/Shutdown enable for CVM

If you run a partition management program like GParted, you will see the following partitions on each Node:

  • InnoLite SATADOM fat16 Partitions – where the ESXi boot and CVM (Service VM) files reside
  • SSD ext4 Partitions – where the Nutanix Home, Cassandra, OpLog, Content Cache and Extent Store reside
  • HDD ext4 Partitions – where the Curator and Extent Store reside

vSphere Client to ESXi host images:

vsphere_client_1 vsphere_client_2 vsphere_client_3 vsphere_client_4 vsphere_client_5 vsphere_client_6 vsphere_client_7 vsphere_client_8 vsphere_client_9 vsphere_client_10 vsphere_client_11 vsphere_client_12 vsphere_client_13 vsphere_client_14 vsphere_client_15 vsphere_client_16

GParted images:

gparted1 gparted2 gparted3 gparted4 gparted5

4 thoughts on “Nutanix XCP Deep-Dive – Part 5 – Installing ESXi Manually with Phoenix

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s