This is the VMware NSX for vSphere Design Deep Dive. I have aggregated all of the design considerations I could find that need to be assessed in a VMware NSX-v architecture design. Brevity and bullet-points are used to keep the information concise and readable. If you want more information on a concept use the Additional Resources section at the end.
This post will be updated with additional information as part of the NSX Link-O-Rama. If you have content to contribute, post a comment below.
I have separated the design decisions into the areas specified by the VCDX-NV blueprint.
- Physical network has the design qualities of simplicity, performance, scalability and reliability
- Physical network must be configured for 1600 MTU or larger (for additional VXLAN header size)
- Physical network should support Layer 2 or Layer 3 Multi-Cast (for Hybrid or Multi-Cast VXLAN Replication)
- VMware vCenter Server 5.5 or later
- VMware vSphere ESXi 5.1 or later (5.1 requires Multi-Cast VXLAN Replication)
- VMware Tools in all supported VMs for Endpoint and Data Security
- Limitation of Scale – 1 vCenter to 1 NSX Manager, Multi vCenter/NSX Manager not supported at this time
- Limitation of Scale – Configuration Maximums: 12 Clusters per NSX Manager
- NSX Manager does not support High Availability – have to rely upon vSphere HA
- Dedicated Layer 2 VLAN for NSX Management (single broadcast domain between NSX Management components)
- Single Logical Interface (LIF) per Logical Switch or VLAN
- Cannot route on Layer 2 bridged interfaces
- VLAN LIFs can only be on one VDS
- You have a set of business requirements that has led to the selection of VMware NSX-v (Conceptual Model and Logical Design) and now you are ready for the Physical Design.
Physical Design Decisions
A. NSX Infrastructure Components
- NSX-v version? Understand the feature disparity between 6.0 and 6.1, eg. ECMP, L2VPN.
- Cluster Design: Separate or combined Compute, Management, Edge Clusters?
- Management Cluster: Dedicated or shared vCenter (with NSX Manager)?
- Logical Routing Deployment: Physical Router as Next Hop (1 Tier) or ESG as Next Hop (2 Tier)?
- NSX Controllers: How many deployed? For Availability or Performance?
- vSphere DRS VM Anti-affinity rules for NSX Controllers: yes or no? (ESG anti-affinity rules created automatically)
- vSphere HA VM Restart priority for NSX Manager, NSX Controllers, ESGs, DLR Controllers: which priority?
- Datastores for NSX Manager, Controllers, DLR Control VMs and ESGs?
- NSX Edge Form Factors: Compact to X-Large
- NSX Edge Availability: Standalone, Active/Standby or ECMP?
- Understanding of Traffic Flows: North/South and East/West
- NTP Master?
- Impact of NSX-v Configuration Maximums
- Naming Standard of NSX components?
- Capacity Planning: how will your design handle future expansion with respect to configuration maximums and current limitations?
- Future Proofing: how will your design adapt to new features such as NSX Federation?
- Management/Monitoring/Troubleshooting: SNMP, RSPAN, Syslog, NetFlow, API integration (including vROps NSX Plugin and LogInsight NSX Plugin)
- Backup/Recovery of NSX infrastructure
- Multi-site BC/DR considerations
B. Virtual Networking
- NSX Controller Address Assignment: Dedicated IP Pool?
- VTEP Address Assignment: Manual (after DHCP timeout), IP Pools or DHCP (required if you have Per Rack VLANs – with IP Helper per ToR switch to fix the Layer 2 boundary issue)
- Routing of vMotion and IP Storage: Yes or no?
- IP Addressing Schema: NSX Manager, NSX Controllers, DLR, ESG, VMkernel (Storage, vMotion, Management, VTEP)
- Logical Switches: quantity and function?
- Interior Routing Protocol: Static, BGP or OSPF
- Exterior Routing Protocol: Static, BGP, OSPF or IS-IS
- VXLAN Transport Zones: Single Global or multiple?
- VXLAN Replication mode: Unicast, Hybrid or MultiCast? (Ease of deployment versus performance and scale)
- VXLAN: Segment ID/VNI Pool Range?
- VTEPs per Host: Single or Multiple? (this depends upon the VDS Teaming and Failover design)
- Distributed Logical Router: VXLAN-VLAN Bridging?
- Number of VDS: One or more?
- VDS Uplink Teaming & Failover: Originating Port, Source MAC Hash, LACP, IP Hash or Explicit Failover? (LBT not supported)
- VDS Network I/O Control: Configured?
- VDS Traffic Shaping: Configured?
- ESG Load Balancing: One-armed or Inline?
C. Physical Networking
- Physical Network: Traditional 3 tier (Core, Aggregation, Access), Collapsed Core or Leaf and Spine?
- Traditional 3 Tier: Spanning Tree, LAG?
- QoS: end-to-end or edge? Integrated with Network I/O Control?
- Multi Rack redundancy: Span Management and Edge Clusters across 3 racks?
- Top of Rack Switch redundancy: One or two ToR switches?
- Host Design: Specific hardware specification for Management and Edge Clusters?
- Per Rack VLANs: Yes or no? (VLANs not extended between racks; different subnets per rack with leaf and spine switched network, requires VMware Request for Qualification)
- Physical network MTU size: 1600 MTU or larger?
- Physical network Multicast: Layer 2 or Layer 3 Multi-Cast? (not required for Unicast)
- External Network/Transport VLANs: How many?
D. NSX Security
- Endpoint and Data Security: AntiVirus and File Scanning polices? (Windows OS supported, Linux is not)
- Edge Services being used: Firewall, NAT, DHCP Server/Relay, Routing Protocols, IPSec/SSL VPNs and L2 VPN?
- L3-L7 features: NSX Native or 3rd party?
- NSX Policy Repository: Via NSX API or local?
- Distributed Firewall Micro-segmentation: Yes or no?
- ESG Failover states: Impact to Firewall & NAT, DHCP Server, Routing Protocols, IPSec/SSL VPNs and L2 VPN?
- Advanced Security: Insertion, Chaining and Steering?
E. NSX – CMP Integration
- Cloud Management Platform integration: vRA or something else?
- Multi-Tenant: Single ESG per tenant with how many Logical Switches?
- Network topologies for Blueprints: External, Routed, Private, NAT
- Security policies for Blueprints
- L4-L7 services for Blueprints
- NSX Link-O-Rama
- VMware Education will be releasing their VMware NSX Design and Deploy course soon
- VMware NSX for vSphere (NSX-V) Network Virtualization Design Guide Version 2.1
- Pluralsight course “NSX configuration and administration” & “Network Services” by Jason Nash
- Reference Design: Deploying NSX with Cisco UCS and Nexus 9000 Infrastructure by VMware
- VMware Brocade Network Virtualization Technical Paper by VMware
- Network Virtualization with Dell Infrastructure and VMware Reference Architecture by VMware