NSX-v – Design Deep Dive

This is the VMware NSX for vSphere Design Deep Dive.  I have aggregated all of the design considerations I could find that need to be assessed in a VMware NSX-v architecture design.  Brevity and bullet-points are used to keep the information concise and readable.  If you want more information on a concept use the Additional Resources section at the end.

This post will be updated with additional information as part of the NSX Link-O-Rama.  If you have content to contribute, post a comment below.

I have separated the design decisions into the areas specified by the VCDX-NV blueprint.

Requirements/Constraints

• Physical network has the design qualities of simplicity, performance, scalability and reliability
• Physical network must be configured for 1600 MTU or larger (for additional VXLAN header size)
• Physical network should support Layer 2 or Layer 3 Multi-Cast (for Hybrid or Multi-Cast VXLAN Replication)
• VMware vCenter Server 5.5 or later
• VMware vSphere ESXi 5.1 or later (5.1 requires Multi-Cast VXLAN Replication)
• VMware Tools in all supported VMs for Endpoint and Data Security
• Limitation of Scale – 1 vCenter to 1 NSX Manager, Multi vCenter/NSX Manager not supported at this time
• Limitation of Scale – Configuration Maximums: 12 Clusters per NSX Manager
• NSX Manager does not support High Availability – have to rely upon vSphere HA
• Dedicated Layer 2 VLAN for NSX Management (single broadcast domain between NSX Management components)
• Single Logical Interface (LIF) per Logical Switch or VLAN
• Cannot route on Layer 2 bridged interfaces
• VLAN LIFs can only be on one VDS

Assumptions

• You have a set of business requirements that has led to the selection of VMware NSX-v (Conceptual Model and Logical Design) and now you are ready for the Physical Design.

Physical Design Decisions

A. NSX Infrastructure Components

1. NSX-v version? Understand the feature disparity between 6.0 and 6.1, eg. ECMP, L2VPN.
2. Cluster Design: Separate or combined Compute, Management, Edge Clusters?
3. Management Cluster: Dedicated or shared vCenter (with NSX Manager)?
4. Logical Routing Deployment: Physical Router as Next Hop (1 Tier) or ESG as Next Hop (2 Tier)?
5. NSX Controllers: How many deployed? For Availability or Performance?
6. vSphere DRS VM Anti-affinity rules for NSX Controllers: yes or no? (ESG anti-affinity rules created automatically)
7. vSphere HA VM Restart priority for NSX Manager, NSX Controllers, ESGs, DLR Controllers: which priority?
8. Datastores for NSX Manager, Controllers, DLR Control VMs and ESGs?
9. NSX Edge Form Factors: Compact to X-Large
10. NSX Edge Availability: Standalone, Active/Standby or ECMP?
11. Understanding of Traffic Flows: North/South and East/West
12. NTP Master?
13. Impact of NSX-v Configuration Maximums
    
14. Naming Standard of NSX components?
15. Capacity Planning: how will your design handle future expansion with respect to configuration maximums and current limitations?
16. Future Proofing: how will your design adapt to new features such as NSX Federation?
17. Management/Monitoring/Troubleshooting: SNMP, RSPAN, Syslog, NetFlow, API integration (including vROps NSX Plugin and LogInsight NSX Plugin)
18. Backup/Recovery of NSX infrastructure
19. Multi-site BC/DR considerations

B. Virtual Networking

1. NSX Controller Address Assignment: Dedicated IP Pool?
2. VTEP Address Assignment: Manual (after DHCP timeout), IP Pools or DHCP (required if you have Per Rack VLANs – with IP Helper per ToR switch to fix the Layer 2 boundary issue)
3. Routing of vMotion and IP Storage: Yes or no?
4. IP Addressing Schema: NSX Manager, NSX Controllers, DLR, ESG, VMkernel (Storage, vMotion, Management, VTEP)
5. Logical Switches: quantity and function?
6. Interior Routing Protocol: Static, BGP or OSPF
7. Exterior Routing Protocol: Static, BGP, OSPF or IS-IS
8. VXLAN Transport Zones: Single Global or multiple?
9. VXLAN Replication mode: Unicast, Hybrid or MultiCast? (Ease of deployment versus performance and scale)
10. VXLAN: Segment ID/VNI Pool Range?
11. VTEPs per Host: Single or Multiple? (this depends upon the VDS Teaming and Failover design)
12. Distributed Logical Router: VXLAN-VLAN Bridging?
13. Number of VDS: One or more?
14. VDS Uplink Teaming & Failover: Originating Port, Source MAC Hash, LACP, IP Hash or Explicit Failover? (LBT not supported)
15. VDS Network I/O Control: Configured?
16. VDS Traffic Shaping: Configured?
17. ESG Load Balancing: One-armed or Inline?

C. Physical Networking

1. Physical Network: Traditional 3 tier (Core, Aggregation, Access), Collapsed Core or Leaf and Spine?
2. Traditional 3 Tier: Spanning Tree, LAG?
3. QoS: end-to-end or edge?  Integrated with Network I/O Control?
4. Multi Rack redundancy: Span Management and Edge Clusters across 3 racks?
5. Top of Rack Switch redundancy: One or two ToR switches?
6. Host Design: Specific hardware specification for Management and Edge Clusters?
7. Per Rack VLANs: Yes or no? (VLANs not extended between racks; different subnets per rack with leaf and spine switched network, requires VMware Request for Qualification)
8. Physical network MTU size: 1600 MTU or larger?
9. Physical network Multicast: Layer 2 or Layer 3 Multi-Cast? (not required for Unicast)
10. External Network/Transport VLANs: How many?

D. NSX Security

1. Endpoint and Data Security: AntiVirus and File Scanning polices? (Windows OS supported, Linux is not)
2. Edge Services being used: Firewall, NAT, DHCP Server/Relay, Routing Protocols, IPSec/SSL VPNs and L2 VPN?
3. L3-L7 features: NSX Native or 3rd party?
4. NSX Policy Repository: Via NSX API or local?
5. Distributed Firewall Micro-segmentation: Yes or no?
6. ESG Failover states: Impact to Firewall & NAT, DHCP Server, Routing Protocols, IPSec/SSL VPNs and L2 VPN?
7. Advanced Security: Insertion, Chaining and Steering?
8. RBAC?

E. NSX – CMP Integration

1. Cloud Management Platform integration: vRA or something else?
2. Multi-Tenant: Single ESG per tenant with how many Logical Switches?
3. Network topologies for Blueprints: External, Routed, Private, NAT
4. Security policies for Blueprints
5. L4-L7 services for Blueprints

Additional Resources

• NSX Link-O-Rama
• VMware Education will be releasing their VMware NSX Design and Deploy course soon
• VMware NSX for vSphere (NSX-V) Network Virtualization Design Guide Version 2.1
• Pluralsight course “NSX configuration and administration” & “Network Services” by Jason Nash
  
• Reference Design: Deploying NSX with Cisco UCS and Nexus 9000 Infrastructure by VMware
• VMware Brocade Network Virtualization Technical Paper by VMware
• Network Virtualization with Dell Infrastructure and VMware Reference Architecture by VMware