NSX-v – Design Deep Dive

This is the VMware NSX for vSphere Design Deep Dive.  I have aggregated all of the design considerations I could find that need to be assessed in a VMware NSX-v architecture design.  Brevity and bullet-points are used to keep the information concise and readable.  If you want more information on a concept use the Additional Resources section at the end.

This post will be updated with additional information as part of the NSX Link-O-Rama.  If you have content to contribute, post a comment below.

I have separated the design decisions into the areas specified by the VCDX-NV blueprint.

Requirements/Constraints

  • Physical network has the design qualities of simplicity, performance, scalability and reliability
  • Physical network must be configured for 1600 MTU or larger (for additional VXLAN header size)
  • Physical network should support Layer 2 or Layer 3 Multi-Cast (for Hybrid or Multi-Cast VXLAN Replication)
  • VMware vCenter Server 5.5 or later
  • VMware vSphere ESXi 5.1 or later (5.1 requires Multi-Cast VXLAN Replication)
  • VMware Tools in all supported VMs for Endpoint and Data Security
  • Limitation of Scale – 1 vCenter to 1 NSX Manager, Multi vCenter/NSX Manager not supported at this time
  • Limitation of Scale – Configuration Maximums: 12 Clusters per NSX Manager
  • NSX Manager does not support High Availability – have to rely upon vSphere HA
  • Dedicated Layer 2 VLAN for NSX Management (single broadcast domain between NSX Management components)
  • Single Logical Interface (LIF) per Logical Switch or VLAN
  • Cannot route on Layer 2 bridged interfaces
  • VLAN LIFs can only be on one VDS

Assumptions

  • You have a set of business requirements that has led to the selection of VMware NSX-v (Conceptual Model and Logical Design) and now you are ready for the Physical Design.

Physical Design Decisions

A. NSX Infrastructure Components

  1. NSX-v version? Understand the feature disparity between 6.0 and 6.1, eg. ECMP, L2VPN.
  2. Cluster Design: Separate or combined Compute, Management, Edge Clusters?
  3. Management Cluster: Dedicated or shared vCenter (with NSX Manager)?
  4. Logical Routing Deployment: Physical Router as Next Hop (1 Tier) or ESG as Next Hop (2 Tier)?
  5. NSX Controllers: How many deployed? For Availability or Performance?
  6. vSphere DRS VM Anti-affinity rules for NSX Controllers: yes or no? (ESG anti-affinity rules created automatically)
  7. vSphere HA VM Restart priority for NSX Manager, NSX Controllers, ESGs, DLR Controllers: which priority?
  8. Datastores for NSX Manager, Controllers, DLR Control VMs and ESGs?
  9. NSX Edge Form Factors: Compact to X-Large
  10. NSX Edge Availability: Standalone, Active/Standby or ECMP?
  11. Understanding of Traffic Flows: North/South and East/West
  12. NTP Master?
  13. Impact of NSX-v Configuration Maximums
  14. Naming Standard of NSX components?
  15. Capacity Planning: how will your design handle future expansion with respect to configuration maximums and current limitations?
  16. Future Proofing: how will your design adapt to new features such as NSX Federation?
  17. Management/Monitoring/Troubleshooting: SNMP, RSPAN, Syslog, NetFlow, API integration (including vROps NSX Plugin and LogInsight NSX Plugin)
  18. Backup/Recovery of NSX infrastructure
  19. Multi-site BC/DR considerations

B. Virtual Networking

  1. NSX Controller Address Assignment: Dedicated IP Pool?
  2. VTEP Address Assignment: Manual (after DHCP timeout), IP Pools or DHCP (required if you have Per Rack VLANs – with IP Helper per ToR switch to fix the Layer 2 boundary issue)
  3. Routing of vMotion and IP Storage: Yes or no?
  4. IP Addressing Schema: NSX Manager, NSX Controllers, DLR, ESG, VMkernel (Storage, vMotion, Management, VTEP)
  5. Logical Switches: quantity and function?
  6. Interior Routing Protocol: Static, BGP or OSPF
  7. Exterior Routing Protocol: Static, BGP, OSPF or IS-IS
  8. VXLAN Transport Zones: Single Global or multiple?
  9. VXLAN Replication mode: Unicast, Hybrid or MultiCast? (Ease of deployment versus performance and scale)
  10. VXLAN: Segment ID/VNI Pool Range?
  11. VTEPs per Host: Single or Multiple? (this depends upon the VDS Teaming and Failover design)
  12. Distributed Logical Router: VXLAN-VLAN Bridging?
  13. Number of VDS: One or more?
  14. VDS Uplink Teaming & Failover: Originating Port, Source MAC Hash, LACP, IP Hash or Explicit Failover? (LBT not supported)
  15. VDS Network I/O Control: Configured?
  16. VDS Traffic Shaping: Configured?
  17. ESG Load Balancing: One-armed or Inline?

C. Physical Networking

  1. Physical Network: Traditional 3 tier (Core, Aggregation, Access), Collapsed Core or Leaf and Spine?
  2. Traditional 3 Tier: Spanning Tree, LAG?
  3. QoS: end-to-end or edge?  Integrated with Network I/O Control?
  4. Multi Rack redundancy: Span Management and Edge Clusters across 3 racks?
  5. Top of Rack Switch redundancy: One or two ToR switches?
  6. Host Design: Specific hardware specification for Management and Edge Clusters?
  7. Per Rack VLANs: Yes or no? (VLANs not extended between racks; different subnets per rack with leaf and spine switched network, requires VMware Request for Qualification)
  8. Physical network MTU size: 1600 MTU or larger?
  9. Physical network Multicast: Layer 2 or Layer 3 Multi-Cast? (not required for Unicast)
  10. External Network/Transport VLANs: How many?

D. NSX Security

  1. Endpoint and Data Security: AntiVirus and File Scanning polices? (Windows OS supported, Linux is not)
  2. Edge Services being used: Firewall, NAT, DHCP Server/Relay, Routing Protocols, IPSec/SSL VPNs and L2 VPN?
  3. L3-L7 features: NSX Native or 3rd party?
  4. NSX Policy Repository: Via NSX API or local?
  5. Distributed Firewall Micro-segmentation: Yes or no?
  6. ESG Failover states: Impact to Firewall & NAT, DHCP Server, Routing Protocols, IPSec/SSL VPNs and L2 VPN?
  7. Advanced Security: Insertion, Chaining and Steering?
  8. RBAC?

E. NSX – CMP Integration

  1. Cloud Management Platform integration: vRA or something else?
  2. Multi-Tenant: Single ESG per tenant with how many Logical Switches?
  3. Network topologies for Blueprints: External, Routed, Private, NAT
  4. Security policies for Blueprints
  5. L4-L7 services for Blueprints

Additional Resources

5 comments

  1. jbone says:

    Great stuff ! Thanks much for taking the time to post this.

  2. […] NSX-v: Design Deep Dive by Rene van den Bedem (VCDX133) […]

  3. Vian says:

    does this article still valid for today?

    • vcdx133 says:

      Certainly is – you will have to read the release notes for 6.3.3 to address the new features that have appeared since this was written, eg. Cross-vCenter NSX.

  4. […] I tweeted about my disappointment and within minutes I had several VMware community heroes responding and offering help. The one and only quadruple(!) VCDX (Rene van den Bedem), VCDX and vCloud Director guru Yves Sandfort, fellow Dutch VCDX Daniel Zuthof, my (VCDX) colleagues at ITQ and many more. Everyone was really supportive and willing to review my documentation set. Sending my documents to 10+ people would likely produce a wide range of (perhaps even conflicting) feedback and at this moment, I feel like I really need to be laser focussed on what’s (f)actually lacking in my application. Because Rene obviously knows how to approach VCDX, and because he gave me some critical advice during my previous VCDX endeavour, I took him up on his offer and sent him my full application package. Rene was extremely fast and thorough in providing his feedback. Within a couple of hours he made clear what was missing. I’m not going to discuss specifics here but a lot of items relate back to stuff Rene has blogged about extensively. Check out this great post: https://vcdx133.com/2015/05/18/nsx-v-design-deep-dive/! […]

Leave a Reply to vcdx133 Cancel reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Powered by WordPress.com.
%d bloggers like this: