Nexus 9000 Cisco Live

Prévia do material em texto

Everyone 
in this 
room is a 
GENIUS
2
What are Best Practices ?
Learning from Others 
Mistakes
3
Learning from your 
mistakes makes you 
SMART
Learning from others 
mistakes makes you 
GENIUS
4
vPC Best Practices and 
Design on NXOS
Nazim Khan, CCIE#39502 (DC/SP)
Network Consulting Engineer, Data Center Group
BRKDCT-2378
Session Focus
• Best Practices and Designs for vPC
• Nexus 2000 (FEX) will only be 
addressed from vPC standpoint
• Fabricpath / vPC+ Overview
• vPC with FCOE
• vPC with VXLAN
• vPC with ACI
.
Pick the great from the good
We Are Not Covering
• vPC troubleshooting
• Scalability
• Fabricpath
• vPC+
• VXLAN
• FCoE
• ACI
Related Sessions at Cisco Live Berlin
Session Id Session Name
BRKDCT-2404 VXLAN deployment models - A practical perspective
BRKDCT-3313 Fabricpath Operations and Troubleshooting
BRKDCT-2458
Nexus 9000/7000/6000/5000 Operations and 
Maintenance Best Practices
BRKACI-2601 Real World ACI Deployment and Migration
BRKDCT-2333 Data Centre Network Failure Detection
9
Agenda
• Feature Overview
• Configuration Best Practices
• Design Best Practices
• vPC Operations and Upgrade
• vPC with Fabric Technologies
• Reference Material
10
Data Center Technology Evolution
FabricPath with vPC+
2010
2009
VPC
2008
STP
2013-2014
MPLS, OTV,
LISP
VXLAN
MPLS, OTV,
LISP
2014-2015
ACI
2010
FEX with vPC
11
Why vPC?
13
vPC
there’s 
something 
about
14
Role of vPC in the Evolution of Data Center
• vPC launched in 2009
• Deployed by almost 95% of Nexus customers
• Used to redundantly connect network entities at the
edge of the Fabric
− Dual-homed servers (bare metal, blades, etc.)
− Network services (Firewalls, Load Balancers, etc.)
Unified Fabric 
15
Agenda
• Feature Overview
− Concepts and Benefits
− Terminology
16
vPC Feature Overview
vPC Concept & Benefits
S1 S2
S3
• No Blocked Ports, More Usable Bandwidth, Load Sharing
• Fast Convergence
STP
S2S1
S3
vPC Logical Topology
S3
S1 S2
vPC Physical Topology
17
Feature Overview
vPC Terminology
Layer 3 Cloud
vPC Member 
PortvPC
Orphan 
Device
Orphan 
Port
vPC 
Peer
CFS
vPC Domain
Peer-Link
vPC Peer
Keepalive Link
S1
S3
S2
18
SW3 SW4
vPC1 vPC2
vPC_PLink
vPC Peer-keepalive 
vPC Failure Scenario 
vPC Peer-Keepalive Link up & vPC Peer-Link down
P S
Keepalive Heartbeat
Secondary vPCS
P Primary vPC
Suspend secondary 
vPC Member Ports
vPC peer-link failure (link loss): 
• vPC peer-keepalive up
• Status of other vPC peer known
• Both peers Active
• Secondary vPC peer disables all vPC’s
• Traffic from vPC primary.
• Orphan devices connected to secondary peer will 
be isolated
For Your
Reference
S2S1
19
vPC Failure Scenario – Dual Active
vPC Peer-Keepalive down followed by vPC Peer-Link down
P S
Secondary vPCS
P Primary vPC
P
SW3 SW4
vPC1 vPC2
vPC_PLink
vPC Peer-keepalive 
Traffic Loss / Uncertain Traffic 
Behavior 
1. vPC peer-keepalive DOWN
2. vPC peer-link DOWN
3. DUAL-ACTIVE or SPLIT BRAIN
• vPC primary peer remains primary and 
secondary peer becomes operational primary 
role
• Result in traffic loss / uncertain traffic behavior
• When links are restored, the operational 
primary (former secondary) keeps the primary 
role & former primary becomes operational 
secondary
For Your
Reference
S1 S2
20
Agenda
• vPC Configuration Best Practices
− Building a vPC domain
− Domain-ID
− Peer-Link
− Peer-Keepalive Link
− Spanning-Tree
− Peer-switch
− Private VLAN (PVLAN)
− Auto-recovery
− Object tracking
21
vPC Configuration Best Practices
Building a vPC domain – Configuration Steps
CFS
1. Define domains
2. Establish Peer Keepalive connectivity
3. Create a Peer link
4. Create vPCs
5. Make Sure Configurations are Consistent
(Order does Matter!)
S1 S2
S3
22
vPC Configuration Best Practices
vPC Domain-ID
vPC Domain 10
vPC Domain 20
• The vPC peer devices use the vPC domain ID to 
automatically assign a unique vPC system MAC 
address
• You MUST use unique Domain id’s for all vPC 
pairs defined in a contiguous layer 2 domain
! Configure the vPC Domain ID – It should be unique within the layer 2 
domain
NX-1(config)# vpc domain 20
! Check the vPC system MAC address
NX-1# show vpc role
<snip>
vPC system-mac : 00:23:04:ee:be:14
S1 S2
S3 S4
S5
23
vPC Configuration Best Practices
vPC Peer-Link
• vPC Peer-link should be a point-to-point connection
• Peer-Link member ports can be 10/40/100GE interfaces
• Peer-Link bandwidth should be designed as per the vPC
• vPC imposes the rule that peer-link should never be blocking
S1 S2
S3
S1 S2
S3
24
vPC Configuration Best Practices
vPC Peer-Keepalive link
Preference Nexus 7X00 / 
9500 series
Nexus 9300 /6000 / 
5X00 / 3X00 series
1 Dedicated link(s) 
(1GE/10GE LC)
mgmt0 interface
2 mgmt0 interface Dedicated link(s) 
(1GE/10GE LC)
3 L3 infrastructure L3 infrastructure
Recommendations 
(in order of 
preference):
25
vPC Configuration Best Practices
vPC Peer-Keepalive link – Dual Supervisors
Standby Management Interface
Active Management Interface
vPC1 vPC2
vPC_PL
Management 
Network
Management Switch
vPC_PKL
vPC_PKL• When using dual supervisors and mgmt0 interfaces 
to carry the vPC peer-keepalive, DO NOT connect
them back to back between the two switches 
• Only one management port will be active a given point 
in time and a supervisor switchover may break keep-
alive connectivity
• Use the management interface when you have an out-
of-band management network (management switch in 
between) 
For Your
Reference
26
vPC Configuration Best Practices
Spanning Tree (STP)
• All switches in Layer 2 domain should run either Rapid-PVST+ or MST
• Do not disable spanning-tree protocol for any VLAN
• Always define the vPC domain as STP root for all VLAN in that domain 
STP is running to manage 
loops outside of vPC domain, 
or before initial vPC 
configuration !
S1 S2
S4
S3
S5
27
vPC Configuration Best Practices
vPC Peer-Gateway
• Allows a vPC switch to act as the active 
gateway for packets addressed to the peer 
router MAC
• Keeps forwarding of traffic local to the vPC node 
and avoids use of the peer-link 
• Allows Interoperability with features of some NAS 
or load-balancer devices
N7k(config-vpc-domain)# peer-gateway
S1 S2
S4S3
28
vPC Configuration Best Practices
vPC Peer-switch
Without Peer-switch
•STP for vPCs controlled by vPC primary.
•vPC primary send BPDU’s on STP designated ports
•vPC secondary device proxies BPDU’s to primary
With Peer-switch
• Peer-Switch makes the vPC peer devices to appear as a 
single STP root
• BPDUs processed by the logical STP root formed by the 2 
vPC peer devices
N7k(config-vpc-domain)# peer-switch
Primary
vPC
Secondary
vPC
BPDUs
Primary
vPC
Secondary
vPC
29
vPC Configuration Best Practices
PVLAN on vPC
• PVLAN configuration across both VPC switches 
should be identical
• PVLAN configuration not supported on Peer-Link 
• Type-1 Compatibility Check
• Port mode is a type-1check
• vPC leg brought down if PVLAN port mode 
different on vPC legs 
• Type-2 Compatibility Check
• PVLAN will bring down mismatched tuple
S1 S2
vPC Primary vPC Secondary
P P
PVLAN-
PROMISC 
(3500, 3501)
PVLAN-
PROMISC 
(3500, 3501)
C
Community 
VLAN
Note : This feature is currently not supported on N9X00
30
Pvlan Isolated trunk 
vPC Configuration Best Practices
PVLAN VPC type 1 Consistency Check
S1 S2
vPC Primary vPC Secondary
P P
S1 S2
vPC Primary vPC Secondary
I I
Pvlan 
Promiscuous 
trunk 
Type 1 Consistency 
Failure
S1 S2
vPC Primary vPC Secondary
TI
S3
S3S3
31
vPC Configuration Best Practices
PVLAN VPC type 2 Consistency Check
S1 S2
vPC Primary vPC Secondary
P P
S1 S2
vPC Primary vPC Secondary
I I
Type 2 Consistency 
Failure
S1 S2
vPC Primary vPC Secondary
I
S3
S3S3
I
PVLAN-
PROMISC 
(10, 201)
PVLAN-
PROMISC 
(10, 201)
Secondary 
Trunk (2,31) 
(3,30), (4,100)
Secondary 
Trunk (2,31) 
(3,30), (4,100)
Secondary 
Trunk (3,31) 
(2,30), (4,100)
Secondary 
Trunk (2,31) 
(3,30), (4,100)
32
vPC Configuration Best Practices
vPC auto-recovery
1. vPC peer-link down : S2 - secondary shuts all its vPC member ports
2. S1 down : vPC peer-keepalive link down : S2 receives no keepalives
3. After 3 keepalive timeouts, S2 changes role and brings up its vPC
vPC Primary
vPC Secondary
P
S
P
Operational 
Primary
S2S1
S3
P S
S1 S2
S3
S1 S2
S3
P S
33
vPC Configuration Best Practices
vPC auto-recovery
Auto-recovery addresses two cases of single switch behavior
•Peer-link fails and after a while primary switch (or keepalive link) fails 
•Both VPC peers are reloaded and only one comes back up 
How it works 
•If Peer-link is down on secondary switch, 3 consecutive missing peer-keepalives will 
trigger auto-recovery 
•After reload (role is ‘none established’) auto-recovery timer (240 sec) expires while 
peer-link and peer-keepalive still down, autorecovery kicks in 
•Switch assumes primary role 
•VPCs are brought up bypassing consistency checks
For Your
Reference
Nexus(config)# vpc domain 1
Nexus(config-vpc-domain)# auto-recovery
34
vPC Configuration Best Practices
Why Object-Tracking ?
Primary Secondary
• Modules hosting peer-link and uplink fail on 
the vPC primary
• Peer-Link is down and vPC Secondary 
shut all its vPC
• Auto-Recovery does not kick in as peer-
keepalive link is active
• Traffic is black holed
S1 S2
S3
S5S4
35
vPC Configuration Best Practices
Object-tracking
S1 S2
• Object Tracking triggered when the track object 
goes down
• vPC object tracking, tracks both peer-link and 
uplinks in a list of Boolean OR
• Traffic forwarded over the remaining vPC peer
! Track the vpc peer link 
track 1 interface port-channel11 line-protocol 
! Track the uplinks 
track 2 interface Ethernet1/1 line-protocol 
track 3 interface Ethernet1/2 line-protocol
! Combine all tracked objects into one. 
! “OR” means if ALL objects are down, this object will go down
track 10 list boolean OR 
object 1 
object 2 
object 3
! If object 10 goes down on the primary vPC peer, 
! system will switch over to other vPC peer and disable all local vPCs
vpc domain 1
track 10
• Suspends the vPCs on the impaired device
S4 S5
S3
36
vPC Configuration Best Practices
Root
Blocked
BPDUs
Network
Network Network
Network
BPDUs
EdgeEdge
Network
Network
BPDUs
Malfunctioning
switch
Stopped receiving 
BPDUS!
BA Inconsistent
BA Inconsistent
%STP-2-BRIDGE_ASSURANCE_BLOCK: Bridge Assurance blocking port Ethernet2/48 
VLAN0700
switch# show spanning vl 700 | in -i bkn
Eth2/48 Altn BKN*4 128.304 Network P2p *BA_Inc
Spanning Tree Bridge Assurance
Stopped receiving 
BPDUS!
Spanning Tree Bridge Assurance
• Turns STP into a bidirectional protocol
• Ensures spanning tree fails “closed” rather than “open”
• All ports with “network” port type send BPDUs regardless of state 
• If network port stops receiving BPDUs, port is placed in BA-Inconsistent state 
(blocked)
Almost like a routing protocol…
%STP-2-BRIDGE_ASSURANCE_BLOCK: Bridge Assurance blocking port 
Ethernet2/48 VLAN0700.
switch# sh spanning vl 700 | in -i bkn
Eth2/48 Desg BKN*4 128.304 Network P2p *BA_Inc
For Your
Reference
38
vPC Configuration Best Practices
vPC & Bridge Assurance (BA)
• STP Bridge Assurance is enabled by default on vPC Peer-Link
• DON’T disable Bridge Assurance on vPC Peer-link
• NO Bridge Assurance on vPC member ports (even with peer-switch)
39
vPC Configuration Best Practices
Unidirectional Link Detection (UDLD)
• Light-weight Layer 2 failure detection protocol 
• Designed for detecting:
• One-way connections due to physical or soft failure 
• Mis-wiring detection (loopback or triangle)
• Cisco proprietary, but listed in informational RFC 5171
• Runs on any single Ethernet link, even inside bundle
• Centralized implementation in switching platforms 
• Message interval: 7 - 90 sec (default: 15 seconds)
• Detection: 2.5 x interval + timeout value (4 sec)  ~ 41 sec
TxRx
TxRx
For Your
Reference
40
vPC Configuration Best Practices
UDLD with vPC
• UDLD NOT recommended on vPC peer-link
• UDLD NOT recommended on vPC member ports if LACP is used
• UDLD only in normal mode on vPC member ports if required
41
Agenda
• vPC Design Best Practices
− Mixed Hardware across vPC Peers
− FHRP with vPC
− Hybrid topology (vPC and non-vPC)
− vPC and Network Services
− vPC Fex Supported Topologies
− Physical port vPC
− vPC as Data Center Interconnect (DCI)
− Dynamic Routing over VPC
− vPC and Multicast
42
Mixed Hardware across vPC Peers : Line Cards
Always use identical line cards on either sides of the peer link and VPC legs !
vPC Peer-link
S1 S2
vPC Primary vPC Secondary
M2M1
vPC
Examples
vPC Peer-link
S1
N7000
S2
N7700
vPC Primary vPC Secondary
F3
vPC
F3
F2E F2E
43
Design Best Practices
X Y vPC
N9K-X9636PQ N9K-X9432PQ
N9K-X9564PX N9K-X9464PX
N9K-X9564TX N9K-X9464TX
N9K-X9536PQ N9K-X9736PQ
vPC Peer-link
S1
N9500
S2
N9500
vPC Primary vPC Secondary
Y
vPC
X
X Y
Mixed Hardware across vPC Peers : Nexus 9500
44
Design Best Practices
Mixed Hardware across vPC Peers : Chassis & Supervisors
• N7000 and N7700 in same vPC Construct -Supported
• VDC type should match on both peer device
• vPC peers can have mixed SUP version* (SUP1, SUP2, SUP2E)
• N5500 and N5600 in same vPC Construct –Not Supported
*Recommended only for short period such as migration
N5500 N5600
vPC Primary vPC Secondary
S1
N7000
S2
N7700
vPC Primary vPC Secondary
S1 S2
45
Design Best Practices
FHRP with vPC
FHRP 
“Standby”: 
Active for 
shared L3 MAC
FHRP 
“Active”: 
Active for 
shared L3 MAC
• FHRP in Active/Active mode with vPC
• No requirement for aggressive FHRP timers
• Best Practice : Use default FHRP timers
S1 S2
S3 S4
46
Design Best Practices
PL3
L2
OSPF/EIGRP
Primary
vPC
Secondary
vPC
OSPF/EIGRP
VLAN 99
Backup Routing Path
• Point-to-point dynamic routing protocol 
adjacency between the vPC peers to 
establish a L3 backup path to the core 
through PL in case of uplinks failure
• Define SVIs associated with FHRP as 
routing passive-interfaces in order to avoid 
routing adjacencies over vPC peer-link
• A single point-to-point VLAN/SVI (aka 
transit vlan) will suffice to establisha L3 
neighbor
• Alternatively, use an L3 point-to-point link 
between the vPC peers to establish a L3 
backup path
Routing Protocol Peer
P
Use one transit vlan to establish L3 routing 
backup path over the vPC peerlink in case L3 
uplinks were to fail, all other SVIs can use 
passive-interfaces
S2S1
S4S3
S5
P
P
P
47
Design Best Practices
Hybrid topology (vPC and non-vPC)
S1 S2
S3 S4
vPC Primary vPC Secondary
vPC1
Bridge Priority
VLAN 1  4K
VLAN 2  8K
Bridge Priority
VLAN 1  8K
VLAN 2  4K
STP Root 
VLAN 1
STP Root 
VLAN 2
STP Root 
VLAN 1
VLAN 2
VLAN 1
(blocked)
VLAN 2
(blocked)
• Supports hybrid topology where vPC and non-vPC are connected to the same vPC domain
• Need additional configuration parameters : spanning-tree pseudo-information
• STP pseudo configuration takes precedence over global STP configuration
peer-switch
48
Design Best Practices
ASA Cluster
ASA Cluster Mode
• Use unique vPC for ASA Cluster Data Links to vPC domain
• Use vPC per ASA device for Cluster Control Link (CCL) to vPC domain
• Leverage peer-switch configuration
Cluster 
Control Link
Cluster 
Data Link
49
Nexus 2000 (FEX) Straight-Through Deployment with VPC
• Port-channel connectivity from the server
• Two Nexus switches bundled into a vPC 
pair
• Suited for servers with Dual NIC and 
capable of running Port-Channel
Fex 100
Fabric Links
Fex 101
VPC
HIF HIF
S1 S2
50
Nexus 2000 (FEX)Active-Active Deployment with VPC
Fabric Links
HIFHIF
Fex 100
Fex 101
S1 S2
• Fabric Extender connected to two Nexus 
5X00 / 6000
• Suited for servers with Single NIC or 
Dual NIC not having port-channel 
capability.
• Scale implications of less FEX per 
system and less VPC
Note : 
• This design is currently not supported on Nexus 9X00
• Nexus 7X00 will support this from release 7.2
51
• N7X00 can support up to 64 FEXs
• N7X00 supports only 15 Active-Active FEX in 7.2(0)D1(1) 
• Straight-Through FEX and Active-Active FEX cannot exist on the 
same ASIC instance
• Layer 3 HIF ports are not supported with Active-Active FEX
• Active-Active FEX is not supported with vPC+
Nexus 2000 (FEX) Active-Active Scale & 
Limitations (N7X00)
Nexus 2000 (FEX) - Enhanced VPC
Fabric Links
Fex 100 Fex 101
S1 S2
• Port-channel connectivity to dual-homed 
FEXs
• From the server perspective a single access 
switch with port-channel support – each line 
card supported by redundant supervisors
• Ideal design for a combination of single 
NIC and Dual NIC servers with port-
channel capability 
• Scale implications of less FEX per 
system and less VPC
Note : 
This design is currently not supported on N7000 / N7700 and 
N9X00
HIFHIF
53
Nexus 2000 (FEX) Active-Active (Unsupported)
54
Physical Port vPC
• vPC configuration on a physical Layer 2 port as opposed to a port-channel
• Front panel ports and FEX ports connected to F2/F2e/F3 only
• Improves scaling as separate PC interface not created for single-link VPC leg
• Key benefit: more than 1000 host facing VPCs with FEX
vPC domain
FEX101 FEX102
e101/1/1 e102/1/1Po1
Port-channel vPC
VPC1 VPC1
Po1
vPC domain
FEX101 FEX102
e101/1/1 e102/1/1
Physical port vPC
VPC1 VPC1
interface e101/1/1
switchport
vpc 1
lacp mode active
55
Long Distance 
Dark Fiber
DC 1 DC 2
C
O
R
E
A
G
G
R
A
C
C
E
S
S
Server Cluster
C
O
R
E
A
G
G
R
A
C
C
E
S
S
Server Cluster
vPC domain 10 vPC domain 20
vPC domain 21vPC domain 11
vPC - Data Center Interconnect(DCI)
Rootguard
B
F
N
E
BPDUguard
BPDUfilter
Network port
Edge or portfast
- Normal port type
R
802.1AE (Optional)
E
- -
-
- -
E
E
E
E
F
F
F
F-
-
- -
-
-
-
B
N N
N
NN
N
R
R
-
RR
RR
R
R
NN
B
-
E
56
Design Best Practices
vPC as Data Center Interconnect (DCI)
PROS
• vPC is easy to configure and it provides robust and resilient interconnect solution
CONS
• Maximum of only two Data Centers can be interconnected
• Layer 3 peering between Data Centers cannot be done through vPC and separate 
links are required
57
Design Best Practices
vPC -Data Center Interconnect (DCI)
• vPC Domain id for vPC layers should be UNIQUE
• BPDU Filter on the edge devices to avoid BPDU propagation
• STP Edge Mode to provide fast Failover times
• No Loop must exist outside the vPC domain
• No L3 peering between Nexus 7000 devices (i.e. pure layer 2)
58
Dynamic routing over vPC ?
59
Dynamic routing over vPC
Use Case 1 : Firewall at Aggregation layer
L3 Cloud
S1
S2
FW-A FW-B
Dynamic Peering Relationship
• Peering Firewalls in routed mode over vPC
• Firewalls may be in active-standby mode
• Static routing / L3 P2P links NOT required
• External and internal traffic traverse same
port channel to firewall.
60
Dynamic routing over vPC
Use Case 2 : Remote Orphan Site Peering in DCI Deployment
• vPC as Data Center Interconnect (DCI)
• Each Switch has routing adjacency with both
vPC device in other DC
• Each DC connected to a remote site by
orphan port
• Remote sites forms routing adjacency with
both peers of its directly connected DC
Remote Site 1 Remote Site 2
S1 S2
S3 S4
61
Dynamic Routing over vPC
New Supported Designs
Dynamic routing over vPC
Supported Designs
P
PP
PP
PP
Note : Supported only in Nexus 7X00 on F3 and F2E Line Cards starting from release 7.2. 
Supported on Nexus 9X00 in ACI mode
Currently not supported on Nexus 5X00, Nexus 3X00, Nexus 9X00 (standalone mode), Nexus 7000 M-series Line card
Layer 3 services devices with vPC Layer 3 over DCI - vPC 
63
Dynamic routing over vPC
Supported Designs
P
PP
P
PP
STP inter-connection using a vPC VLAN Orphan device with vPC peers over vPC VLAN
Note : Supported only in Nexus 7X00 on F3 and F2E Line Cards starting from release 7.2. 
Supported on Nexus 9X00 in ACI mode
Currently not supported on Nexus 5X00, Nexus 3X00, Nexus 9X00 (standalone mode), Nexus 7000 M-series Line card
64
Dynamic routing over vPC
Supported Designs
Peering with vPC peers over FEX vPC host interfaces
PP
P
Note : Supported only in Nexus 7X00 on F3 and F2E Line Cards starting from release 7.2(0)D1(1) 
Currently not supported on Nexus 5X00, Nexus 3X00, Nexus 9X00 (standalone mode), Nexus 7000 M-series Line card
65
Dynamic Routing over vPC
Unsupported Designs
66
Dynamic routing over vPC
Unsupported Design
• The routing metric on S1 is less than the routing 
metric on S2 (preferred path using S1).
• Traffic from A to B may hash to S2. This traffic 
will need to traverse to peer-link to get to B 
through S1.
• Due to the vPC loop avoidance rule S1 will not 
allow traffic to flow to B.
Router1
Router2
Po1
Po2
A
B
Po100
Int VLAN 10
Metric 20
Int VLAN 10
Metric 10
Int VLAN 10
Int VLAN 20
Int VLAN 20Int VLAN 20
S2
SVI
SVI
S1
Peering across vPC interfaces with unequal L3 
metrics
67
Dynamic routing over vPC
Configuration
L3 over vPC Configuration on Nexus 7x00 platform
Command: Layer3 peer-router 
Mode: config-vpc-domain
Default: Disabled
•Requirements
• Command configured on both the peers.
• “Peer-Gateway” should be enabled.
• Peer link should be up.
• Both peer should run image supporting L3 over vPC feature.
•Auto Enabling “Peer-Gateway”
• If “Layer3 peer-router” command is enabled without “Peer-Gateway” a syslog will be
displayed to enable “Peer-Gateway”.Need to configure on BOTH the 
peers
68
Dynamic routing over vPC
Example Configuration and Verification on Nexus 7x00
PP
P
vpc domain 200
peer-keepalive destination 
10.10.12.42 source 10.10.12.52
peer-gateway
layer3 peer-router
vpc domain 200
peer-keepalive destination 
10.10.12.52 source 10.10.12.42
peer-gateway
layer3 peer-router
show vpc brief 
Peer Gateway : Enabled
Operational Layer3 Peer : Enabled
(output truncated for display) 
show vpc brief 
Peer Gateway : Enabled
Operational Layer3 Peer : Enabled
(output truncated for display) 
69
Benefits of Dynamic Routing over vPC
• No Static routes
• No Parallel links
• No design changes and loss of business
• Route peering across vPC’s over existing infrastructure
• Routing between vPC DCI
• Most wanted by majority vPC customers
70
B
SVI 1
IP Y
VIP A
SVI 1
IP Z
VIP A
S1 S2
A
Router
Dynamic Routing over vPC
• Don’t attach routers to VPC domain via L2 port-channel
• Common workarounds:
• Individual L3 links for routed traffic
• Static route to FHRP VIP
Router
L3 ECMP
Router
SVI 1
IP Y
VIP A
SVI 1
IP Z
VIP A
SVI 1
IP Y
VIP A
SVI 1
IP Z
VIP A
SVI 2
IP X
SVI 2
IP X
SVI 2
IP X
Static Route to VIP A
S1 S1 S2S2
Devices without L3 over vPC support
71
Design Best Practices
vPC and Multicast
Source • vPC supports PIM-SM only
• vPC uses CFS to sync IGMP state
• Sources in vPC domain
− both vPC peers are forwarders
− Duplicates avoided via vPC loop-avoidance logic
• Sources in Layer 3 cloud
− Active forwarder elected on unicast metric
− vPC Primary elected active forwarder in case metric 
are equal
Receivers
S2S1
Source
72
Agenda
• vPC Operations and Upgrade
− vPC Self Isolation
− vPC Shutdown
− Graceful Insertion and Removal
− ISSU / ISSD with vPC
73
3. S2 takes over as operational Primary and S1 is isolated from the vPC domain
vPC Configuration Best Practices
vPC Self-Isolation
1. Error Triggered : All Line cards Fail or All Vlans’s down on peer-link
2. S1 sends “self-isolation” message through the peer-keepalive
vPC Primary
vPC Secondary
P
S
P
Operational 
Primary
S2S1
S3
P S
S1 S2
S3
Error 
Triggered
Self- Isolate
ISOLATED
S1 S2
S3
P S
74
vPC Configuration Best Practices
Example Configuration and Verification on Nexus 7x00
vPC domain 100
peer-keepalive destination 
10.126.216.44
peer-gateway
self-isolation
vPC domain 100
peer-keepalive destination 
10.126.216.41
peer-gateway
self-isolation
sh vPC brief
vPC domain id : 100
Self-isolation : Enabled
(output truncated for display) 
sh vPC brief
vPC domain id : 100
Self-isolation : Enabled
(output truncated for display) 
75
vPC Configuration Best Practices
vPC Self-Isolation
• vPC self-isolation is turned OFF by default
• No Impact on vPC operation if sellf-isolation enabled
• Functional only when enabled on both vPC peers.
• Not part of vPC type-1 and type-2 consistency checks
76
vPC Shutdown
vPC Configuration Best Practices
• Isolates a switch from the vPC complex
• Isolated switch can be debugged, reloaded, or 
even removed physically, without affecting the 
vPC traffic going through the non-isolated switch
Primary Secondary
vPC
S1 S2
S3
switch# configure terminal 
switch(config)# vpc domain 100
switch(config-vpc)# shutdown
77
vPC vPC
system mode maintenance
One command!
Pre-change System Snapshot
Change window begins
Graceful Insertion and Removal
78
vPC vPC
One command!
Pre/Post-change Snapshot Comparison
Change window complete
system mode normal
Graceful Insertion and Removal
79
Graceful Insertion and Removal
• Flexible framework providing a comprehensive, systemic method to isolate a 
node.
• Configuration profile foundation in NX-OS
• Initial support for:
• vPC/vPC+
• ISIS
• OSPF
• EIGRP
• BGP
• Interface
• Per VDC on Nexus 7x00
Platform Release
Nexus 5x00/6000 NX-OS 7.1
Nexus 7x00 NX-OS 7.2
Nexus 9000 NX-OS 7.X
80
ISSU / ISSD with vPC
• ISSU is the recommended system upgrade in a 
multi-device vPC environment
• vPC system can be independently upgraded with 
no disruption to traffic 
• Upgrade is serialized and must be run one peer at 
a time (config lock will prevent synchronous 
upgrades)
• Configuration is locked on “other” vPC peer during 
ISSU 
• Similar process of downgrades (ISSD) 
• Check ISSU / ISSD compatibility matrix & ensure 
ISSU is supported from current to target release 
5.2(x) / 6.2(x)
81
Agenda
• vPC with Fabric Technologies
− vPC with Fabricpath (vPC+)
− vPC with FCOE
− vPC with VXLAN
− vPC with ACI
82
FabricPath: an Ethernet Fabric
Shipping on Nexus 7x00, Nexus 600x and Nexus 5x00
N7K(config)# interface ethernet 1/1
N7K(config-if)# switchport mode fabricpath
• Eliminates Spanning tree limitations
• High resiliency, fast network re-convergence
• Any VLAN, Anywhere in the Fabric
• Connect a group of switches using an arbitrary topology
• With a simple CLI, aggregate them into a Fabric
FabricPath
83
• Physical architecture of vPC and vPC+ is the same from the access edge
• Functionality/Concepts of vPC and vPC+ are the same
• Key differences are addition of Virtual Switch ID and Peer Link is a FP Core Port 
• vPC+ is not supported on Nexus 9X00 & Nexus 3X00 Series
Architecture of vPC and FabricPath with vPC+
VPC vs VPC+
FP VLAN’s
vPC+vPC
FP Port
FP
CE Port
CE VLAN’s
CE
84
Dynamic Routing over vPC+
P Routing Protocol Peer
N55xx, N56xx, 
N6000 
Router/ Firewall 
Dynamic Peering Relationship
• Layer 3 devices can form routing adjacencies with 
both the vPC+ peers over vPC
• The peer link ports and VLAN are configured in 
FabricPath mode.
• N55xx, N56xx, N6000 support this design with 
IPv4/IPv6 unicast and PIM-SM multicast
• This design is not supported on N7X00
Fabricpath Link
vPC
P P
P
Fabricpath Core
85
vPC with FCoE
Unified Fabric Design
• vPC with FCoE is ONLY supported between hosts and 
N5X00 or N5X00 & N2232 pairs.
• Must follow specific rules:
• A ‘vfc’ interface can only be associated with a 
single-port port-channel.
• While the port-channel configurations are the 
same on both switches, the FCoE VLANs are 
different.
• FCoE VLANs are ‘not’ carried on the vPC peer-link 
(automatically pruned):
• FCoE and FIP ethertypes are ‘not’ forwarded 
over the vPC peer link.
• vPC carrying FCoE between two FCF’s is NOT 
supported.
• Best Practice: Use static port channel rather than LACP 
with vPC and boot from SAN.
[If NX-OS is prior to 5.1(3)N1(1)]
VLAN 10,30
VLAN 10,20
STP Edge Trunk
VLAN 10 ONLY HERE!
Fabric A Fabric B
LAN Fabric
Nexus 5000 
FCF-A
Nexus 5000 
FCF-B
vPC contains only 2 X 10GE 
links – one to each Nexus 5X00
86
 Problems being addressed:
• VLAN scale – VXLAN extends the L2 segment ID field to 24-bits, potentially 
allowing for up to 16 million unique L2 segments over the same network
• Layer 2 segment elasticity over Layer 3 boundary – VXLAN encapsulates L2 
frame in IP-UDP header
 High Level Technology Overview:
• MAC-in-UDP encapsulation.
• Leverages multicast in the transport network to simulate flooding behavior for 
broadcast, unknown unicast and multicast in the same segment
• Leverage ECMP to achieve optimal path usage over the transport network
Why VXLAN ?
87
VXLAN Packet Format
FCS
Outer 
Mac Header
Outer 
IP Header
UDP Header VXLAN 
HeaderOriginal L2 Frame FCS
D
s
t.
 
M
A
C
 A
d
d
r.
S
rc
. 
M
A
C
 A
d
d
r.
V
L
A
N
 T
yp
e
0
x
8
1
0
0
V
L
A
N
 I
D
T
a
g
E
th
e
r 
T
y
p
e
0
x
0
8
0
0
IP
 H
e
a
d
e
r
M
is
c
D
a
ta
P
ro
to
c
o
l
0
x
1
1
H
e
a
d
e
r
C
h
e
c
k
s
u
m
O
u
te
r
S
rc
. 
IP
O
u
te
r
D
s
t.
 I
P
U
D
P
S
rc
. 
P
o
rt
V
X
L
A
N
 P
o
rt
U
D
P
 L
e
n
g
th
C
h
e
c
k
s
u
m
0
x
0
0
0
0
V
X
L
A
N
R
R
R
R
1
R
R
R
R
e
s
e
rv
e
d
V
N
ID
R
e
s
e
rv
e
d
14 Bytes 
(4 bytes optional) 20 Bytes 8 Bytes 8 Bytes
48 48 16 16 16 72 8 16 32 32 16 16 16 16 8 24 24 8
• VXLAN is a Layer 2 overlay scheme over a Layer 3 network. 
• VXLAN uses Ethernet in UDP encapsulation
• VXLAN uses a 24-bit VXLAN Segment ID (VNI) to identify Layer-2 segments
For Your
Reference
88
VXLAN Terminology
VTEP – Virtual Tunnel End Point
• VXLAN terminates its tunnels on VTEPs (Virtual Tunnel End Point).
• VTEP has two interfaces :
1. Bridging functionality for local hosts
2. IP identification in the core network for VXLAN encapsulation / de-encapsulation. 
Local LAN Segment
IP Interface
End SystemEnd System
VTEP
Transport IP Network
Local LAN Segment
IP Interface
End SystemEnd System
VTEP
89
vPC VTEP
VXLAN
VLAN
vPC VTEP vPC VTEP
• When vPC is enabled an ‘anycast’ VTEP 
address is programmed on both vPC 
peers
• Multicast topology prevents BUM traffic 
being sent to the same IP address across 
the L3 network (prevents duplication of 
flooded packets)
• vPC peer-gateway feature must be 
enabled on both peers
• VXLAN header is ‘not’ carried on the vPC 
Peer link
90
VXLAN & VPC
VPC Configuration
vtep
1
vtep
2
vtep
3
vtep
4
H1
10.10.10.10
VLAN 10
(vpc)
H2
10.10.10.20
VLAN 10
(vpc)
VTEP1
vlan 10
vn-segment 10000
interface loopback 0
ip address <VTEP individual IP – orphan)
ip address <VTEP anycast IP – per VPC domain> secondary
!
interface nve1
source-interface loopback0
member vni 10000 mcast-group 235.1.1.1
Map VNI to VLAN
VXLAN Tunnel Interface
Source Interface
individual IP is used for single attached Hosts
anycast IP is used for VPC attached Hosts
VTEP2
vlan 10
vn-segment 10000
interface loopback 0
ip address <VTEP individual IP - orphan>
ip address <VTEP anycast IP – per VPC domain> secondary
!
interface nve1
source-interface loopback0
member vni 10000 mcast-group 235.1.1.1
For Your
Reference
91
VXLAN & VPC
VPC Configuration
vtep
1
vtep
2
vtep
3
vtep
4
H1
10.10.10.10
VLAN 10
(vpc)
H2
10.10.10.20
VLAN 10
(vpc)
VTEP1
vlan 10
vn-segment 10000
interface loopback 0
ip address 1.1.1.1/32
ip address 1.1.1.201/32 secondary
!
Interface nve1
source-interface loopback0
member vni 10000 mcast-group 235.1.1.1
VTEP3
vlan 10
vn-segment 10000
interface loopback 0
ip address 1.1.1.3/32
ip address 1.1.1.202/32 secondary
!
Interface nve1
source-interface loopback0
member vni 10000 mcast-group 235.1.1.1
VTEP2
vlan 10
vn-segment 10000
interface loopback 0
ip address 1.1.1.2/32
ip address 1.1.1.201/32 secondary
!
Interface nve1
source-interface loopback0
member vni 10000 mcast-group 235.1.1.1
VTEP4
vlan 10
vn-segment 10000
interface loopback 0
ip address 1.1.1.4/32
ip address 1.1.1.202/32 secondary
!
Interface nve1
source-interface loopback0
member vni 10000 mcast-group 235.1.1.1
For Your
Reference
92
VXLAN & VPC
Dual attached Host to dual attached Host (Layer-2)
vtep
1
vtep
2
vtep
3
vtep
4
H1
10.10.10.10
VLAN 10
(vpc)
H2
10.10.10.20
VLAN 10
(vpc)
• Host 1 (H1) and Host 2 (H2) are dual 
connected to a VPC domain
• As H1 is behind a VPC interface, the 
anycast VTEP IP is the source for 
the the VXLAN encapsulation
• As H2 is behind a VPC interface, the 
anycast VTEP IP is the target 
vtep
20
vtep
30
93
Nexus 9000 + APIC = ACI
APIC
APIC
APIC
94
External 
Network
App DBWeb
QoS
Filter
QoS
Service
QoS
Filter
ACI uses a policy based approach 
that focuses on the application.
95
vPC and ACI
ACI fabric utilised for control-plane vPC
Domains
vPC 
peers
ACI
fabric
• No dedicated peer-link between vPC peers:
Fabric itself serves as the MCT
vPC vPC• CFS (Cisco Fabric Services) is replaced by 
Zero Message Queue (ZMQ)
vtep
1
vtep
2
vtep
3
• No out-of-band mechanism to detect peer 
liveliness:
Due to rich fabric-connectivity (leaf-spine), it is 
very unlikely that peers will have no active 
path between them
• As ACI fabric is VXLAN-based, an anycast 
VTEP is shared by both leaf switches in a 
vPC domain
96
Agenda
• Reference Material
97
Reference Material
• vPC Best Practices Design Guide: 
http://www.cisco.com/c/dam/en/us/td/docs/switches/datacenter/sw/design/vpc_design/vpc_best_practices_design_guid
e.pdf
• vPC design guides:
http://www.cisco.com/en/US/partner/products/ps9670/products_implementation_design_guides_list.html
• vPC and VSS Interoperability white Paper:
http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps708/white_paper_c11_589890.html
• VXLAN Overview :
http://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-switches/white-paper-c11-729383.html
• Fabrcipath whitepaper :
http://www.cisco.com/c/en/us/products/collateral/switches/nexus-7000-series-switches/white_paper_c11-687554.html
 ACI Overview 
http://www.cisco.com/c/en/us/products/collateral/cloud-systems-management/aci-fabric-controller/white-paper-c11-729587.html
For Your
Reference
98
vPC in 2016
vPC Benefits
Fabricpath
VXLAN
Key Take-Aways 
• No Blocked Ports
• High availability
• Fast Convergence
• Eliminates Spanning-Tree *
• High resiliency
• vPC+ for legacy switches, 
servers, hosts
• L2 segment scalability
• VTEP redundancy with 
vPC
VXLAN, ACI, Fabricpath
• Policy Based
• Fabric for vPC control 
plane
ACI
• Unified Fabric for LAN & 
SAN
FCoE
99
Call to Action
• Visit the World of Solutions for
• Cisco Campus
• Walk in Labs 
• Technical Solution Clinics
• Meet the Engineer
• Lunch and Learn Topics
• DevNet zone related sessions
100
Complete Your Online Session Evaluation
• Please complete your online session
evaluations after each session.
Complete 4 session evaluations
& the Overall Conference Evaluation
(available from Thursday)
to receive your Cisco Live T-shirt.
• All surveys can be completed via
the Cisco Live Mobile App or the
Communication Stations 
101
there’s Something About 
vPC
Many Things
102
Thank you
103