Cisco APIC Troubleshooting Guide, Release 4.1(x)

208 Pages·2017·3.61 MB·English

Cisco APIC Troubleshooting Guide, Release 4.1(x)
FirstPublished:2019-01-30
AmericasHeadquarters
CiscoSystems,Inc.
170WestTasmanDrive
SanJose,CA95134-1706
USA
http://www.cisco.com
Tel:[phone]
800553-NETS(6387)
Fax:[phone]
THESPECIFICATIONSANDINFORMATIONREGARDINGTHEPRODUCTSINTHISMANUALARESUBJECTTOCHANGEWITHOUTNOTICE.ALLSTATEMENTS,
INFORMATION,ANDRECOMMENDATIONSINTHISMANUALAREBELIEVEDTOBEACCURATEBUTAREPRESENTEDWITHOUTWARRANTYOFANYKIND,
EXPRESSORIMPLIED.USERSMUSTTAKEFULLRESPONSIBILITYFORTHEIRAPPLICATIONOFANYPRODUCTS.
THESOFTWARELICENSEANDLIMITEDWARRANTYFORTHEACCOMPANYINGPRODUCTARESETFORTHINTHEINFORMATIONPACKETTHATSHIPPEDWITH
THEPRODUCTANDAREINCORPORATEDHEREINBYTHISREFERENCE.IFYOUAREUNABLETOLOCATETHESOFTWARELICENSEORLIMITEDWARRANTY,
CONTACTYOURCISCOREPRESENTATIVEFORACOPY.
TheCiscoimplementationofTCPheadercompressionisanadaptationofaprogramdevelopedbytheUniversityofCalifornia,Berkeley(UCB)aspartofUCB'spublicdomainversionof
theUNIXoperatingsystem.Allrightsreserved.Copyright©1981,RegentsoftheUniversityofCalifornia.
NOTWITHSTANDINGANYOTHERWARRANTYHEREIN,ALLDOCUMENTFILESANDSOFTWAREOFTHESESUPPLIERSAREPROVIDED“ASIS"WITHALLFAULTS.
CISCOANDTHEABOVE-NAMEDSUPPLIERSDISCLAIMALLWARRANTIES,EXPRESSEDORIMPLIED,INCLUDING,WITHOUTLIMITATION,THOSEOF
MERCHANTABILITY,FITNESSFORAPARTICULARPURPOSEANDNONINFRINGEMENTORARISINGFROMACOURSEOFDEALING,USAGE,ORTRADEPRACTICE.
INNOEVENTSHALLCISCOORITSSUPPLIERSBELIABLEFORANYINDIRECT,SPECIAL,CONSEQUENTIAL,ORINCIDENTALDAMAGES,INCLUDING,WITHOUT
LIMITATION,LOSTPROFITSORLOSSORDAMAGETODATAARISINGOUTOFTHEUSEORINABILITYTOUSETHISMANUAL,EVENIFCISCOORITSSUPPLIERS
HAVEBEENADVISEDOFTHEPOSSIBILITYOFSUCHDAMAGES.
AnyInternetProtocol(IP)addressesandphonenumbersusedinthisdocumentarenotintendedtobeactualaddressesandphonenumbers.Anyexamples,commanddisplayoutput,network
topologydiagrams,andotherfiguresincludedinthedocumentareshownforillustrativepurposesonly.AnyuseofactualIPaddressesorphonenumbersinillustrativecontentisunintentional
andcoincidental.
Allprintedcopiesandduplicatesoftcopiesofthisdocumentareconsidereduncontrolled.Seethecurrentonlineversionforthelatestversion.
Ciscohasmorethan200officesworldwide.AddressesandphonenumbersarelistedontheCiscowebsiteatwww.cisco.com/go/offices.
CiscoandtheCiscologoaretrademarksorregisteredtrademarksofCiscoand/oritsaffiliatesintheU.S.andothercountries.ToviewalistofCiscotrademarks,gotothisURL:www.cisco.com
gotrademarks.Third-partytrademarksmentionedarethepropertyoftheirrespectiveowners.TheuseofthewordpartnerdoesnotimplyapartnershiprelationshipbetweenCiscoandany
othercompany.(1721R)
©2019CiscoSystems,Inc.Allrightsreserved.
CONTEN TS
PREFACE Preface xiii
Audience xiii
DocumentConventions xiii
RelatedDocumentation xv
DocumentationFeedback xv
CHAPTER 1 NewandChanged 1
NewandChangedInformation 1
CHAPTER 2 TroubleshootingOverview 3
TroubleshootingBasics 4
CHAPTER 3 TroubleshootingAPICCrashScenarios 7
CiscoAPICClusterFailureScenarios 7
ClusterTroubleshootingScenarios 7
ClusterFaults 10
TroubleshootingApplicationCentricInfrastructureCrashScenarios 11
TroubleshootingFabricNodeandProcessCrash 11
APICProcessCrashVerificationandRestart 13
TroubleshootinganAPICProcessCrash 15
CHAPTER 4 RecoveringCiscoAPICPasswordsandAccessingSpecialLogins 17
RecoveringtheAPICPassword 17
UsingtheRescue-userAccounttoErasetheCiscoAPICConfigurationUsingtheNX-OSStyleCLI 18
UsingtheFallbackLoginDomaintoLogintotheLocalDatabase 18
CiscoAPICTroubleshootingGuide,Release4.1(x)
iii
Contents
CHAPTER 5 CiscoAPICTroubleshootingOperations 21
ShuttingDowntheAPICSystem 21
ShuttingDowntheAPICControllerUsingtheGUI 21
UsingtheAPICReloadOptionUsingtheGUI 22
ControllingtheLEDLocatorUsingtheGUI 23
CHAPTER 6 UsingtheCiscoAPICTroubleshootingTools 25
EnablingandViewingACLContractandDenyLogs 26
AboutACLContractPermitandDenyLogs 26
EnablingACLContractPermitandDenyLoggingUsingtheGUI 27
EnablingACLContractPermitLoggingUsingtheNX-OSCLI 28
EnablingACLContractPermitLoggingUsingtheRESTAPI 28
EnablingTabooContractDenyLoggingUsingtheGUI 29
EnablingTabooContractDenyLoggingUsingtheNX-OSCLI 30
EnablingTabooContractDenyLoggingUsingtheRESTAPI 30
ViewingACLPermitandDenyLogsUsingtheGUI 31
ViewingACLPermitandDenyLogsUsingtheRESTAPI 32
ViewingACLPermitandDenyLogsUsingtheNX-OSCLI 33
UsingAtomicCounterPoliciesforGatheringStatistics 35
AtomicCounters 35
AtomicCountersGuidelinesandRestrictions 36
ConfiguringAtomicCounters 37
EnablingAtomicCounters 37
TroubleshootingUsingAtomicCounterswiththeRESTAPI 38
EnablingandViewingDigitalOpticalMonitoringStatistics 39
EnablingDigitalOpticalMonitoringUsingtheGUI 39
EnablingDigitalOpticalMonitoringUsingtheRESTAPI 40
ViewingDigitalOpticalMonitoringStatisticsWiththeGUI 41
TroubleshootingUsingDigitalOpticalMonitoringWiththeRESTAPI 41
ViewingandUnderstandingHealthScores 42
HealthScoreTypes 42
FilteringbyHealthScore 43
ViewingTenantHealth 43
CiscoAPICTroubleshootingGuide,Release4.1(x)
iv
Contents
ViewingFabricHealth 43
ViewingMOHealthinVisore 43
DebuggingHealthScoresUsingLogs 44
ViewingFaults 44
EnablingPortTrackingforUplinkFailureDetection 45
PortTrackingPolicyforFabricPortFailureDetection 45
ConfiguringPortTrackingUsingtheGUI 46
PortTrackingUsingtheNX-OSCLI 46
PortTrackingUsingtheRESTAPI 47
ConfiguringSNMPforMonitoringandManagingDevices 47
AboutSNMP 47
SNMPAccessSupportinACI 48
ConfiguringtheSNMPPolicyUsingtheGUI 48
ConfiguringanSNMPTrapDestinationUsingtheGUI 50
ConfiguringanSNMPTrapSourceUsingtheGUI 50
MonitoringtheSystemUsingSNMP 51
ConfiguringSPANforTrafficMonitoring 51
AboutSPAN 51
MultinodeSPAN 52
SPANGuidelinesandRestrictions 53
ConfiguringSPANUsingtheGUI 56
ConfiguringaTenantSPANSessionUsingtheCiscoAPICGUI 56
ConfiguringaSPANFilterGroupUsingtheAPICGUI 57
ConfiguringanAccessSPANPolicyUsingtheCiscoAPICGUI 58
ConfiguringaFabricSPANPolicyUsingtheCiscoAPICGUI 59
ConfiguringaLayer3EPGSPANSessionforExternalAccessUsingtheAPICGUI 60
ConfiguringaDestinationGroupforanAccessSPANPolicyUsingtheCiscoAPICGUI 61
ConfiguringaDestinationGroupforaFabricSPANPolicyUsingtheCiscoAPICGUI 61
ConfiguringSPANUsingtheNX-OSStyleCLI 62
ConfiguringLocalSPANinAccessMode 62
ConfiguringaSPANFilterGroup 65
AssociatingaSPANFilterGroup 66
ConfiguringERSPANinAccessMode 68
ConfiguringERSPANinFabricMode 71
CiscoAPICTroubleshootingGuide,Release4.1(x)
v
Contents
ConfiguringERSPANinTenantMode 73
ConfiguringaGlobalSPAN-On-DropSessionUsingtheCLI 75
ConfiguringSPANUsingtheRESTAPI 77
ConfiguringaFabricDestinationGroupforanERSPANDestinationUsingtheRESTAPI 77
ConfiguringaGlobalDropSourceGroupUsingtheRESTAPI 77
ConfiguringaLeafPortasaSPANDestinationUsingtheRESTAPI 77
ConfiguringaSPANAccessSourceGroupUsingtheRESTAPI 78
ConfiguringaSPANFabricSourceGroupUsingtheRESTAPI 78
ConfiguringanAccessDestinationGroupforanERSPANDestinationUsingtheRESTAPI 79
UsingStatistics 79
ViewingStatisticsintheGUI 80
SwitchStatisticsCommands 80
ManagingStatisticsThresholdsUsingtheGUI 81
StatisticsTroubleshootingScenarios 82
StatisticsCleanup 83
SpecifyingSyslogSourcesandDestinations 84
AboutSyslog 84
CreatingaSyslogDestinationandDestinationGroup 85
CreatingaSyslogSource 86
EnablingSyslogtoDisplayinNX-OSCLIFormat,UsingtheRESTAPI 87
DiscoveringPathsandTestingConnectivitywithTraceroute 88
AboutTraceroute 88
AboutWindowsandLinuxTraceroute 89
TracerouteGuidelinesandRestrictions 90
PerformingaTracerouteBetweenEndpoints 91
UsingtheTroubleshootingWizard 92
GettingStartedwiththeTroubleshootingWizard 92
GeneratingTroubleshootingReports 94
TopologyintheTroubleshootingWizard 95
UsingtheFaultsTroubleshootingScreen 96
UsingtheDrop/StatisticsTroubleshootingScreen 97
UsingtheContractsTroubleshootingScreen 99
UsingtheEventsTroubleshootingScreen 100
UsingtheTracerouteTroubleshootingScreen 100
CiscoAPICTroubleshootingGuide,Release4.1(x)
vi
Contents
UsingtheAtomicCounterTroubleshootingScreen 102
UsingtheSPANTroubleshootingScreen 102
CreatingaSPANSessionUsingtheCiscoAPICTroubleshootingCLI 102
L4-L7ServicesValidatedScenarios 103
ListofAPIsforEndpointtoEndpointConnections 104
interactiveAPI 105
createsessionAPI 106
modifysessionAPI 107
atomiccounterAPI 107
tracerouteAPI 107
spanAPI 108
generatereportAPI 109
schedulereportAPI 109
getreportstatusAPI 110
getreportslistAPI 110
getsessionslistAPI 111
getsessiondetailAPI 111
deletesessionAPI 111
clearreportsAPI 112
contractsAPI 112
ListofAPIsforEndpointtoLayer3ExternalConnections 112
interactiveAPI 113
createsessionAPI 113
modifysessionAPI 114
atomiccounterAPI 115
tracerouteAPI 116
spanAPI 117
generatereportAPI 118
schedulereportAPI 119
getreportstatusAPI 120
getreportslistAPI 120
getsessionslistAPI 120
getsessiondetailAPI 121
deletesessionAPI 122
CiscoAPICTroubleshootingGuide,Release4.1(x)
vii
Contents
clearreportsAPI 123
contractsAPI 123
ratelimitAPI 124
13extAPI 124
CheckingforConfigurationSynchronizationIssues 125
ViewingUserActivities 125
AccessingUserActivities 126
EmbeddedLogicAnalyzerModule 126
AbouttheEmbeddedLogicAnalyzerModule 126
GeneratinganELAMReportintheSimplifiedOutputforModularSwitches 126
GeneratinganELAMReportintheSimplifiedOutputforFixedForm-FactorSwitches 128
CHAPTER 7 ManuallyRemovingDisabledInterfacesandDecommissionedSwitchesfromtheGUI 129
ManuallyRemovingDisabledInterfacesandDecommissionedSwitchesfromtheGUI 129
CHAPTER 8 DecommissioningandRecommissioningSwitches 131
DecommissioningandRecommissioningSwitches 131
CHAPTER 9 TroubleshootingStepsforEndpointConnectivityProblems 133
TroubleshootingEndpointConnectivity 133
InspectingEndpointandTunnelInterfaceStatus 134
InspectingtheEndpointStatus 134
InspectingtheTunnelInterfaceStatus 135
ConnectinganSFPModule 135
CHAPTER 10 TroubleshootingEVPNType-2RouteAdvertisement 137
TroubleshootingEVPNType-2RouteDistributiontoaDCIG 137
CHAPTER 11 PerformingaRebuildoftheFabric 141
RebuildingtheFabric 141
CHAPTER 12 VerifyingIP-BasedEPGConfigurations 143
VerifyingIP-BasedEPGConfigurationsUsingtheGUI 143
CiscoAPICTroubleshootingGuide,Release4.1(x)
viii
Contents
VerifyingIP-EPGConfigurationsUsingSwitchCommands 144
CHAPTER 13 RecoveringaDisconnectedLeaf 147
RecoveringaDisconnectedLeafUsingtheRESTAPI 147
CHAPTER 14 TroubleshootingaLoopbackFailure 149
IdentifyingaFailedLineCard 149
CHAPTER 15 DeterminingWhyaPIMInterfaceWasNotCreated 151
APIMInterfaceWasNotCreatedForanL3OutInterface 151
APIMInterfaceWasNotCreatedForaMulticastTunnelInterface 152
APIMInterfaceWasNotCreatedForaMulticast-EnabledBridgeDomain 152
CHAPTER 16 ConfirmingthePortSecurityInstallation 153
ConfirmingYourPortSecurityInstallationUsingVisore 153
ConfirmingYourHardwarePortSecurityInstallationUsingtheCiscoNX-OSCLI 153
CHAPTER 17 TroubleshootingQoSPolicies 157
TroubleshootingCiscoAPICQoSPolicies 157
CHAPTER 18 DeterminingtheSupportedSSLCiphers 159
AboutSSLCiphers 159
DeterminingtheSupportedSSLCiphersUsingtheCLI 160
CHAPTER 19 RemovingUnwanted_ui_Objects 161
RemovingUnwanted_ui_ObjectsUsingtheRESTAPI 163
CHAPTER 20 TroubleshootingMultipodandMulti-SiteIssues 165
TroubleshootingMultipodandMulti-Site 165
APPENDIX A acidiagCommand 167
APPENDIX B ConfiguringExportPoliciesforTroubleshooting 175
CiscoAPICTroubleshootingGuide,Release4.1(x)
ix
Contents
AboutExportingFiles 175
FileExportGuidelinesandRestrictions 175
ConfiguringaRemoteLocation 176
ConfiguringaRemoteLocationUsingtheGUI 176
ConfiguringaRemoteLocationUsingtheRESTAPI 176
ConfiguringaRemoteLocationUsingtheNX-OSStyleCLI 177
SendinganOn-DemandTechSupportFile 178
SendinganOn-DemandTechSupportFileUsingtheGUI 178
SendinganOn-DemandTechSupportFileUsingtheRESTAPI 178
APPENDIX C FindingtheSwitchInventory 181
FindingYourSwitchInventoryUsingtheGUI 181
FindingYourSwitchInventoryUsingtheNX-OSCLI 181
FindingYourSwitchInventoryUsingtheRESTAPI 184
APPENDIX D CiscoAPICClusterManagement 187
ExpandingtheCiscoAPICCluster 187
ContractingtheCiscoAPICCluster 187
ClusterManagementGuidelines 188
ExpandingtheAPICClusterSize 189
ReducingtheAPICClusterSize 189
ReplacingCiscoAPICControllersintheCluster 190
ExpandingtheClusterExamples 191
ExpandingtheAPICClusterUsingtheGUI 191
ExpandingtheAPICClusterUsingtheRESTAPI 192
ContractingtheClusterExamples 192
ContractingtheAPICClusterUsingtheGUI 192
ContractingtheAPICClusterUsingtheRESTAPI 193
CommissioningandDecommissioningCiscoAPICControllers 194
CommissioningaCiscoAPICintheClusterUsingtheGUI 194
DecommissioningaCiscoAPICControllerintheClusterUsingtheGUI 194
ReplacingaCiscoAPICinaClusterUsingtheCLI 195
APPENDIX E CiscoAPICSSDReplacement 197
CiscoAPICTroubleshootingGuide,Release4.1(x)
x