Beyond Fault Management Implementing a NOC to Maintain High Availability

144 Pages·2004·3.07 MB·English

PS-510 © 2001, Cisco Systems, Inc. All rights reserved. 111
Beyond Fault Management
Implementing a NOC to
Maintain High Availability
Jim Thompson
Jay Thondakudi
© 2001, Cisco Systems, Inc. All rights reserved. 2
MAKING IT INTERACTIVE
© 2001, Cisco Systems, Inc. All rights reserved. 333
Copyright © 2001, Cisco Systems, Inc. All rights reserved. Printed in USA.
Presentation_ID.scr
Agenda
• Availability Measurement and your business
• Overview of a NOC
• Network Management Framework
• Fault Management
• Performance Management
• Tool Issues
• People, Processes and Procedures
• Back to the Concept of the NOC
© 2001, Cisco Systems, Inc. All rights reserved. 444
Method for Attaining a Highly Available
Network
or a road to five 9’s
• Establish a Standard Measurement
Method
• Define Business Goals as Related
to Metrics
• Categorize Failures, Root Causes,
and Improvements
• Take Action for Root Cause
Resolution and Improvement
Implementation
© 2001, Cisco Systems, Inc. All rights reserved. 555
Why should we care about network
availability?
Recent studies by Sage Research determined that
US based Service Providers encountered:
• Percent of downtime that is
unscheduled: 44%
• 18%of customers experience
over 100 hours of unscheduled
downtime or an availability of
98.5%
• Average cost of network
downtime per year: $21.6 million
or$2,169per minute!
Downtime -Costs too much!!!
SOURCE: Sage Research, IIPP SSeerrvviiccee PPrroovviiddeerr DDoowwnnttiimmee SSttuuddyy:: AAnnaallyyssiiss ooff DDoowwnnttiimmee CCaauusseess,,CCoossttss aanndd
CCoonnttaaiinnmmeenntt SSttrraatteeggiieess,, AAuugguusstt 1177,, 22000011,, PPrreeppaarreedd ffoorr CCiissccoo SSPPLLOOBB
© 2001, Cisco Systems, Inc. All rights reserved. 666
Copyright © 2001, Cisco Systems, Inc. All rights reserved. Printed in USA.
Presentation_ID.scr
Upside has Downside
Application Downtime
••NNuummbbeerr ooff eemmppllooyyeeeess ••DDiirreecctt RReevveennuuee LLoossss
axax ffbbffeeuuccrrttddeeeeddnn xxee ddhh oohhuuoorruussrr llooyy uurrttaattee ••CpCpaaooyymmmmppeeeennnnttssssaattoorryy
••LLoosstt ffuuttuurree rreevveennuuee
••BBiilllliinngg lloosssseess
••IInnvveessttmmeenntt rreevveennuuee
lloosssseess
••CCuussttoommeerrss ••RReevveennuuee RReeccooggnniittiioonn
••SSuupppplliieerrss ••CCaasshh FFllooww
••FFiinnaanncciiaall mmaarrkkeettss ••LLoosstt ddiissccoouunnttss
••BBuussiinneessss PPaarrttnneerrss ((aaccccoouunnttss ppaayyaabbllee))
••CCaarreeeerrss lloosstt ••TTeemmppoorraarryy eemmppllooyyeeeess ••PPaayymmeenntt gguuaarraanntteeeess
••EEqquuiippmmeenntt rreennttaall ••CCrreeddiitt rraattiinngg
SSoouurrccee:: GGaarrttnneerr RReesseeaarrcchh ••OOvveerrttiimmee ccoossttss ••SSttoocckk pprriiccee
AVVID Team;rmarwah © 2001, Cisco Systems, Inc. All rights ••reseTTrverrdaa.vveell eexxppeennsseess 777
What are your business goals?
•Financial
ROI Economic Value Added Revenue/Employee
•Productivity
•Time to Market
•Organizational Mission
•Customer Perspective
Satisfaction Retention Market Share
Define your ‘end-state’?
What is your goal?
© 2001, Cisco Systems, Inc. All rights reserved. 888
BBuussiinneessss O Obbjjeeccttiivveess & & A Avvaaiillaabbiilliittyy
Definition of Network Availability for your company–generalized (i.e.):
Network availability management is the optimization of the availability and reliability of
networking services and of the supporting network infrastructure, with the objective of
ensuring all business requirements are met.
Availability management policywill set a definite course of action for availability
measurement, reporting, and procedures. This strategy for availability management
will be based on defined and agreed upon business requirements that will help drive
policy to determine present and future resource decisions.
The role of network managementincluding availability and performance
management functions, is in managing growth and achieving efficiencies of scale as a
critical strategic objective for your company
© 2001, Cisco Systems, Inc. All rights reserved. 999
Copyright © 2001, Cisco Systems, Inc. All rights reserved. Printed in USA.
Presentation_ID.scr
Availability Business Requirements
• Yield Control
Yield per customer by volume & mix
Maximizing the yield per bandwidth, per router
• Activity-based accounting
Cost of downtime
Cost of waiting for a needed part or tool
Cost of reworking or redesign
© 2001, Cisco Systems, Inc. All rights reserved. 111000
Availability Business Requirements
Availability as a Basis for Productivity Data
Measurement of Total-Factor Productivity
Benchmarking the Organization
Overall Organizational Performance Metric
Availability as a Basis for Organizational
Competency
Availability as a Core competency
Availability Improvement as an Innovation Metric
Resource Allocation Information
Identify defects
Identify root cause
Measure MTTR –tied to process
© 2001, Cisco Systems, Inc. All rights reserved. 111111
How does your network enable your
business to reach its goals?
•User Requirements
Timeliness Interactivity Reliability
Quality Adaptability Security Affordability
•Application Requirements
Mission Criticality
Controlled-Rate Applications
Real-time
•Performance Requirements
Delay Reliability Capacity
•Network Requirements
Scaling Services Interoperability
Performance Monitoring Troubleshooting
© 2001, Cisco Systems, Inc. All rights reserved. 111222
Copyright © 2001, Cisco Systems, Inc. All rights reserved. Printed in USA.
Presentation_ID.scr
Agenda
• Availability Measurement and your business
• Overview of a NOC
• Network Management Framework
• Fault Management
• Performance Management
• Tool Issues
• People, Processes and Procedures
• Back to the Concept of the NOC
© 2001, Cisco Systems, Inc. All rights reserved. 111333
What is a NOC?
A Helpdesk?
A trouble-ticket logging center?
A Break-fix team?
All of the above?
© 2001, Cisco Systems, Inc. All rights reserved. 111444
What is a NOC?
Maybe a worst case
discussion on
CHAOS Theory!
© 2001, Cisco Systems, Inc. All rights reserved. 111555
Copyright © 2001, Cisco Systems, Inc. All rights reserved. Printed in USA.
Presentation_ID.scr
What is a NOC?
Are there references?
• RFC1302 -Building a Network Information
Services Infrastructure
• Authored February ’92 by Merit
contributors
© 2001, Cisco Systems, Inc. All rights reserved. 111666
What is a NOC?
• From RFC1302
Definition of a NIC and a NOC
A Network Information Center (NIC) is an
organization whose goal is to provide
informational, administrative, and procedural
support, primarily to users of its network and,
secondarily, to users of the greater Internet
and to other service agencies.
© 2001, Cisco Systems, Inc. All rights reserved. 111777
What is a NOC?
• Definition of a NIC and a NOC (cont)
A Network Operations Center (NOC) is an
organization whose goal is to oversee and
maintain the daily operations of a network. …
A NIC must work closely with its NOC to
ensure users get the best service possible.
© 2001, Cisco Systems, Inc. All rights reserved. 111888
Copyright © 2001, Cisco Systems, Inc. All rights reserved. Printed in USA.
Presentation_ID.scr
Are there other resources?
• North American Network Operator’s Group
www.nanog.org
www.merit.edu/internet/
Informative mailing lists
Sean “Backhoe King” Donelan
• puck.nether.net/netops/
NOC Telephone/Contact List!
© 2001, Cisco Systems, Inc. All rights reserved. 111999
Are there other resources?
www.nocbuilder.com
© 2001, Cisco Systems, Inc. All rights reserved. 222000
Example of Network Management Goals
• Increase/Maintain Network Availability
• Provide meaningful analysis and correlation
of events to assist in FAST fault resolution
• Provide effective use of engineering
resources by automating repetitive tasks and
de-complexifingnetwork operation
© 2001, Cisco Systems, Inc. All rights reserved. 222111
Copyright © 2001, Cisco Systems, Inc. All rights reserved. Printed in USA.
Presentation_ID.scr
Example of Network Management Goals
• Ensure network configuration information is
available for network/device restoration.
• Monitor to ensure that no network outage is
caused by a device which has reached its
performance limitations.
• Provide regular reports which summarize the
network for the various management teams
which require them.
© 2001, Cisco Systems, Inc. All rights reserved. 222222
What kind of NOC?
RRReeeaaaccctttiiivvveee
PPPrrroooaaaccctttiiivvveee
OOOpppeeerrraaatttiiiooonnnaaalll
© 2001, Cisco Systems, Inc. All rights reserved. 222333
Develop A Plan
“First comes thought; then organization
of that thought, into ideas and plans;
then transformation of those plans
into reality.
The beginning, as you will observe, is
in your imagination.”
Napoleon Hill
© 2001, Cisco Systems, Inc. All rights reserved. 222444
Copyright © 2001, Cisco Systems, Inc. All rights reserved. Printed in USA.
Presentation_ID.scr
What are our goals?
• What type of shop are we?
Lights Out & Remote… … or BIC-EOT
© 2001, Cisco Systems, Inc. All rights reserved. 222555
Initial Considerations
• What are our Availability/SLA
requirements? Maintenance Windows?
• Is the NOC simply identify and escalate…
… or also fix?
• How many managed devices are we
responsible for?
• Roughly how many events/day (hour,
minute) do we currently get or expect?
© 2001, Cisco Systems, Inc. All rights reserved. 222666
Initial Considerations (cont’d)
• How much $$ can we allocate?
• What are the skills of the operators?
• What types of devices are we monitoring?
• What are the technologies in use?
© 2001, Cisco Systems, Inc. All rights reserved. 222777
Copyright © 2001, Cisco Systems, Inc. All rights reserved. Printed in USA.
Presentation_ID.scr
Develop A Plan!
“Given two equally likely solutions to a
problem… The simplest one is usually
correct.”
Occam’sRazor
© 2001, Cisco Systems, Inc. All rights reserved. 222888
Develop A Plan!
“Given a choice between two
methodologies, choose the simplest
--the method which requires the
fewest resources. .”
NOC Corollary to Occam’sRazor
© 2001, Cisco Systems, Inc. All rights reserved. 222999
Agenda
• Availability Measurement and your business
• Overview of a NOC
• Network Management Framework
• Fault Management
• Performance Management
• Tool Issues
• People, Processes and Procedures
• Back to the Concept of the NOC
© 2001, Cisco Systems, Inc. All rights reserved. 333000
Copyright © 2001, Cisco Systems, Inc. All rights reserved. Printed in USA.
Presentation_ID.scr