The IT network is at the heart of most enterprises, supporting business-critical applications, providing the data on which business decisions are made and facilitating communications with customers, partners, suppliers and coworkers. More than ever before, it is a strategic asset to the business, and any downtime or degradation in network or application performance will directly impact the organization’s bottom line. To deliver the service levels agreed with the business, the challenge is twofold: proactively improve and optimize performance to ensure that the network delivers what users require, and resolve any problems that arise as quickly as possible to minimize downtime. This White Paper looks at methodology of solving network and application performance issues and outlines a new approach to getting to root cause more quickly.
- TABLE OF CONTENTS
- Step One: Monitor/Alert
- Step Two: Investigate
- Step Three: Isolate
- Step Four: Root Cause Analysis and Problem Resolution
Getting to the root cause of network and application issues is increasingly difficult and time-consuming in today’s enterprise networks. Virtualization is extending from the data center to the desktop, cloud services are growing in popularity and BYOD (Bring Your Own Device) is here to stay, reflecting shifting work patterns and cultural change.
Problems may result from a proliferation of Wi-Fi devices, excessive use of bandwidth by unauthorized applications, configuration errors, poor application delivery infrastructure or many other sources. The increasing inclusion of voice and video adds more complexity and pushes bandwidth to its limits.
Solving performance problems is made more difficult and time-consuming by the challenge of trying to ascertain whose responsibility they are, particularly when all groups are reporting green KPIs.
The process for tackling network performance problems
To get to the root cause of network performance issues, a four step troubleshooting process is followed:
Figure 1: The problem solving workflow.
The tools available to assist problem solving fall into two categories: Network Management Systems (NMS) and packet capture and analysis tools.
The NMS primarily plays a role at the monitor/alert phase, monitoring the company’s routers and servers and asking whether they are working and responding as expected. However, some NMS are so complex to set up that they are only able to manage down to Layer 3 devices, so switches are not monitored at Layer 2. Polling data is aggregated over many minutes and so is smoothed out, hiding the impact of usage spikes. Additionally, because the NMS is centrally located, measurements made with the intention of understanding end user response times are inaccurate because the test is using a different part of the network to reach the device under investigation.
As a network engineer progresses through the troubleshooting process, the usefulness of the NMS decreases and it fails to provide the detailed information required to investigate performance problems fully.
A recent NETSCOUT® survey of approximately 3,000 network professionals found that 82% of respondents ranked network and application performance problems as a concern or critical issue, with 52% stating that an NMS has insufficient capabilities to get to the root cause of the problem most or all of the time. 51% of respondents said that they needed to leave their desk some or most of the time to troubleshoot the problem.
To obtain more detailed information, the engineer has to turn to freeware or commercial packet capture and analysis tools. These have a limited role in the alert stage as they only view a single point in the network, but come into their own at the root cause analysis stage. The complexity of packet analysis tools requires skilled and experienced engineers, and they are time consuming to use, as the result can be too much data – millions of packets to wade through, displayed through different user interfaces. This makes the troubleshooting process much more difficult and time-consuming.
Where problems can hide in the network
The gap between these tools – an NMS without comprehensive information and complex packet capture tools – increases MTTR. Nagging, intermittent problems can ‘hide’ in the network, reducing both productivity and the credibility of the IT department.
To investigate and resolve performance issues quickly, the engineer needs end-to-end visibility across the network: a dedicated solution for automated network and application analysis that fills the gap between traditional NMS and packet capture.
The needs to address:
- Unmanaged equipment, which may have been purchased because it is less expensive but will cost more to troubleshoot when problems occur, as there is no visibility of the health of each network segment and utilization levels cannot be monitored. In contrast, with a managed switch a network engineer can go to any switch port and see what the errors look like, view the utilization and see who is connected to that port.
- Undocumented networks, a continual problem given that frequent changes on a network make any documentation out-of-date shortly after completion. Physically trying to trace the path would take a long time, but without accurate documentation the engineer does not know which packets are flowing where. What is required is a means of discovering the real-time path through the network.
- Too much data, when the problem may lie in just a few packets. Problem-solving would be much faster with an automated method of sifting through captured packets to find the bad ones – an application centric analysis that takes a top down approach.
- Problems in the past, which only come to the engineer’s attention hours after they have occurred. What’s needed is a means of going back in time by capturing and analyzing large amounts of granular data over an extended period of time, say 24 hours, to catch intermittent issues.
- New technology that isn’t monitored, such as 10Gb Ethernet or 802.11n Wi-Fi. Many organizations have not invested in instrumentation for such technologies because they believe the substantial increase in capacity will overcome any problems.
- Wireless devices – the engineer needs a way to identify and monitor Wi-Fi devices, including BYOD, and to identify Wi-Fi and non-Wi-Fi interference from Bluetooth devices, cordless telephones, microwaves etc. using spectrum analysis.
- Problems that reside outside the network, so that the engineer can identify them and hand the performance issue and supporting evidence to other IT teams or external service providers, with sufficient information to enable further investigation and a rapid solution.
A new approach to problem solving
What is needed is an holistic network and application performance solution that captures all the data in the network and provides intelligent analysis to enable engineers to isolate root cause more quickly, or identify if the real problem lies outside the network. It needs to collect, aggregate, correlate and mediate all information, including flow, SNMP data and information gathered from other devices, with granularity up to one millisecond. Data should be displayed through a single user configurable dashboard, so that guided workflows can be applied to isolate the root cause of the issue quickly. By taking away the need to make assumptions and enabling the user to follow a logical process until the issue is identified and resolved, MTTR is reduced and the network engineer becomes more effective.
A network and application performance solution facilitates all stages of the troubleshooting process and provides the visibility needed to support network optimization.
The first requirement when addressing and resolving network problems is a system that provides a timely alert that a problem has occurred. The worst case scenario is to find out through a call from a user, in which case the engineer is already on the back foot. Many network management tools alerts have to be manually configured for each network by setting the system to ping or discover all the devices in each broadcast domain. With an always on network and application performance solution, however, automated discovery and guided workflows make it quick and easy to immediately see which are connected. This dramatically reduces the time required for set-up and monitoring.
Performance data is continually collected and stored in a database and displayed via a GUI on a performance dashboard, which the user can configure to suit their own requirements. Performance is monitored against a user defined baseline (e.g. the SLA) and anything outside this is immediately shown as an alarm. The user can then view the problem in varying degrees of detail as they begin the investigation stage.
Network and application performance systems can also be integrated with existing network management systems such as HP OpenView or Tivoli Netcool, and pass information and alarms to service management and operational dashboard solutions.
STEP TWO: INVESTIGATE
The network engineer now needs to investigate the scope of the problem. In order to facilitate rapid and accurate investigation, the solution needs to be able to collect and store all pertinent data, e.g. SNMP, flows, packets, end-user-response time etc. and store these for future analysis. A network and application performance solution also provides a real time method of discovering the path from the client to the service or application, significantly reducing the amount of time required; the path between the two devices can then be found and monitored for any problems across internal and external networks and the devices in the path. Results are displayed in a graphical format to facilitate understanding and rapid root cause analysis.
For optimal effectiveness the system should provide interfaces with both 1Gbps and 10Gbps connectivity, and be capable of capturing data at line speed on the wire. Some solutions can trace a path through the network from a client to a server identifying Layer 2 and Layer 3 devices in the path and providing the granularity needed to identify the source of the problem.
If the issue lies with a client or group of clients, the engineer needs to carry out a performance or application response test to identify if the problem is a wired or wireless network issue. By providing wired and wireless tools integrated using the same user interface, the network and application system enables a single test to identify the source of the problem.
Malware outbreaks also can be identified as part of this process, including the originating IP address, enabling the engineer to identify root causes of downtime that other tools miss.
STEP THREE: ISOLATE
At this stage the problem has been isolated to a single network segment, switch, router, server or application and the path, devices and ports in the path have been identified. Now the path needs to be analyzed, requiring traffic statistics for each link to determine whether the issue is due to a faulty device, link media, noise or interference, or traffic overload.
One of the great advantages of SNMP (Simple Network Management Protocol) is its ability to help isolate fault domains. Using SNMP to query each connection point along the way will determine if a traffic bottleneck is the source of the slowdown. This is straightforward if the devices in the path are managed and the engineer has the passwords or community strings to interrogate the devices. Otherwise he or she has to connect a tool in each link without disrupting the network to view the packets and traffic statistics. This can be extremely time consuming if there are a lot of links over a large geographic area, and may require multiple tools in different locations.
An automated network infrastructure health check using a network and application performance tool makes it possible to monitor all of the SNMP supported devices, looking at application flows for those showing packet loss or high utilization by querying the SNMP MIBs on the routers and reporting back at regular intervals. Whether there are tens or hundreds of switches in the network, the process is simple and quick.
Some problems will only be visible by being at the point where the problem has arisen. This requires a portable device with the right testing capabilities and the right interface to connect at the problem point, whether that is in front of a client or a 10G link in a datacenter. With many people working remotely, having a tool which gives this visibility is vital – and this will only increase in importance with the growth of BYOD.
A portable tool can also be shipped to a remote site to see what is happening with unmanaged equipment in the network without the need for an accompanying engineer. Ideally it should be able to perform path analysis, measure application infrastructure health and application flows and analyze WLAN performance, as well as reviewing roaming and retry capability and investigating any interference from outside devices.
If there are no links which are oversubscribed or have frame errors then the problem is not likely to be the network – but this can only be confirmed if the engineer has analyzed the links in a reasonable time and the problem he or she is trying to fix still exists. This requires the historic data captured by the network and application performance system.
STEP FOUR: ROOT CAUSE ANALYSIS AND RESOLUTION
At this stage the engineer will confirm the cause of the issue, formulate and implement a fix and validate the solution. If the problem is not located in the network and is not the server response or the result of overloading resources, more detailed information is required by capturing and analyzing packets. It is important to have isolated the link or triaged the problem between the server, the network and the application first, as packet analysis can be extremely time-consuming and requires a considerable amount of skill and experience.
To get to the root cause more quickly it is best to take a top down approach to the analysis, beginning at the application level. For example, if the path is good but the response time is poor, the problem could be a virtualized server, an application running on multiple tiers or a bug in the application.
One option is to use a packet analyzer that can easily show the application level and packet ladder diagrams. Span or mirrored tap connections are easy to configure but can lose packets with heavy traffic loads and will not show Layer 1 errors as these are blocked by the Layer 2 switch providing the span. Passive taps are best but connecting them breaks the connection, which will disrupt users of the services this link provides. If performance is suffering, this usually does not cause a problem, but it may affect those using this link to connect to other services.
A better solution is to construct the network with taps already placed in strategic position in front of server farms, data centers, routers to external links and in the core of the network. This enables the captures can be taken without breaking the network. If this is not possible the engineer may have to resort to span or port mirroring, bearing in mind the accompanying problems and inaccuracies.
A network and application performance solution provides an automated method of sifting through the captured packets to find the bad ones. It uses an application centric approach, with a GUI that shows each data flow with a visual indicator to indicate problems. The engineer simply clicks on this to drill down and see exactly which packet or packets have a problem. This can be further aided by capturing packets at multiple points in the infrastructure to determine where the problem exists. It requires the ability to carry out multi-segment analysis, triggering data capture at multiple points at the same time and then merging the results to provide the whole picture.
Effective root cause analysis can be carried out at either the data center or remote sites to see if problems are server or application related. Some tools can pull management information from physical or virtual servers to reveal performance and resource issues.
By collecting and analyzing historic granular data, the network and application performance system also enables the engineer to go back in time to review the symptoms that were occurring when the problem first appeared, enabling intermittent problems to be identified and resolved.
A network application and performance solution provides the visibility engineers need to document and audit the health of their corporate network. It enables them to spot poor performance and identify where the paths of applications or servers are running slowly, so that the slowest and most critical paths can be addressed. The information obtained can be used to prioritize projects such as server upgrades and make the business case for approval. It can also support installation of new equipment and applications by verifying that what has been done has worked and ensuring that it has not had a negative impact on performance elsewhere. The data can also prove (or otherwise) the impact of changes to the network such as virtualization, WAN optimization or data center consolidation.
NETSCOUT SYSTEMS, INC. (NASDAQ: NTCT) is a market leader in real-time service assurance and cybersecurity solutions for today’s most demanding service provider, enterprise and government networks. NETSCOUT’s Adaptive Service Intelligence (ASI) technology continuously monitors the service delivery environment to identify performance issues and provides insight into network-based security threats, helping teams to quickly resolve issues that can cause business disruptions or impact user experience. NETSCOUT delivers unmatched service visibility and protects the digital infrastructure that supports our connected world.