|
|
This chapter presents problem-solving scenarios focusing on identifying, isolating, and solving problems that impede throughput performance in internetworks.
These example problem-solving scenarios address specific situations and illustrate the process of problem isolation and resolution. The scenarios provided here span different protocols, media, and problem types. The objective is to illustrate a problem-solving method based on the problem-solving model defined in Chapter 1, "Troubleshooting Overview." The scenarios that follow focus specifically on situations in which traffic is getting to its intended destination, but network users complain about slow host response, connections dropping, or sporadic resource availability.
Each scenario includes the following components:
Chapter 9, "Troubleshooting Internet Performance," presents a series of symptom modules that provide snapshots of common symptoms, possible causes, and suggested actions for the protocols and technologies addressed in this publication.
An overview of scenarios and symptom modules is provided in "How to Use this Publication" in Chapter 1, "Troubleshooting Overview."
 | Caution Throughout this and other chapters, the use of debug commands is suggested for obtaining information about network traffic and router status. Use these commands with great care. In general, it is recommended that these commands only be used under the direction of your router technical support representative when troubleshooting specific problems. Enabling debugging can disrupt operation of the router when internets are experiencing high load conditions. When you finish using a debug command, remember to disable it with its specific undebug command or with the undebug all command. |
In general, performance slowdowns are considered lower-priority problems than reachability issues. However, poorly performing internetworks can degrade organizational productivity and often can effectively halt operation of network applications if communications degenerate enough. Performance problems can manifest themselves in many ways. Slow host response, dropped connections, and high error counts all suggest that network performance is not optimal. Unfortunately, the actual sources of performance problems are often difficult to detect.
This chapter presents a series of situational discussions, including the application of various diagnostic tools. Every possible scenario cannot be covered. Indeed, the scenarios included here only scratch the surface of possible situations. However, certain common themes typically tie all connectivity problems together. This chapter is provided in an effort to illustrate the use of troubleshooting tools and techniques to identify those common themes.
The following problem-solving scenarios are presented in this chapter:
The following case illustrates a situation in which performance degrades significantly after a Novell IPX internet is "upgraded" from a 2400-baud link over a phone line to a 9600-baud synchronous serial line.
Server responsiveness noticeably slows following an upgrade from a 2400-baud, direct dial-up interconnection between a client and a server to router-based link over a 9600-baud synchronous serial line.
Figure 8-1 illustrates a map of the internetwork change discussed in this case. The following list summarizes relevant elements of the environment:
Given the situation, insufficient bandwidth is the best candidate for poor server responsiveness.
As illustrated in Figure 8-1, the "upgraded" environment will be inherently slower than the original direct, dial-up interconnection.
In the original configuration, Server-A communicates with Client-A without any encapsulation. Although the modems do attach a header to each transmission, information exchanged between Server-A and Client-A is essentially all data and specifically varies in size depending on the kind of communication occurring.
In the "upgraded" configuration, the Ethernet segments to which Server-A and Client-A are attached require a minimum packet size of 60 bytes (which includes a six-byte destination address, a six-byte source address, a two-byte type or length field, and data). The overhead associated with Ethernet encapsulation (for packets smaller than 60 bytes) can easily overwhelm the 9600-baud line.
One possible solution is to disable fast switching. When fast switching is disabled, the router uses the network layer packet size instead of the link layer packet size (as in fast switching). In addition, more buffering is available to handle peak loads when fast switching is disabled.
However, with such a narrow serial pipe, adding bandwidth is your best option. The question is, how much bandwidth does the application require?
The amount of additional bandwidth required will vary depending on your situation. Certainly, if multiple clients are trying to access multiple servers, converting the 9600-baud line to a 56-Kbps line would be reasonable.
The following case illustrates a situation in which performance degrades significantly after a bridged Novell IPX internet is converted to routing.
Server responsiveness slows by an approximate factor of four after Novell IPX routing is implemented in place of bridging.
Figure 8-2 illustrates a map of the internetwork change discussed in this case.
The following list summarizes relevant elements of the environment:
Given the situation, the maximum packet size limitation associated with standard NetWare in a routed environment is the best candidate for poor server responsiveness.
In a bridged environment, Server-B allows transmission of the maximum packet size associated with the media in the internetwork (1130 bytes for Ethernet, 4202 bytes for 4-Mbps Token Ring and 16-Mbps Token Ring).
However, in a router-based internet, standard Novell servers allow for a maximum packet size of 512 bytes, regardless of media. Packet routing defaults to this smallest common size whenever multiple network numbers are detected. In addition, prior to Software Release 8.3 (3), Cisco routers did not support Novell's Large Internet Packet Exchange NetWare-loadable module (LIPX.NLM) on Token Ring. This module was previously referred to as BIGPACK.NLM.
There are three actions that can help improve performance between Server-B and Client-B in this router-based internet:
| Caution Be sure that all media and transport protocols are accounted for when implementing LIPX.NLM. If any segment does not support the larger packets, connectivity can be disrupted throughout the internet. |
The following case illustrates a situation in which performance over a router interconnecting two 16-Mbps Token Rings is slower than a comparable interconnection of two Ethernet segments.
Server responsiveness is slow over the router separating two 16-Mbps rings.
Figure 8-3 illustrates a map of the relevant elements of this environment:
Given the situation, the router is probably implementing system software that predates Software Release 9.1. This is the best candidate for poor server responsiveness.
Prior to Software Release 9.1, Novell IPX routing over Token Ring was slow switched. As of Software Release 9.1, the routers support fast switching of Novell IPX.
Prior to Software Release 8.3(3), Cisco routers did not support Novell's Large Internet Packet Exchange NetWare-loadable module (LIPX.NLM) for Token Ring. This module was previously referred to as BIGPACK.NLM.
Use the show version command to determine the software revision level.
Assuming that the software version predates Software Release 9.1, the solution to this situation is as follows:
| Caution Be sure that all media and transport protocols are accounted for when implementing LIPX.NLM. If any segment does not support the larger packets, connectivity can be disrupted throughout the internet. |
The following case illustrates a situation in which performance is extremely slow over an Ethernet backbone separating two routers.
Slow server response among multiple Ethernet segments separated by two routers and an Ethernet backbone.
Figure 8-4 illustrates a map of the relevant elements of this environment:
Given the situation, congestion is the best candidate for poor performance over the backbone.
The following discussion briefly outlines the process of problem isolation for the situation discussed.
In general, you can use the following two methods to determine whether there is a congestion problem over the backbone:
Step 1: Use the show interfaces command and examine the display output for relative load, high and increasing levels of input errors, and drops.
Step 2: Attach a network analyzer onto the backbone and look for high levels of collisions and bandwidth utilization in excess of 30 percent.
Refer to "Slow Host Response over 56-Kbps HDLC Link" earlier in this chapter for information about general troubleshooting of performance problems in a routed internet. Also, refer to Chapter 7, "Troubleshooting WAN Connectivity," and Chapter 9, "Troubleshooting Internet Performance," for more information about diagnosing congestion problems.
If you do determine that congestion over the Ethernet backbone is high, the only real option is to increase bandwidth. You can do this by either adding additional Ethernet segments or by replacing the Ethernet backbone with a faster media, such as FDDI.
This scenario focused on improving performance over a backbone segmenting multiple Ethernets by increasing bandwidth using one of two options:
Figure 8-5 illustrates these options.
The following case illustrates a situation in which performance is less than optimal over parallel T1 links joining two routers.
One line appears to be heavily loaded, while the other is either idling or indicates very low load. Users complain of slow response and intermittent connection drops.
Figure 8-6 illustrates a map of the relevant elements of this environment:
Given the situation, the router probably is keeping only one routing table entry per target network.This is very likely to cause poor performance over the parallel serial lines. In the worst case, traffic is only routed through one line, while the second line is idle.
In general, you can use the following method to determine whether traffic is being unevenly distributed between the parallel lines:
Step 1: Use the show interfaces command and examine the displayed load for each interface. Also examine the number of input and output drops and the five-minute output and input error counts. Record the observed values.
Step 2: Use the clear counters command and continue to monitor changes in the counters over time with the show interfaces command.
Step 3: Look for values that are substantially uneven, (for example serial interface S0 indicates 300,000 packets total input, while serial interface S1 indicates only 1,000).
Step 4: If you determine that traffic is being unevenly distributed over the serial links, use the novell maximum-paths paths command to set the number of multiple paths the router will use when transmitting traffic to any particular destination. Instead of keeping only one routing table entry, the router will now remember up to the number of paths specified when determining how to route traffic. In essence, this forces load balancing over the two lines when paths is set to 2.
This scenario focused on improving performance over parallel links. The recommended solution here is to implement the novell maximum-paths paths global configuration command with a paths specification of 2.
The following case illustrates a situation in which performance is slow over parallel links of differing speeds that join two routers.
One line appears to be heavily loaded, while the other is either idling or indicates very low load. Users complain of slow response and intermittent connection drops.
Figure 8-7 illustrates a map of the relevant elements of this environment:
Because Novell's RIP routing protocol does not account for line speed, load cannot be balanced effectively between these two links. Given the situation, this load-balancing problem is probably causing poor performance over the parallel serial lines.
As in the prior case, it is quite possible that traffic will only be routed through one line, while the second line is idle. And, since RIP does not consider line speed, the 9.6-Kbps line could be completely overwhelmed, while the T1 line is relatively quiet.
In general, you can use the same method to determine whether traffic is being unevenly distributed between the parallel lines as discussed in the prior case:
Step 1: Use the show interfaces command and examine the displayed load for each interface. Also examine the number of input and output drops and the five-minute output and input error counts. Record the observed values.
Step 2: Use the clear counters command and continue to monitor changes in the counters over time with the show interfaces command.
Step 3: Look for values that are substantially uneven, (for example Serial0 indicates 300,000 packets total input, while Serial1 indicates only 1000).
Step 4: If you determine that traffic is being unevenly distributed over the serial links, and the novell maximum-paths paths command is already implemented, your only solution is to make the speed on both lines match or eliminate the slow-speed line altogether.
This scenario focused on improving performance over uneven parallel links. The recommended solution here is to force the speed of the parallel links to match or to eliminate the slow-speed link.
The problem scenario that follows focuses on performance in a TCP/IP internetwork featuring parallel serial links joining two geographically separated locations via Cisco routers. The analysis focuses on isolating problems and then considering options for relieving congestion.
R&D users at Remote-Lab complain of poor host response and slow performance when connecting to hosts at the Main-Campus. In addition, during certain times of the day, large files are being transferred over the serial network. At these times, traffic becomes especially slow, but does not stop.
Figure 8-8 illustrates a map of the internetwork discussed in this case. The following list summarizes relevant elements of the environment:
The first thing to do is to identify all likely or possible causes; the second step is to eliminate each one. Given the situation, the following problems are the best candidates for interconnection failure:
The following discussion works through the process of problem isolation, then suggests possible solutions.
The following procedure illustrates the process of investigating potential hardware problems.
Step 1: The first thing to do is to determine the condition of the serial lines. A preliminary check can be done using the show interfaces EXEC command. Figure 8-9 illustrates a typical display that would be returned by the system if the interfaces are minimally operational and the system can communicate with them.
Look for input errors and output drops being high. These would suggest that the serial line is being overutilized.
Step 2: Assume that the serial line is determined to be basically functional. That is, you see that the router reports "Serial 0 is up, line protocol is up." Now, use an extended ping test to isolate where traffic is being slowed. Look for drops, failures, and timeouts. Figure 8-10 illustrates an example of an extended ping test where failures are detected.
Step 3: Ping various nodes in the path, starting with the router closest to the remote hosts, looking for the point where drops start occurring.
For instance, ping from Router-Lab to Host-2L. If pings are successful, Ethernet-B can be eliminated as the source of congestion problems.
Next, ping from Router-Main to Host-1M. If pings are successful, Ethernet-A can be eliminated as the source of congestion problems.
Step 4: If these tests indicate no problems, ping between the routers. First, ping from Router-Main to the IP address associated with interface Ethernet1 on Router-Lab. Next, ping each of the serial interfaces on Router-Lab. If you find any ping failure on the serial lines, refer to serial debugging as discussed in Chapter 7, "Troubleshooting WAN Connectivity," and to the additional information provided in Chapter 1, "Troubleshooting Overview."
Step 5: If you determine that the problem is indeed one of congestion because of bandwidth overutilization, you must decide whether it is more effective to add bandwidth (in the form of another serial circuit) or to make an adjustment in the router configuration.
Step 6: If you see load values of about 50 percent, input drops, and output drops, you might consider implementing priority queuing to force Telnet to be given higher precedence over other packet types. This helps ensure reasonable connection service to users, even during periods when file transfers are taking place. Figure 8-11 illustrates a configuration for Router-Lab that establishes priority queuing and assigns port 23 (Telnet) a higher priority than other TCP/IP protocols, such as mail (port 25).
Step 7: If you are seeing load of close to 90 percent, as well as input errors and output drops, priority queuing is not likely to do any good. With consistently high congestion, your best solution is additional or faster serial links.
This scenario focused on the following topics in resolving performance problems in TCP/IP internetworks:
When designing and implementing internetworks, it is important to factor in any potential changes and expectations of growth. This is especially important when certain network elements are at risk of becoming bottlenecks--such as point-to-point serial links. Bandwidth that appears to be sufficient today may be inadequate in a year. And the budget may not exist to add another drop or replace the existing service.
The problem scenario that follows explores a situation in which performance over a serial link is not meeting user requirements. In this case, the analysis focuses on isolating the problem and then using certain router configuration capabilities to ease throughput bottlenecks.
Users at Remote-Site complain of consistently degraded performance when connecting to hosts at the Home-Office. Performance previously was acceptable, but now slows substantially during peak use periods.
Figure 8-12 illustrates a map of the internetwork discussed in this case. The following list summarizes relevant elements of the environment:
Given the situation, the following problems are the best candidates for interconnection failure:
The following discussion works through the process of problem isolation. Upon inspection, it appears that the serial interface is being overutilized (in other words, the 56-Kbps bandwidth is no longer sufficient). The question is: What can be done, short of installing another drop?
The discussion that follows isolates the problem, then suggests a configuration-based solution.
The following procedure illustrates the process of investigating potential hardware problems.
Step 1: The first thing to do is to determine the condition of the serial line. A preliminary check can be done using the show interfaces EXEC command. Figure 8-13 illustrates a typical display that the system returns when the interfaces are minimally operational and the system can communicate with them.
Of interest in this display is the fact that the value for input errors is relatively low, but the values for interface resets and output drops are both high. Another clue is that the load field indicates that the link is experiencing a load of about 75 percent of available bandwidth. This information combines to suggest that the serial line is functional, but is being overutilized.
To monitor changes in the number of dropped packets, follow this brief sequence:
To further confirm that the serial link is being overutilized, use the show buffers command. Figure 8-14 illustrates the output from this command. In this example, there are a large number of failures and misses. This suggests that there is some kind of problem with the system-level buffers and that the traffic the routers are trying to transmit exceeds the interface bandwidth.
Step 2: The router configuration files are the next place to look for clues in this scenario. If fast switching is not explicitly disabled for all protocols, this is the first configuration change to try. Fast switching is enabled by default. For DECnet, disable fast switching with the no decnet route-cache interface subcommand. This change forces the router to use system-level memory buffers (instead of board-level buffers), which under these conditions, can improve overall throughput.
Step 3: Although disabling fast switching can improve performance over the serial link, assume that problems still persist during peak demand times. The next step is to prioritize traffic using the priority queuing function. By assigning a "high" priority to bridged (LAT) packets, the LAT traffic takes precedence over any other traffic. Again, this enhances performance, but might not entirely eliminate peak period sluggishness.
Step 4: The final action in this case is to upgrade software to a more recent software version (for example, from revision 8.2 to 9.0). Software Releases 8.3 and higher allow administrators to configure specific buffer sizes. By setting a minimum number of system buffers as available at all times, it is possible to significantly reduce the bottleneck at the serial link.
Figure 8-15 illustrates a complete configuration listing for Router-Home (obtained using the write terminal command) that includes the changes suggested in Steps 2 through 4.
Clearly, this case revolved around an interface that was overworked. The immediate reaction to this situation might be to add another link in parallel. Ultimately, adding bandwidth is probably required. But that might not be an immediately available option. Perhaps the protocol being used cannot handle load balancing, or you simply cannot afford the added expense of another physical link within your current budget.
The actions offered in this example explore options that use the existing physical configuration, but reconfigure the way traffic is handled. To recap, the following modifications can help optimize traffic over an overloaded 56-Kbps link:
|
|
