Latency Troubleshooting Tips
  1. Using traceroute
  2. Identifying where latency occurs
  3. Causes of latency
  4. Actions you can take to resolve latency issues

Please note: This is intended to be a brief introduction to troubleshooting latency issues and is not intended to be a comprehensive guide.

Using traceroute

The first step in troubleshooting latency is to determine the path the traffic follows. The easiest way to do this is to use traceroute, a tool available on most computers and routers. On UNIX systems and Cisco routers the command is invoked by typing traceroute, while in Windows the command is tracert (typed in a DOS window). Following the command, enter the IP address or hostname of the destination you are trying to reach. The output of traceroute is a listing of each router the traffic passes through on its way to the destination and the time it took to get to that router from the originating system:

sl-gw8-joe#traceroute example-host.sprintcustomer.com
Translating "example-host.sprintcustomer.com"...domain server (10.10.10.2) [OK]

Type escape sequence to abort.
Tracing the route to example-host.sprintcustomer.com (10.151.27.3)

1 sl-bb22-joe.sprintlink.net (10.36.144.17) 1 msec 2 msec 1 msec
2 sl-bb25-bob.sprintlink.net (10.232.19.178) 16 msec 16 msec 12 msec
3 sl-bb21-sam.sprintlink.net (10.232.20.91) 54 msec 50 msec 50 msec
4 sl-bb20-sam.sprintlink.net (10.232.3.157) 50 msec 50 msec 54 msec
5 sl-bb22-ed.sprintlink.net (10.232.20.98) 52 msec 51 msec 55 msec
6 sl-gw14-ed.sprintlink.net (10.232.27.10) 51 msec 53 msec 52 msec
7 sl-customer-1.sprintlink.net (10.232.141.20) 63 msec 65 msec 61 msec
8 cust-router.sprintcustomer.com (10.151.27.10) 70 msec 68 msec 72 msec
9 example-host.sprintcustomer.com (10.151.27.3) 78 msec 75 msec 74 msec


Note: Each time listed is the total amount of time that one packet took to get from the originating machine to that hop. It is not the amount of time from the previous hop to the following hop. The output of different implementations of traceroute may look different than the example given above (traceroute on a Cisco router), but they should all provide similar information.


Identifying where latency occurs

Once you know the path that your traffic follows, you can examine the traceroute output to determine where the latency occurs. Look for a hop where the times shown increase dramatically (usually by several hundred milliseconds). For example:

sl-gw8-joe#traceroute example-host.sprintcustomer.com
Translating "example-host.sprintcustomer.com"...domain server (10.10.10.2) [OK]

Type escape sequence to abort.
Tracing the route to example-host.sprintcustomer.com (10.151.27.3)

1 sl-bb22-joe.sprintlink.net (10.36.144.17) 1 msec 0 msec 2 msec
2 sl-bb25-bob.sprintlink.net (10.232.19.178) 15 msec 13 msec 12 msec
3 sl-bb21-sam.sprintlink.net (10.232.20.91) 53 msec 50 msec 52 msec
4 sl-bb20-sam.sprintlink.net (10.232.3.157) 55 msec 51 msec 54 msec
5 sl-bb22-ed.sprintlink.net (10.232.20.98) 54 msec 53 msec 51 msec
6 sl-gw14-ed.sprintlink.net (10.232.27.10) 52 msec 53 msec 57 msec
7 sl-customer-1.sprintlink.net (10.232.141.20) 763 msec 842 msec 801 msec
8 cust-router.sprintcustomer.com (10.151.27.10) 770 msec 867 msec 882 msec
9 example-host.sprintcustomer.com (10.151.27.3) 866 msec 901 msec 878 msec

In the example above, there is a large increase in the times listed at hop 7. Therefore, the latency is likely occurring across the link between the routers at hop 6 and hop 7. You should also take into account the distance between hops when determining if there is high latency in a path. For example, the times increase somewhat between hops 2 and 3, but this may not be indicative of a problem if the distance between the routers at those hops is hundreds or thousands of miles. You may also see isolated high hop times, but these do not typically indicate a problem. These isolated high hop times are due to the lower priority that routers place on ICMP packets (used by routers to respond to traceroute packets), which occasionally causes the router's response to be delayed. The occasional high response time in any traceroute or ping output is normal.

The latency problem may still reside elsewhere despite what the traceroute indicates. Traceroute only gives you the forward path to the destination host, not the return path. The return path may be completely different (asymmetric routing), and there may be latency on that path. Asymmetric routing is not unusual on the Internet and is not in itself a problem. To see the return path, you will need to obtain a traceroute performed on the destination host tracing the path back to you.

Note: Not every router that resolves to a hostname ending in "sprintlink.net" is owned or maintained by Sprint. SprintLink customer routers also resolve to hostnames ending in "sprintlink.net". These routers will typically have names of the form "sl-xxx-##.sprintlink.net" where "xxx" is a Sprint internal code name for the customer, and "##" is some number. An example can be seen in hop 7 above. Customer routers are not typically maintained by Sprint, nor can Sprint upgrade any customer links or alter a customer's traffic flow or routing without an order directly from the customer. Please see SprintLink Naming Convention for more details.


Causes of latency

Latency can be caused by several different problems, a few of which are listed here:

Link Overutilization: The most common cause of latency is overutilization of a link. When the amount of traffic passing over an IP link increases to to a high percentage of the total bandwidth of that link, the latency across that link begins to increase. The closer the utilization gets to 100%, the worse the latency becomes.

Router Overutilization: When routers become overutilized, (CPU utilization approaches 100%), their ability to route traffic quickly becomes impaired. This may be due to the router attempting to handle more traffic than it was designed for, or a shortage of router resources such as memory. Router overutilization may also be caused by malicious attacks on the router, such as denial-of-service attacks.

Firewall Issues: Firewalls can be affected by many of the same problems as routers. These include overutilization due to lack of resources and malicious attacks.


Actions you can take to resolve latency issues

Latency on SprintLink: If the latency appears to be on SprintLink, and you are a SprintLink customer, please open a ticket and we will address the issue. See the SprintLink Contacts Page for information on contacting our Service Management Center. If you are not a customer of Sprint, but have identified the latency problem to be on the SprintLink network, please send an email to noc@sprint.net, including all relevant information you have, and we will address the issue. Sprint will not open trouble tickets for non-customers.

Please note that Sprint cannot affect the service or bandwidth of any Sprint customer without explicit direction from that customer. We will not modify, alter, or affect in any way any customer's routing or traffic flow at the request of a third party. Please see SprintLink Naming Convention page for more details on how routers are named and identifying customer routers versus routers on the SprintLink backbone.

Latency elsewhere: If the latency appears to be occurring on a network other than SprintLink, including a SprintLink customer's network, we recommend you contact the administrators of the destination site to pursue a resolution to the issue.

Sprint cannot access, modify, upgrade or otherwise affect in any way networks that are not owned and maintained by Sprint. Sprint cannot divulge any customer information including, but not limited to, bandwidth capacities, quantity of links or contact information, to any outside party.