[outages] comcast/sprint oddities

Tue Mar 12 23:33:53 EDT 2013

Was the below Sprint->Comcast traceroute taken while the issue was/is
still actively happening?

You should probably do multiple traceroutes from both endpoints, or get
better tools like mtr/WinMTR which can do this repeatedly.

What you've shown below indicates that you're only seeing an increase in
latency on the ingress side of the Sprint connection (i.e.
Comcast->Sprint), starting at one specific router (hop #13) on the
Sprint side.

If this was an actual service-impacting issue, you would be seeing an
increase in latency on the egress (Sprint->Comcast) traceroute as well,
given that traceroute attempts to solicit an ICMP time-exceeded message
from a device; think round-trip-time and how response packets "should"
take the same path that your Comcast->Sprint traceroute would.

Phrased differently: if the issue is actively happening, you should be
seeing "spiky" latency in both the Comcast->Sprint and Sprint->Comcast
traceroutes, just at different hops/network points.

What you've shown below refutes that, which to me indicates ICMP
prioritisation being used on Sprint routers.

If you really wanted to bring this up with Sprint, you would need to
have a relationship with them, and you would *need* to provide source
and destination IPs.

I would recommend setting up scripts/tools that run mtr from a cronjob
and save the output to a file, and do this on hosts at both ends of the
network (the Comcast end, and the Sprint end).

-- 
| Jeremy Chadwick                                   jdc at koitsu.org |
| UNIX Systems Administrator                http://jdc.koitsu.org/ |
| Mountain View, CA, US                                            |
| Making life hard for others since 1977.             PGP 4BD6C0CB |

On Wed, Mar 13, 2013 at 03:18:58AM +0000, Blake Pfankuch - Mailing List wrote:
> Sorry about that.  The reflexive tracert is below.  I would rather not list source and destination addresses on a public list.  These are directly connected hosts on each side, not routers, no ICMP prioritization for sure on the Sprint peered side, however I am at the mercy of Comcast Business on the other side...
> 
>   4    16 ms     7 ms     5 ms  sl-gw16-che-1-2-0.sprintlink.net [160.81.219.133]
>   5     5 ms     6 ms     6 ms  sl-crs2-che-0-4-2-3.sprintlink.net [144.232.6.51]
>   6    16 ms    15 ms    16 ms  sl-crs2-oma-0-2-2-0.sprintlink.net [144.232.18.121]
>   7    26 ms    25 ms    25 ms  144.232.1.74
>   8    26 ms    25 ms    25 ms  144.232.1.104
>   9    26 ms    26 ms    26 ms  144.232.6.102
>  10    32 ms    35 ms    35 ms  be-12-cr01.350ecermak.il.ibone.comcast.net [68.86.84.189]
>  11    49 ms    51 ms    51 ms  he-1-12-0-0-cr01.denver.co.ibone.comcast.net [68.86.85.250]
>  12    58 ms    59 ms    59 ms  he-0-9-0-0-ar02.aurora.co.denver.comcast.net [68.86.90.150]
> 
> This trace has first 3 and last 4 hops removed.  Previous trace has first 4 and last 2 hops removed.  Definitely looks like something funky, and was hoping someone might be able give me a little insight on where to go from here.
> 
> -----Original Message-----
> From: Jeremy Chadwick [mailto:jdc at koitsu.org] 
> Sent: Tuesday, March 12, 2013 9:02 PM
> To: Blake Pfankuch - Mailing List
> Cc: outages at outages.org
> Subject: Re: [outages] comcast/sprint oddities
> 
> Blake,
> 
> 1. You need to provide traceroutes from both endpoints, given the very likely possibility of asymmetric routing,
> 
> 2. You need to provide both source and destination IP addresses,
> 
> 3. Please make sure your destinations are not routers -- they need to be actual hosts.  If network or host ACLs/firewalls inhibit the ability to reach the destinations, that can make things a bit more difficult given the possibility of ICMP prioritisation.  I say this with acknowledgement of your statement that the same hop tends to show 60ms during normal hours but increases during evenings.
> 
> Necessary details/specifics are covered here:
> 
> http://www.nanog.org/meetings/nanog47/presentations/Sunday/RAS_Traceroute_N47_Sun.pdf
> 
> Thanks.
> 
> -- 
> | Jeremy Chadwick                                   jdc at koitsu.org |
> | UNIX Systems Administrator                http://jdc.koitsu.org/ |
> | Mountain View, CA, US                                            |
> | Making life hard for others since 1977.             PGP 4BD6C0CB |
> 
> On Wed, Mar 13, 2013 at 02:50:47AM +0000, Blake Pfankuch - Mailing List wrote:
> > I have seen this the past few nights in a row, however based on a problem we have been having with a customer VPN im thinking its been going on longer.  Tracert section below.  Hop 13 has been really high latency for me during issues.  When its acting up, latency between home and work is 160-200ms.  At 6am this morning, it was 60ms and was consistent all day until 530pm today according to smokeping.
> >  5    19 ms    20 ms    20 ms  te-0-10-0-13-ar02.aurora.co.denver.comcast.net [68.86.103.82]
> >   6    23 ms    16 ms    16 ms  he-3-9-0-0-cr01.denver.co.ibone.comcast.net [68.86.92.21]
> >   7    53 ms    62 ms    31 ms  68.86.89.190
> >   8    34 ms    30 ms    29 ms  pos-0-0-0-0-pe01.1950stemmons.tx.ibone.comcast.net [68.86.86.90]
> >   9    35 ms    31 ms    29 ms  sl-st31-dal-.sprintlink.net [144.232.25.33]
> > 10    31 ms    31 ms    32 ms  144.232.11.207
> > 11    86 ms    86 ms    30 ms  144.232.1.162
> > 12    42 ms    45 ms    40 ms  sl-crs2-kc-0-5-5-0.sprintlink.net [144.232.24.7]
> > 13   550 ms   110 ms   186 ms  144.232.1.101
> > 14   193 ms   300 ms   119 ms  sl-crs2-che-0-0-2-0.sprintlink.net [144.232.18.120]
> > 15   114 ms   102 ms   104 ms  sl-gw16-che-15-0-0.sprintlink.net [144.232.6.50]
> > 
> > Anyone seeing something similar or anyone insightful from either organization who might be able to help out?
> > 
> > Thanks,
> > Blake
> 
> > _______________________________________________
> > Outages mailing list
> > Outages at outages.org
> > https://puck.nether.net/mailman/listinfo/outages