[outages] SF South Bay: chronic latency/packet loss between Abovenet/Comcast at Great Oaks

[Ren Provo]

Hi Jeremy,

When the issue was raised a week or two ago there seemed to be a route
announcement issue for 72.20.98.67.  When your colo provider changed
their policy did they update filters with their upstream?

Cheers, -ren, who will confirm there is no congestion with Abovenet on
the port in SJC to Comcast.

On Tue, Apr 10, 2012 at 9:44 PM, Jeremy Chadwick
<outages at jdc.parodius.com> wrote:
> There's an issue I've been tracking for a few months now pertaining to a
> network link between Abovenet and Comcast which appears to become
> saturated (or impacted negatively in some way) at nearly the same time
> every night, and lasts for numerous hours, then ceases -- on a
> near-daily basis (no exaggeration).
>
> Latency and packet loss occur during this time, with latency hitting
> 150ms (sometimes higher), with packet loss ranging from 0.5% to 2.0%.
> I've been storing periodic traceroutes/mtrs for over a month showing
> this problem, and been tracking start/end times as well.
>
> Thankfully I own devices/have connectivity on both ends (src and dst,
> thus can provide mtrs/traceroutes from both directions.  Analysis so
> far, done by myself as well as senior network techs at my co-lo
> provider, confirms this issue is with a link between Abovenet/Comcast,
> likely within the San Jose Great Oaks POP (which I'm familiar with as
> part of my job).
>
> I opened up a ticket on DSLR/BBR's Comcast Direct forum (which only
> Comcast employees can respond to/view tickets for) over a month ago.
> Someone has been viewing it, but nobody has replied except me.
>
> I've since made the issue public, where (of course) the general Internet
> community does not quite understand how peering arrangements/contracts
> work (people think that any company who has a contract with Abovenet can
> report issues, but that is simply not the case; you must be a POC for
> the transport to report issues with it), nor do they understand how a
> co-lo provider changing route preferencing can impact the provider
> financially (based on billing metrics, etc.).  My co-lo provider is very
> strict with their routing policies, and it has to do with financial
> reasons that are their own business, not mine.
>
> The public thread is here, which also includes start/end times,
> traceroutes (both directions), and so on.  I update it every day when
> the issue happens, and ~90% of the time edit my posts when the issue
> ends.
>
> http://www.dslreports.com/forum/r27085601-SF-South-Bay-Chronic-IP-network-latency-nightly
>
> Anyway, all the technical details aside:
>
> Is there anyone on this list who works for Comcast who can contact me
> off-list who is willing to investigate this and drive it to completion?
>
> An alternative would be for someone to contact me off-list with the name
> or Email address of someone (or division) who handles issues like this
> at Comcast.  I'd love for Abovenet to get involved, but I have no
> contractual obligation to them.  (If there is an Abovenet individual who
> is willing to investigate this "pro bono" per se, that would be
> awesome, but I imagine such is often above one's pay grade).
>
> --
> | Jeremy Chadwick                              jdc at parodius.com |
> | Parodius Networking                     http://www.parodius.com/ |
> | UNIX Systems Administrator                 Mountain View, CA, US |
> | Making life hard for others since 1977.             PGP 4BD6C0CB |
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages