[outages] SF South Bay: chronic latency/packet loss between Abovenet/Comcast at Great Oaks

[Jeremy Chadwick]

There's an issue I've been tracking for a few months now pertaining to a
network link between Abovenet and Comcast which appears to become
saturated (or impacted negatively in some way) at nearly the same time
every night, and lasts for numerous hours, then ceases -- on a
near-daily basis (no exaggeration).

Latency and packet loss occur during this time, with latency hitting
150ms (sometimes higher), with packet loss ranging from 0.5% to 2.0%.
I've been storing periodic traceroutes/mtrs for over a month showing
this problem, and been tracking start/end times as well.

Thankfully I own devices/have connectivity on both ends (src and dst,
thus can provide mtrs/traceroutes from both directions.  Analysis so
far, done by myself as well as senior network techs at my co-lo
provider, confirms this issue is with a link between Abovenet/Comcast,
likely within the San Jose Great Oaks POP (which I'm familiar with as
part of my job).

I opened up a ticket on DSLR/BBR's Comcast Direct forum (which only
Comcast employees can respond to/view tickets for) over a month ago.
Someone has been viewing it, but nobody has replied except me.

I've since made the issue public, where (of course) the general Internet
community does not quite understand how peering arrangements/contracts
work (people think that any company who has a contract with Abovenet can
report issues, but that is simply not the case; you must be a POC for
the transport to report issues with it), nor do they understand how a
co-lo provider changing route preferencing can impact the provider
financially (based on billing metrics, etc.).  My co-lo provider is very
strict with their routing policies, and it has to do with financial
reasons that are their own business, not mine.

The public thread is here, which also includes start/end times,
traceroutes (both directions), and so on.  I update it every day when
the issue happens, and ~90% of the time edit my posts when the issue
ends.

http://www.dslreports.com/forum/r27085601-SF-South-Bay-Chronic-IP-network-latency-nightly

Anyway, all the technical details aside:

Is there anyone on this list who works for Comcast who can contact me
off-list who is willing to investigate this and drive it to completion?

An alternative would be for someone to contact me off-list with the name
or Email address of someone (or division) who handles issues like this
at Comcast.  I'd love for Abovenet to get involved, but I have no
contractual obligation to them.  (If there is an Abovenet individual who
is willing to investigate this "pro bono" per se, that would be
awesome, but I imagine such is often above one's pay grade).

-- 
| Jeremy Chadwick                              jdc at parodius.com |
| Parodius Networking                     http://www.parodius.com/ |
| UNIX Systems Administrator                 Mountain View, CA, US |
| Making life hard for others since 1977.             PGP 4BD6C0CB |