[outages] Comcast residential in south SF Bay Area

[Jeremy Chadwick]

This is the 3rd time in the past couple months (at most) where Comcast
has done this type of maintenance in Silicon Valley.  Important detail:
this is cable network/node maintenance, not IP network.

Every time this work has been done in the past few months, it manifests
in mostly the same way.  Per discussions with other list members as well
as with some folks in Comcast Network Maintenance Engineering, what I
see for my specific area is not necessarily what's going on a city or
two over (or sometimes even within the same city!).  That said, this is
a "general overview" of what I've personally witnessed (with months of
logs to back it up):

They appear to be trying to introduce additional downstream channels.
Where I live (Old Mountain View), the freqs were 735/741/747/753MHz.
During the last maintenance (2-3 weeks ago) they got this working (in
contrast the maintenance 3-4 weeks prior to that, which was a failure
and had to be rolled back), but for Mountain View there was a period of
over 24 hours of 25-30% packet loss when reaching the CMTS.  Tonight's
work is now a horse of a different colour: they appear to have pulled
735-753MHz and instead introduced 657MHz (?!?).  So right now I've 21
bonded DS channels, which is a bit odd (pun intended), with a very
elevated rate of DS correctables and occasional batches of
uncorrectables.  The latter is likely due to technicians still working
on things (it's 02:53 PST), but the former, well... I'll just say it:

There are network issues here in my area, particularly an abnormally
high rate of DS correctables, i.e. RS/FEC is being applied heavily,
where the pre-FEC BER rate is no where near what it was prior to July
2016.  I've been in contact with Network Maintenance Engineering techs
and supervisors out of Menlo Park for months about this, and met in
person/on foot in the field several times.  Every time they do the above
work, though, the issue is exacerbated -- in almost all cases DS SNR
gets substantially worse (2-3dB at times), as does DS power.  It's often
a specific frequency range that suffers this way.  What makes this extra
weird is that for a period of almost 48 hours on 9/24, the issues I
describe in this paragraph *completely disappeared*, then reappeared out
of no where; this leaves me with the impression there is some
interconnect wiring, or hub, or *something* on the network between here
and Santa Clara that got jostled in such a way where the issue went
away.  It really smells interference seepage.

I can't speak for others, but all this is work is extra annoying for me,
because every time they add new frequencies I have a boatload of scripts
and DBs to alter/modify/manipulate as a result.  It takes me a couple
hours, and in a couple occasions right when I got everything updated,
they'd make another change and I'd have to do the same again.  Like
for tonight, I'm actually afraid to change any of my code because for
all I know in 2 hours they might change it all again.

I'm fine with maintenance, but this feels much like someone "playing
around in production", rather than having a plan of action and proper
rollback procedure.  It's more like "eh, it mostly kinda works, I guess
we'll leave it like that for now", rinse lather repeat every 2-4 weeks.

-- 
| Jeremy Chadwick                                   jdc at koitsu.org |
| UNIX Systems Administrator                http://jdc.koitsu.org/ |
| Making life hard for others since 1977.             PGP 4BD6C0CB |

On Wed, Nov 30, 2016 at 12:28:54AM -0800, Grant Ridder via Outages wrote:
> Comcast appears to be experiencing an outage in the south SF Bay Area.  Not sure how wide spread it is but confirmed on the automated customer service line
> 
> Grant
> 
> Sent from my iPhone
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages