[outages] Comcast DNS outage on the evening of Sunday, November 28

[Jeremy Chadwick]

On Tue, Nov 30, 2010 at 11:57:46AM -0600, Frank Bulk - iName.com wrote:
> There was all kinds of press coverage about the outage, but not a lot of
> technical detail.  
> 
> Based on reports, Comcast Internet customers in Massachusetts, Connecticut,
> Maryland, Virginia, New York, New Hampshire, and parts of Washington, D.C.
> were without access their DNS server(s) from about 8:30 pm Eastern to 11:30
> pm Eastern, though some say it started as early at 8 pm and ended after
> midnight.  NBC was told by a Comcast technician that outages were spread out
> over six states in the Northeast and Mid-Atlantic.  Comcast said that phone
> and CATV customers were not affected.
> 
> According to published reports, Comcast was performing regular maintenance
> on their DNS servers and something went wrong.  A Comcast spokesman said on
> Monday that "Last night, our engineers identified a server issue that was
> affecting Internet service in Boston, parts of New England and the
> DC/Beltway areas".
> 
> Comcast did not share how many customers were affected.
> 
> BTW, did anyone look at http://dns.comcast.net/status.php during this
> outage?

Reading material for those who are interested, but no technical
explanation given either (not surprising; outages like this are often
kept quiet for some time while the company tries to figure out what
happened, why, and get impacted customer counts).

http://www.dslreports.com/shownews/Comcast-Suffers-Major-East-Coast-Outage-111568

Depending on Comcast's infrastructure and the nature of the failure (I'm
on the west coast and witnessed nothing), it might not be possible to
determine the number of impacted customers.  For example, if I shut off
one of our primary nameservers (powered it down, shut down named/BIND,
etc.) I'd have no idea how many people were "impacted"; I'd have to
speculate based on the number of queries seen against the secondary.

Anyway, all the what-ifs should probably be discussed on outages-discuss
and not here, but I just wanted to throw that out there.

-- 
| Jeremy Chadwick                                   jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.              PGP: 4BD6C0CB |