[outages] AT&T / Verizon - Philadelphia and Moosic areas (PA)

[Jeremy Chadwick]

We witnessed something similar, but the time frame does not match up
with what you provided.

At 19:11 PST we witnessed packet loss to a Verizon router, followed by
the physif on two of our routers going down (SONET links, OC12) at 19:17
PST.  Things recovered by 19:19 PST.  The physifs are directly tied to
AT&T MPLS but for Verizon IP traffic.  Our routers are located
(physically) in an AT&T datacenter in Arizona.  So in our case AT&T
would be the LEC, not Verizon.

Routers in question, with SONET pathing data (indicating router names
and IP addresses of those AT&T routers on AT&T's side):

router01:
  Last flapped   : 2011-12-10 03:17:19 UTC (01:56:12 ago)
  Host name  : phmaz31ck
  Interface  : POS4/1
  IP Address : 12.113.38.5

router02:
  Last flapped   : 2011-12-10 03:17:19 UTC (01:57:32 ago)
  Host name  : phmaz31ck
  Interface  : POS4/2
  IP Address : 12.113.38.9

I don't know if this is what you saw or not.

I will point out, however, that AT&T has a very long and
well-established history of "doing whatever, whenever", no matter how
large the scale.  A few years ago we witnessed them doing router OS
upgrades across the entire United States over the course of two
evenings, with absolutely no BGP preferencing announcements done prior;
just straight up "bam, router down, too bad".  I am not implying this is
what AT&T was doing, but rather that they may have administratively
done something to part of their mesh without bothering to tell anyone,
thinking "nobody would notice".

-- 
| Jeremy Chadwick                                jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                   Mountain View, CA, US |
| Making life hard for others since 1977.               PGP 4BD6C0CB |

On Fri, Dec 09, 2011 at 10:38:52PM -0600, Stefan wrote:
> Lost two sites, simultaneously, over 2.5 hrs ago, in PA area. MPLS
> services from AT&T, supposedly running with Verizon as LEC. Anybody
> else?!!
> 
> ***Stefan Mititelu
> http://twitter.com/netfortius
> http://www.linkedin.com/in/netfortius
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages