[outages] Metro & North Atlanta (AT&T Metro-Ethernet Outage)

[Aubrey Wells]

All our circuits are back up now. Anybody else getting good news right
about now? Reportedly this took out 1000+ MetroE circuits. Bad day to
be a net eng at AT&T...


On Thu, Sep 20, 2012 at 3:32 PM, Aubrey Wells <aubrey at vocalcloud.com> wrote:
> Update from ATT with more details:
>
> The issue has been isolated to a LAG bundle between two 7609 routers
> in Atlanta and ATS is isolating the offending VLAN so issue can be
> mitigated. Network teams have identified the traffic is impacting
> ATLNGACS4CW. The source of the traffic has been traced back to
> ATLNGAPP4AW. Network teams are attempting to back trace the source of
> the traffic on ATLNGAPP4AW to either another router or an access
> circuit on this device. Troubleshooting through the network continues
> to locate the root source of the VLAN loop. This is a process of
> removing LAG bundles and core trunks from service until the CPU and
> traffic normalize. The process is underway with ATS and the Cisco is
> being engaged for additional support.  There is no ETTR at this time.
>
> On Thu, Sep 20, 2012 at 3:11 PM, Seth Mattinen <sethm at rollernet.us> wrote:
>> On 9/20/12 12:03 PM, Aubrey Wells wrote:
>>> I think it goes further than that. I have customers as far south as
>>> Newnan and as far north as Alpharetta down, all coming out of
>>> different CO/wirecenters.
>>>
>>
>>
>> Then it could be the top router that takes the all uplinks from the
>> aggregation ones that ties them all together that failed. The last time
>> I inquired how opt-e-man was laid out it's pretty basic. It's the cheap
>> no frills no redundancy Ethernet offering. Otherwise you have to pay for
>> SONET.
>>
>> ~Seth
>> _______________________________________________
>> Outages mailing list
>> Outages at outages.org
>> https://puck.nether.net/mailman/listinfo/outages