[outages] comcast chicago blackholing

[John Neiberger]

Okay, we should be good to go. For real this time. Had some lingering
fabric-related issues to clear up. Everything looks good now. Let me
know if anyone sees any more problems through that router.

Thanks,
John

On Tue, Nov 17, 2015 at 12:37 PM, John Neiberger <jneiberger at gmail.com> wrote:
> May have spoken too soon. Something is still apparently sick, so we've
> taken the router back out of routing for now.
>
> John
>
> On Tue, Nov 17, 2015 at 12:17 PM, John Neiberger <jneiberger at gmail.com> wrote:
>> And by "partial" I mean that it was very sick, but not sick enough to
>> trigger a switchover to the backup RSP. That had to be done manually.
>>
>> John
>>
>> On Tue, Nov 17, 2015 at 12:15 PM, John Neiberger <jneiberger at gmail.com> wrote:
>>> Thanks, all. Our side should be resolved now. Had a combination of a
>>> linecard fabric failure and a partial RSP failure. We're back in
>>> working shape and BGP from our core to that edge router is back up.
>>> Should be safe for peers to turn sessions back up.
>>>
>>> John
>>>
>>> On Tue, Nov 17, 2015 at 10:15 AM, Jared Mauch <jared at puck.nether.net> wrote:
>>>>         This is both an issue over V6 and V4, eg:
>>>>
>>>> Jareds-Mac-mini:~% traceroute puck
>>>> traceroute to puck.nether.net (204.42.254.5), 64 hops max, 52 byte packets
>>>>  1  10.0.0.1 (10.0.0.1)  0.603 ms  0.418 ms  0.403 ms
>>>>  2  173-167-0-110-michigan.hfc.comcastbusiness.net (173.167.0.110)  2.044 ms  2.019 ms  2.179 ms
>>>>  3  96.120.40.141 (96.120.40.141)  17.399 ms  12.408 ms  9.639 ms
>>>>  4  68.85.85.37 (68.85.85.37)  14.371 ms  8.378 ms  15.355 ms
>>>>  5  te-0-8-0-6-ar02.pontiac.mi.michigan.comcast.net (68.87.191.153)  14.025 ms
>>>>     te-0-8-0-8-ar02.pontiac.mi.michigan.comcast.net (69.139.255.69)  14.749 ms  18.706 ms
>>>>  6  be-33668-cr02.350ecermak.il.ibone.comcast.net (68.86.90.45)  21.955 ms  20.456 ms  19.259 ms
>>>>  7  he-0-16-0-0-pe03.350ecermak.il.ibone.comcast.net (68.86.88.146)  18.333 ms  17.245 ms  16.952 ms
>>>> ^C
>>>>
>>>> Jareds-Mac-mini:~% traceroute6 puck
>>>> traceroute6 to puck.nether.net (2001:418:3f4::5) from 2601:401:3:6a00:ad42:ce75:8a0:c370, 64 hops max, 12 byte packets
>>>>  1  2601:401:3:6a00:20f:f7ff:feab:1ec0  1.135 ms  0.965 ms  0.948 ms
>>>>  2  2601:401:3:6a00:4694:fcff:fecd:64bc  3.710 ms  3.544 ms  21.221 ms
>>>>  3  2001:558:6007:24::1  30.051 ms  10.909 ms  10.868 ms
>>>>  4  te8-2-ur02.wannarbor.mi.michigan.comcast.net  10.664 ms  10.451 ms  10.570 ms
>>>>  5  xe-11-1-0-0-sur01.rochestrhlls.mi.michigan.comcast.net  14.826 ms
>>>>     te-0-8-0-5-ar02.taylor.mi.michigan.comcast.net  13.890 ms  13.179 ms
>>>>  6  te-0-7-0-13-ar01.pontiac.mi.michigan.comcast.net  18.917 ms  15.503 ms
>>>>     te-0-8-0-11-ar01.pontiac.mi.michigan.comcast.net  13.686 ms
>>>>  7  be-33668-cr01.350ecermak.il.ibone.comcast.net  22.316 ms  22.014 ms  28.086 ms
>>>>  8  hu-0-16-0-1-pe03.350ecermak.il.ibone.comcast.net  20.933 ms  20.740 ms  20.831 ms
>>>> ^C
>>>>
>>>>         - Jared
>>>>
>>>>
>>>> On Tue, Nov 17, 2015 at 10:02:58AM -0700, John Neiberger wrote:
>>>>> Can you give me some more details please? I'll take a look.
>>>>>
>>>>> Thanks,
>>>>> John
>>>>>
>>>>> On Tue, Nov 17, 2015 at 10:00 AM, Jared Mauch via Outages
>>>>> <outages at outages.org> wrote:
>>>>> >
>>>>> >         Anyone know what is going on here?  It seems to have started
>>>>> > around 1600 utc and is still ongoing.
>>>>> >
>>>>> >         - Jared
>>>>> >
>>>>> > --
>>>>> > Jared Mauch  | pgp key available via finger from jared at puck.nether.net
>>>>> > clue++;      | http://puck.nether.net/~jared/  My statements are only mine.
>>>>> > _______________________________________________
>>>>> > Outages mailing list
>>>>> > Outages at outages.org
>>>>> > https://puck.nether.net/mailman/listinfo/outages
>>>>
>>>> --
>>>> Jared Mauch  | pgp key available via finger from jared at puck.nether.net
>>>> clue++;      | http://puck.nether.net/~jared/  My statements are only mine.