[outages] comcast chicago blackholing

[John Neiberger]

May have spoken too soon. Something is still apparently sick, so we've
taken the router back out of routing for now.

John

On Tue, Nov 17, 2015 at 12:17 PM, John Neiberger <jneiberger at gmail.com> wrote:
> And by "partial" I mean that it was very sick, but not sick enough to
> trigger a switchover to the backup RSP. That had to be done manually.
>
> John
>
> On Tue, Nov 17, 2015 at 12:15 PM, John Neiberger <jneiberger at gmail.com> wrote:
>> Thanks, all. Our side should be resolved now. Had a combination of a
>> linecard fabric failure and a partial RSP failure. We're back in
>> working shape and BGP from our core to that edge router is back up.
>> Should be safe for peers to turn sessions back up.
>>
>> John
>>
>> On Tue, Nov 17, 2015 at 10:15 AM, Jared Mauch <jared at puck.nether.net> wrote:
>>>         This is both an issue over V6 and V4, eg:
>>>
>>> Jareds-Mac-mini:~% traceroute puck
>>> traceroute to puck.nether.net (204.42.254.5), 64 hops max, 52 byte packets
>>>  1  10.0.0.1 (10.0.0.1)  0.603 ms  0.418 ms  0.403 ms
>>>  2  173-167-0-110-michigan.hfc.comcastbusiness.net (173.167.0.110)  2.044 ms  2.019 ms  2.179 ms
>>>  3  96.120.40.141 (96.120.40.141)  17.399 ms  12.408 ms  9.639 ms
>>>  4  68.85.85.37 (68.85.85.37)  14.371 ms  8.378 ms  15.355 ms
>>>  5  te-0-8-0-6-ar02.pontiac.mi.michigan.comcast.net (68.87.191.153)  14.025 ms
>>>     te-0-8-0-8-ar02.pontiac.mi.michigan.comcast.net (69.139.255.69)  14.749 ms  18.706 ms
>>>  6  be-33668-cr02.350ecermak.il.ibone.comcast.net (68.86.90.45)  21.955 ms  20.456 ms  19.259 ms
>>>  7  he-0-16-0-0-pe03.350ecermak.il.ibone.comcast.net (68.86.88.146)  18.333 ms  17.245 ms  16.952 ms
>>> ^C
>>>
>>> Jareds-Mac-mini:~% traceroute6 puck
>>> traceroute6 to puck.nether.net (2001:418:3f4::5) from 2601:401:3:6a00:ad42:ce75:8a0:c370, 64 hops max, 12 byte packets
>>>  1  2601:401:3:6a00:20f:f7ff:feab:1ec0  1.135 ms  0.965 ms  0.948 ms
>>>  2  2601:401:3:6a00:4694:fcff:fecd:64bc  3.710 ms  3.544 ms  21.221 ms
>>>  3  2001:558:6007:24::1  30.051 ms  10.909 ms  10.868 ms
>>>  4  te8-2-ur02.wannarbor.mi.michigan.comcast.net  10.664 ms  10.451 ms  10.570 ms
>>>  5  xe-11-1-0-0-sur01.rochestrhlls.mi.michigan.comcast.net  14.826 ms
>>>     te-0-8-0-5-ar02.taylor.mi.michigan.comcast.net  13.890 ms  13.179 ms
>>>  6  te-0-7-0-13-ar01.pontiac.mi.michigan.comcast.net  18.917 ms  15.503 ms
>>>     te-0-8-0-11-ar01.pontiac.mi.michigan.comcast.net  13.686 ms
>>>  7  be-33668-cr01.350ecermak.il.ibone.comcast.net  22.316 ms  22.014 ms  28.086 ms
>>>  8  hu-0-16-0-1-pe03.350ecermak.il.ibone.comcast.net  20.933 ms  20.740 ms  20.831 ms
>>> ^C
>>>
>>>         - Jared
>>>
>>>
>>> On Tue, Nov 17, 2015 at 10:02:58AM -0700, John Neiberger wrote:
>>>> Can you give me some more details please? I'll take a look.
>>>>
>>>> Thanks,
>>>> John
>>>>
>>>> On Tue, Nov 17, 2015 at 10:00 AM, Jared Mauch via Outages
>>>> <outages at outages.org> wrote:
>>>> >
>>>> >         Anyone know what is going on here?  It seems to have started
>>>> > around 1600 utc and is still ongoing.
>>>> >
>>>> >         - Jared
>>>> >
>>>> > --
>>>> > Jared Mauch  | pgp key available via finger from jared at puck.nether.net
>>>> > clue++;      | http://puck.nether.net/~jared/  My statements are only mine.
>>>> > _______________________________________________
>>>> > Outages mailing list
>>>> > Outages at outages.org
>>>> > https://puck.nether.net/mailman/listinfo/outages
>>>
>>> --
>>> Jared Mauch  | pgp key available via finger from jared at puck.nether.net
>>> clue++;      | http://puck.nether.net/~jared/  My statements are only mine.