[outages] comcast chicago blackholing

[John Neiberger]

And by "partial" I mean that it was very sick, but not sick enough to
trigger a switchover to the backup RSP. That had to be done manually.

John

On Tue, Nov 17, 2015 at 12:15 PM, John Neiberger <jneiberger at gmail.com> wrote:
> Thanks, all. Our side should be resolved now. Had a combination of a
> linecard fabric failure and a partial RSP failure. We're back in
> working shape and BGP from our core to that edge router is back up.
> Should be safe for peers to turn sessions back up.
>
> John
>
> On Tue, Nov 17, 2015 at 10:15 AM, Jared Mauch <jared at puck.nether.net> wrote:
>>         This is both an issue over V6 and V4, eg:
>>
>> Jareds-Mac-mini:~% traceroute puck
>> traceroute to puck.nether.net (204.42.254.5), 64 hops max, 52 byte packets
>>  1  10.0.0.1 (10.0.0.1)  0.603 ms  0.418 ms  0.403 ms
>>  2  173-167-0-110-michigan.hfc.comcastbusiness.net (173.167.0.110)  2.044 ms  2.019 ms  2.179 ms
>>  3  96.120.40.141 (96.120.40.141)  17.399 ms  12.408 ms  9.639 ms
>>  4  68.85.85.37 (68.85.85.37)  14.371 ms  8.378 ms  15.355 ms
>>  5  te-0-8-0-6-ar02.pontiac.mi.michigan.comcast.net (68.87.191.153)  14.025 ms
>>     te-0-8-0-8-ar02.pontiac.mi.michigan.comcast.net (69.139.255.69)  14.749 ms  18.706 ms
>>  6  be-33668-cr02.350ecermak.il.ibone.comcast.net (68.86.90.45)  21.955 ms  20.456 ms  19.259 ms
>>  7  he-0-16-0-0-pe03.350ecermak.il.ibone.comcast.net (68.86.88.146)  18.333 ms  17.245 ms  16.952 ms
>> ^C
>>
>> Jareds-Mac-mini:~% traceroute6 puck
>> traceroute6 to puck.nether.net (2001:418:3f4::5) from 2601:401:3:6a00:ad42:ce75:8a0:c370, 64 hops max, 12 byte packets
>>  1  2601:401:3:6a00:20f:f7ff:feab:1ec0  1.135 ms  0.965 ms  0.948 ms
>>  2  2601:401:3:6a00:4694:fcff:fecd:64bc  3.710 ms  3.544 ms  21.221 ms
>>  3  2001:558:6007:24::1  30.051 ms  10.909 ms  10.868 ms
>>  4  te8-2-ur02.wannarbor.mi.michigan.comcast.net  10.664 ms  10.451 ms  10.570 ms
>>  5  xe-11-1-0-0-sur01.rochestrhlls.mi.michigan.comcast.net  14.826 ms
>>     te-0-8-0-5-ar02.taylor.mi.michigan.comcast.net  13.890 ms  13.179 ms
>>  6  te-0-7-0-13-ar01.pontiac.mi.michigan.comcast.net  18.917 ms  15.503 ms
>>     te-0-8-0-11-ar01.pontiac.mi.michigan.comcast.net  13.686 ms
>>  7  be-33668-cr01.350ecermak.il.ibone.comcast.net  22.316 ms  22.014 ms  28.086 ms
>>  8  hu-0-16-0-1-pe03.350ecermak.il.ibone.comcast.net  20.933 ms  20.740 ms  20.831 ms
>> ^C
>>
>>         - Jared
>>
>>
>> On Tue, Nov 17, 2015 at 10:02:58AM -0700, John Neiberger wrote:
>>> Can you give me some more details please? I'll take a look.
>>>
>>> Thanks,
>>> John
>>>
>>> On Tue, Nov 17, 2015 at 10:00 AM, Jared Mauch via Outages
>>> <outages at outages.org> wrote:
>>> >
>>> >         Anyone know what is going on here?  It seems to have started
>>> > around 1600 utc and is still ongoing.
>>> >
>>> >         - Jared
>>> >
>>> > --
>>> > Jared Mauch  | pgp key available via finger from jared at puck.nether.net
>>> > clue++;      | http://puck.nether.net/~jared/  My statements are only mine.
>>> > _______________________________________________
>>> > Outages mailing list
>>> > Outages at outages.org
>>> > https://puck.nether.net/mailman/listinfo/outages
>>
>> --
>> Jared Mauch  | pgp key available via finger from jared at puck.nether.net
>> clue++;      | http://puck.nether.net/~jared/  My statements are only mine.