[Outages] Level3 / Yipes

[Adam Kujawski]

I've seen several Emergency Maintenance notifications from Level3 last
night, from all across their network -- apparently they were rebooting
a bunch (all?) of their routers:

"Demand maintenance is needed to revert router code to a previous
version to prevent possible network outages. This change should only
take a few minutes while the router reloads, and service impact will
be minimal."

>From where I sit, I saw a bunch of route instability and loss of
reachability across their network for up to 30 minutes. When I finally
got ahold of Level3 around 12:30AM, they said they were just about to
start on the west coast, doing one router at a time.

-Adam


On 8/30/07, Jayson Navitsky <jnavitsky at gmail.com> wrote:
> I'm seeing some entertaining routes to sites on Yipes/Level3,
> according to Yipes! support Level3 just lost an OC48 between NY &
> Chicago.
>
> (This is to a site on Yipes)
>   1   137 ms   134 ms   159 ms  5.sub-66-174-22.myvzw.com [66.174.22.5]
>   2     *        *        *     Request timed out.
>   3    98 ms   131 ms   132 ms  113.sub-66-174-120.myvzw.com [66.174.120.113]
>   4   169 ms   130 ms   133 ms  65.sub-66-174-22.myvzw.com [66.174.22.65]
>   5   147 ms   155 ms   131 ms  98.sub-66-174-22.myvzw.com [66.174.22.98]
>   6     *        *        *     Request timed out.
>   7   173 ms   133 ms   290 ms  64.200.80.25
>   8   426 ms   133 ms    76 ms  nwrknj1wcx1-pos6-0.wcg.net [64.200.68.138]
>   9   111 ms   131 ms   131 ms  nycmny2wcx3-pos11-1.wcg.net [64.200.240.233]
>  10   130 ms   157 ms   134 ms  64.200.249.154
>  11   134 ms   144 ms   139 ms  64.200.249.149
>  12     *      158 ms     *     64.200.249.150
>  13     *      144 ms   162 ms  64.200.249.149
>  14     *      181 ms     *     64.200.249.150
>  15     *        *      163 ms  64.200.249.149
>  16   204 ms     *      178 ms  64.200.249.150
>  17     *        *      198 ms  64.200.249.149
>  18     *        *        *     Request timed out.
>  19     *      194 ms   185 ms  64.200.249.149
>  20     *      334 ms     *     64.200.249.150
>  21     *        *        *     Request timed out.
>
>
> -J
> _______________________________________________
> Outages mailing list
> Outages at isotf.org
> http://isotf.org/mailman/listinfo/outages
>