[outages] Lumen issues between East Coast and West Coast?

Anurag Bhatia me at anuragbhatia.com
Tue Jul 9 10:27:03 EDT 2024


I saw issues fixing at 9:16 UTC but came up again at 11:42 and went on
until 14:03. Since the last 23mins it seems fine again.

(Not a direct Lumen customer but my VM and object storage provider have a
Lumen network in between).

On Tue, Jul 9, 2024 at 7:53 PM Tim Burke <tim at mid.net> wrote:

> One of my Lumen transit circuits in Houston came back up at 2:30am but
> seeing triple-digit latency across it.
>
> Another Lumen circuit (same facility in Houston) off of another router
> still remains hard down.
>
> On Jul 9, 2024, at 9:14 AM, Dustin Ortiz via Outages <outages at outages.org>
> wrote:
>
> From lumen regarding the Generator outages at the Houston Gateway site:
>
> 2024-07-09 12:58:07 GMT - Upon evaluation, technicians have identified
> that the chillers are not effectively cooling the equipment as intended.
> Efforts are underway to diagnose and resolve the issue. In support of these
> efforts, Lumen Management has reached out to the equipment vendors for
> additional assistance as technicians continue to work towards restoring the
> nodes.
>
> 2024-07-09 11:36:09 GMT - The chillers have begun to cool the equipment
> and Field Operations continues in efforts to begin powering up priority
> network elements.
>
> 2024-07-09 10:20:14 GMT - The 2.5 Mega Watt generator has been brought
> online and field teams are diligently working to activate the chillers,
> which will facilitate the cooling of our equipment. As the cooling process
> commences, technicians are sequentially restoring critical core equipment.
> Additionally, we have procured several replacement cards in anticipation of
> potential equipment damage caused by elevated temperatures
>
> 2024-07-09 08:57:26 GMT - Crews have advised that a 2-megawatt generator
> has arrived at the gateway site in addition to the initial generator -
> rated at 25-kilowatts, and wiring efforts are preparing to commence at this
> time. Teams have advised that wiring is underway and anticipated to be
> completed on the 2-megawatt generator in approximately 1 hour, and the
> completion of wiring on the 25-kilowatt generator will follow. Once the
> installation of the 2-megawatt generator is completed, crews have advised
> that the expanded load capacity will be used to power 2 chillers which will
> be used to begin cooling site temperatures and power priority equipment as
> additional installation efforts are completed. Additionally, an existing
> generator onsite has been recovered, and DC power has been stabilized. The
> next update will be provided upon the installation of the 2-megawatt
> generator, or as new information becomes available.
>
> 2024-07-09 08:40:13 GMT - The first generator has arrived onsite and Lumen
> teams have advised that preparations are underway to commence with the
> wiring process to install the generator. While the single generator is not
> anticipated to power the site singlehandedly, it is anticipated that the
> addition will power some equipment necessary to cool the site and assist
> with balancing the power load. Lumen teams are closely monitoring
> conditions and completing assessments on a regular basis. The next update
> will be provided upon the completed wiring of the first generator, or as
> new information becomes available.
>
> 2024-07-09 07:41:25 GMT - Investigations by the Critical Infrastructure
> Technical Support (CITS) Team revealed the two on-site generators at a
> Lumen Gateway Facility in Houston, TX were experiencing power loading
> issues. As a result, the site was running on one on-site generator for most
> of the day and the generator was able to support the load. At approximately
> 01:00 GMT, the second on-site generator began experiencing issues as well.
> The generators will turn on and begin to run which begins to power the
> three on-site Direct Current (DC) power plants; however, after about 30
> minutes, the generator is unable to regulate itself and then shuts off. As
> a result of this, teams were having to toggle between charging the DC plant
> via the generators while running, and cooling the site via the cooling
> system that is powered by the on-site DC batteries. At 02:24 GMT, teams
> began to gracefully shut down shelves within an Infinera ring to help
> conserve power to the affected site and to protect any equipment that may
> be bouncing as a result of the generator failures. Other equipment at the
> site is being evaluated as the outage progresses.
>
> There are currently two replacement generators en route to the site. One
> generator is coming from Dallas, TX and is expected to be on site and wired
> up within 2 hours. The second generator is en route from Midland, TX and
> should be arriving on site within the next 6-12 hours. Both generators are
> needed to power the site due to the size of the gateway site. The next
> update will be provided by 16:00 GMT or as new information becomes
> available.
>
> 2024-07-09 03:53:10 GMT - On July 09, 2024, at 03:40 GMT, Lumen was
> alerted to service impact in Houston, TX caused by severe area weather
> conditions resulting from Hurricane Beryl. As this network fault is
> impacting multiple clients, the event has increased visibility with Lumen
> leadership; however, as local authorities have issued a state of emergency
> requiring civilian parties to shelter in place, Lumen has postponed further
> action until the order has been lifted to ensure the safety of personnel.
> In the interim, Lumen is actively monitoring Hurricane Beryl's path and
> will remain in close contact with local, state, and federal offices of
> emergency management to provide close coordination and timely information.
> Each outage situation will be evaluated, and a determination made if a
> remote restoration action is viable or if on-site support is necessary. For
> the latter, and if evacuations occur, Field Operations personnel will
> dispatch once government authorization is received. The safety of our
> employees is of utmost concern; we are taking every step to ensure their
> safety, which may delay our network repair response times. Updates will
> continue to be provided every 4 hours or as new information is received.
>
>
>
> *Dustin Ortiz | *VP Systems Engineering
> *214-945-2201 | dortiz at zthernet.com <dortiz at zthernet.com>*
>
> *From:* Outages <outages-bounces at outages.org> *On Behalf Of *Anurag
> Bhatia via Outages
> *Sent:* Tuesday, July 9, 2024 4:29 AM
> *To:* Outage Mailing list <outages at outages.org>
> *Subject:* Re: [outages] Lumen issues between East Coast and West Coast?
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
>
> (Seems like my last message is on hold for moderation)
>
> I see the issue is resolved just now. It was there from 1:30 am to 9:16 am
> UTC.
>
>
>
>
> On Tue, Jul 9, 2024 at 2:11 PM Anurag Bhatia <me at anuragbhatia.com> wrote:
>
> Since 1:30am UTC - I am getting massive 800ms latency on EU -> US East >
> US West routing over Lumen AS3356.
>
> Wonder if it's just me (my endpoints) or anyone else getting these issues
> as well?
>
>
> mtr from my server in Germany to Lumen in California:
>
>   1. AS51167  ip-1-75-136-213.static.contabo.net
> <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=ipht-%2F%2Fpt%3A651-213-7-is13.atc.t.tcoebonnat&s=78n42c16MVZCatawSzOAbXiuLzg>
>  (213.136.75.1)       1.0%   100    6.3   2.0   0.3  12.5   2.7
>   2. AS???    10.0.50.3                                               1.0%
>   100    6.9   2.4   0.3  17.1   3.0
>   3. AS3356   lag-104-951.ear3.Frankfurt1.Level3.net
> <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=lphtg%2F%2Fat%3A14-1e95.0-n.arfrak3fv1urlleet.ten.3&s=sn-UEMQHVUU9XYarXpFr600Vth8>
>  (62.67.36.137)   0.0%   100    8.8  10.2   8.1  29.9   4.0
>   4. AS3356   ae43.3221.ebr2.frf1.neo.colt.net
> <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht4%2F%2Fet%3Ae23.r1.b32nr2.o1.efften.tloc.&s=PoNcXIFn9faAV4OJRwJzBwaqQpI>
>  (171.75.10.150)        0.0%   100    8.9  10.2   8.3  29.1   3.8
>   5. AS3356   ae3.3.ebr4.frf1.neo.colt.net
> <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht3%2F%2Fet%3A.e.3rr4f.bcnf1lo.o.eten.t&s=SVDSBsti6BgCQHm2LSCIS7M5wxY>
>  (171.75.0.43)              0.0%   100   10.7  10.9   8.3  28.7   4.5
>   6. AS3356   ae49.49.ebr1.dus1.neo.colt.net
> <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht4%2F%2Fet%3Ar99..eb14.o1ducne.s.ten.tlo&s=lLj74I1dI3EAbec-FjCS60M3eaM>
>  (171.75.0.79)            0.0%   100   12.1  13.5  11.7  33.5   3.9
>   7. AS3356   ae10.10.ebr3.dus1.neo.colt.net
> <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht1%2F%2Fet%3Ar00..eb31.o1ducne.s.ten.tlo&s=aeogCMeSk7LOs3CzvlygqcT_D2Q>
>  (171.75.0.29)            0.0%   100   11.7  11.7  11.5  11.9   0.1
>   8. AS3356   ae1.100.ebr4.dus1.neo.colt.net
> <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht1%2F%2Fet%3Ar0.1.eb40.o1ducne.s.ten.tlo&s=YoKtpI5IOo16ZEqIZUKOjdBaCWg>
>  (171.75.0.45)            0.0%   100   11.7  11.7  11.6  12.1   0.1
>   9. AS3356   ae49.49.ebr3.ams1.neo.colt.net
> <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht4%2F%2Fet%3Ar99..eb34.o1amcne.s.ten.tlo&s=SoLhdJw3zMtgWdDnr1UcZsLdRFo>
>  (171.75.0.121)           0.0%   100   14.7  14.6  14.5  14.9   0.1
>  10. AS3356   ae1.100.ebr4.ams1.neo.colt.net
> <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht1%2F%2Fet%3Ar0.1.eb40.o1amcne.s.ten.tlo&s=GDTwRVBe8SplBHqrbPoU3b6-0ho>
>  (171.75.0.15)            0.0%   100   14.7  14.6  14.6  15.0   0.1
>  11. AS3356   ae47.47.ebr4.lon1.neo.colt.net
> <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht4%2F%2Fet%3Ar77..eb44.o1locne.n.ten.tlo&s=FeULy0zOgR7KfT3DPHvgNUUs53E>
>  (171.75.0.54)            0.0%   100   21.3  21.2  21.1  21.4   0.1
>  12. AS3356   ae1.100.ebr3.lon1.neo.colt.net
> <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht1%2F%2Fet%3Ar0.1.eb30.o1locne.n.ten.tlo&s=dFakOg6Eq_qDjEorYrjLELsps8o>
>  (171.75.0.46)            0.0%   100   21.2  21.2  21.1  27.0   0.6
>  13. AS3356   ae48.48.ebr3.lon2.neo.colt.net
> <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht4%2F%2Fet%3Ar88..eb34.o2locne.n.ten.tlo&s=FgBVNcouxNeefID11Pvo4CVCh7I>
>  (171.75.0.59)            0.0%   100   21.3  21.3  21.1  23.5   0.2
>  14. AS3356   ae43.43.ebr1.nyc6.level3.net
> <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht4%2F%2Fet%3Ar33..eb14.v6nylleec.ten.3&s=cg9sqzTnTbYH55OWNzYDnL5s66s>
>  (171.75.16.1)              0.0%   100   90.3  90.3  90.1  90.6   0.1
>  15. AS3356   ae9.9.ebr3.NewYork6.level3.net
> <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht9%2F%2Fet%3A.e.9er3n.blrwyv6.eokten.3le&s=iJ2Hho1oUUIT-OarVl_c5E8pDlw>
>  (4.69.226.222)           0.0%   100   90.7  91.7  90.3 104.3   2.5
>  16. AS3356   ae1.11.ebr4.NewYork6.level3.net
> <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht1%2F%2Fet%3A4..1nbr.1e.oewek6lyrten.3lev&s=AdrzHMpY4tnoB9EYO0QT3ExzOfU>
>  (4.69.209.38)           0.0%   100  105.4  91.7  90.3 105.5   2.6
>  17. AS3356   ae10.10.ebr1.SanJose1.level3.net
> <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht1%2F%2Fet%3Ar00..eb11.1jsalse.noten.3leve&s=yYgFmKz5WsmUhsGu6IBKFW1RJqs>
>  (4.69.160.78)          1.0%   100  804.5 800.9 669.2 871.6  46.6
>  18. AS3356   ae1.0.ebr2.SanJose1.level3.net
> <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht1%2F%2Fet%3A.e.0ar2s.blsnjv1.eoeten.3le&s=zM84PLGELMdmUMx6E7ZUUQtVGoc>
>  (4.69.142.70)            0.0%   100  819.3 802.1 667.4 879.3  47.5
>  19. AS3356   ae2.30.bear2.Sacramento1.level3.net
> <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht2%2F%2Fet%3Ar..3.ea20bnrsaometca3e1.nel.lvte&s=gzqPYSqc17FtpgaKe5VbgiNNmSY>
>  (4.69.141.77)      11.0%   100  825.1 801.5 668.0 865.0  45.1
>  20. AS3356   system.0.bear1.Sacramento1.level3.net
> <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=sphts%2F%2Fyt%3Ae.ter.bam0ma1.nraesce.to3evl1lten.&s=HgYEJnrRdawfz5Wi45acR276RGw>
>  (4.69.183.132)    0.0%   100  821.3 800.5 662.7 865.9  47.1
>
>
>
>
> Thanks.
>
>
>
> --
> Anurag Bhatia
> anuragbhatia.com
> <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=%2Fphtn%3A%2Fatstgurahaiabmoc.&s=S-51H0xHJ-Z8oewcounpVrWuXp0>
>
>
>
> --
> Anurag Bhatia
> anuragbhatia.com
> <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=%2Fphtn%3A%2Fatstgurahaiabmoc.&s=S-51H0xHJ-Z8oewcounpVrWuXp0>
> ------------------------------
>
> This email has been scanned for spam & viruses. If you believe this email
> should have been stopped by our filters, click here
> <https://portal.mailanyone.net/index.html#/outer/reportspam?token=dXNlcj1kb3J0aXpAenRoZXJuZXQuY29tO3RzPTE3MjA1MTc0NTM7dXVpZD02NjhEMDM0NkJDNDY3MDY0QzBERTkxRjUzNzU3RjY2RTt0b2tlbj04MjAwYzBmMTJmZTU4YzY5YzkzNDZlZDIxODU4YmI3NDVlYjEzMWExOw%3D%3D>
>  to report it.
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
>

-- 
Anurag Bhatia
anuragbhatia.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20240709/b9f854ac/attachment-0001.htm>


More information about the Outages mailing list