[outages] Lumen issues between East Coast and West Coast?
Tim Burke
tim at mid.net
Tue Jul 9 13:04:02 EDT 2024
Latest update from Lumen outage case - they appear to be turning gear up now:
2024-07-09 16:41:39 GMT - Field personnel have successfully connected additional air handlers at the site, resulting in a steady decrease in temperature. As conditions return to normal, technicians have commenced the process of powering up the equipment. An additional generator is being dispatched from Tulsa and is anticipated to arrive at the site around 21:00 GMT. Upon its arrival, technicians will initiate the integration of the generator to supply extra voltage to the site.
On Jul 9, 2024, at 9:23 AM, Tim Burke via Outages <outages at outages.org> wrote:
One of my Lumen transit circuits in Houston came back up at 2:30am but seeing triple-digit latency across it.
Another Lumen circuit (same facility in Houston) off of another router still remains hard down.
On Jul 9, 2024, at 9:14 AM, Dustin Ortiz via Outages <outages at outages.org> wrote:
From lumen regarding the Generator outages at the Houston Gateway site:
2024-07-09 12:58:07 GMT - Upon evaluation, technicians have identified that the chillers are not effectively cooling the equipment as intended. Efforts are underway to diagnose and resolve the issue. In support of these efforts, Lumen Management has reached out to the equipment vendors for additional assistance as technicians continue to work towards restoring the nodes.
2024-07-09 11:36:09 GMT - The chillers have begun to cool the equipment and Field Operations continues in efforts to begin powering up priority network elements.
2024-07-09 10:20:14 GMT - The 2.5 Mega Watt generator has been brought online and field teams are diligently working to activate the chillers, which will facilitate the cooling of our equipment. As the cooling process commences, technicians are sequentially restoring critical core equipment. Additionally, we have procured several replacement cards in anticipation of potential equipment damage caused by elevated temperatures
2024-07-09 08:57:26 GMT - Crews have advised that a 2-megawatt generator has arrived at the gateway site in addition to the initial generator - rated at 25-kilowatts, and wiring efforts are preparing to commence at this time. Teams have advised that wiring is underway and anticipated to be completed on the 2-megawatt generator in approximately 1 hour, and the completion of wiring on the 25-kilowatt generator will follow. Once the installation of the 2-megawatt generator is completed, crews have advised that the expanded load capacity will be used to power 2 chillers which will be used to begin cooling site temperatures and power priority equipment as additional installation efforts are completed. Additionally, an existing generator onsite has been recovered, and DC power has been stabilized. The next update will be provided upon the installation of the 2-megawatt generator, or as new information becomes available.
2024-07-09 08:40:13 GMT - The first generator has arrived onsite and Lumen teams have advised that preparations are underway to commence with the wiring process to install the generator. While the single generator is not anticipated to power the site singlehandedly, it is anticipated that the addition will power some equipment necessary to cool the site and assist with balancing the power load. Lumen teams are closely monitoring conditions and completing assessments on a regular basis. The next update will be provided upon the completed wiring of the first generator, or as new information becomes available.
2024-07-09 07:41:25 GMT - Investigations by the Critical Infrastructure Technical Support (CITS) Team revealed the two on-site generators at a Lumen Gateway Facility in Houston, TX were experiencing power loading issues. As a result, the site was running on one on-site generator for most of the day and the generator was able to support the load. At approximately 01:00 GMT, the second on-site generator began experiencing issues as well. The generators will turn on and begin to run which begins to power the three on-site Direct Current (DC) power plants; however, after about 30 minutes, the generator is unable to regulate itself and then shuts off. As a result of this, teams were having to toggle between charging the DC plant via the generators while running, and cooling the site via the cooling system that is powered by the on-site DC batteries. At 02:24 GMT, teams began to gracefully shut down shelves within an Infinera ring to help conserve power to the affected site and to protect any equipment that may be bouncing as a result of the generator failures. Other equipment at the site is being evaluated as the outage progresses.
There are currently two replacement generators en route to the site. One generator is coming from Dallas, TX and is expected to be on site and wired up within 2 hours. The second generator is en route from Midland, TX and should be arriving on site within the next 6-12 hours. Both generators are needed to power the site due to the size of the gateway site. The next update will be provided by 16:00 GMT or as new information becomes available.
2024-07-09 03:53:10 GMT - On July 09, 2024, at 03:40 GMT, Lumen was alerted to service impact in Houston, TX caused by severe area weather conditions resulting from Hurricane Beryl. As this network fault is impacting multiple clients, the event has increased visibility with Lumen leadership; however, as local authorities have issued a state of emergency requiring civilian parties to shelter in place, Lumen has postponed further action until the order has been lifted to ensure the safety of personnel. In the interim, Lumen is actively monitoring Hurricane Beryl's path and will remain in close contact with local, state, and federal offices of emergency management to provide close coordination and timely information. Each outage situation will be evaluated, and a determination made if a remote restoration action is viable or if on-site support is necessary. For the latter, and if evacuations occur, Field Operations personnel will dispatch once government authorization is received. The safety of our employees is of utmost concern; we are taking every step to ensure their safety, which may delay our network repair response times. Updates will continue to be provided every 4 hours or as new information is received.
Dustin Ortiz | VP Systems Engineering
214-945-2201 | dortiz at zthernet.com<mailto:dortiz at zthernet.com>
From: Outages <outages-bounces at outages.org<mailto:outages-bounces at outages.org>> On Behalf Of Anurag Bhatia via Outages
Sent: Tuesday, July 9, 2024 4:29 AM
To: Outage Mailing list <outages at outages.org<mailto:outages at outages.org>>
Subject: Re: [outages] Lumen issues between East Coast and West Coast?
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
(Seems like my last message is on hold for moderation)
I see the issue is resolved just now. It was there from 1:30 am to 9:16 am UTC.
On Tue, Jul 9, 2024 at 2:11 PM Anurag Bhatia <me at anuragbhatia.com<mailto:me at anuragbhatia.com>> wrote:
Since 1:30am UTC - I am getting massive 800ms latency on EU -> US East > US West routing over Lumen AS3356.
Wonder if it's just me (my endpoints) or anyone else getting these issues as well?
mtr from my server in Germany to Lumen in California:
1. AS51167 ip-1-75-136-213.static.contabo.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=ipht-%2F%2Fpt%3A651-213-7-is13.atc.t.tcoebonnat&s=78n42c16MVZCatawSzOAbXiuLzg> (213.136.75.1) 1.0% 100 6.3 2.0 0.3 12.5 2.7
2. AS??? 10.0.50.3 1.0% 100 6.9 2.4 0.3 17.1 3.0
3. AS3356 lag-104-951.ear3.Frankfurt1.Level3.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=lphtg%2F%2Fat%3A14-1e95.0-n.arfrak3fv1urlleet.ten.3&s=sn-UEMQHVUU9XYarXpFr600Vth8> (62.67.36.137) 0.0% 100 8.8 10.2 8.1 29.9 4.0
4. AS3356 ae43.3221.ebr2.frf1.neo.colt.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht4%2F%2Fet%3Ae23.r1.b32nr2.o1.efften.tloc.&s=PoNcXIFn9faAV4OJRwJzBwaqQpI> (171.75.10.150) 0.0% 100 8.9 10.2 8.3 29.1 3.8
5. AS3356 ae3.3.ebr4.frf1.neo.colt.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht3%2F%2Fet%3A.e.3rr4f.bcnf1lo.o.eten.t&s=SVDSBsti6BgCQHm2LSCIS7M5wxY> (171.75.0.43) 0.0% 100 10.7 10.9 8.3 28.7 4.5
6. AS3356 ae49.49.ebr1.dus1.neo.colt.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht4%2F%2Fet%3Ar99..eb14.o1ducne.s.ten.tlo&s=lLj74I1dI3EAbec-FjCS60M3eaM> (171.75.0.79) 0.0% 100 12.1 13.5 11.7 33.5 3.9
7. AS3356 ae10.10.ebr3.dus1.neo.colt.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht1%2F%2Fet%3Ar00..eb31.o1ducne.s.ten.tlo&s=aeogCMeSk7LOs3CzvlygqcT_D2Q> (171.75.0.29) 0.0% 100 11.7 11.7 11.5 11.9 0.1
8. AS3356 ae1.100.ebr4.dus1.neo.colt.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht1%2F%2Fet%3Ar0.1.eb40.o1ducne.s.ten.tlo&s=YoKtpI5IOo16ZEqIZUKOjdBaCWg> (171.75.0.45) 0.0% 100 11.7 11.7 11.6 12.1 0.1
9. AS3356 ae49.49.ebr3.ams1.neo.colt.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht4%2F%2Fet%3Ar99..eb34.o1amcne.s.ten.tlo&s=SoLhdJw3zMtgWdDnr1UcZsLdRFo> (171.75.0.121) 0.0% 100 14.7 14.6 14.5 14.9 0.1
10. AS3356 ae1.100.ebr4.ams1.neo.colt.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht1%2F%2Fet%3Ar0.1.eb40.o1amcne.s.ten.tlo&s=GDTwRVBe8SplBHqrbPoU3b6-0ho> (171.75.0.15) 0.0% 100 14.7 14.6 14.6 15.0 0.1
11. AS3356 ae47.47.ebr4.lon1.neo.colt.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht4%2F%2Fet%3Ar77..eb44.o1locne.n.ten.tlo&s=FeULy0zOgR7KfT3DPHvgNUUs53E> (171.75.0.54) 0.0% 100 21.3 21.2 21.1 21.4 0.1
12. AS3356 ae1.100.ebr3.lon1.neo.colt.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht1%2F%2Fet%3Ar0.1.eb30.o1locne.n.ten.tlo&s=dFakOg6Eq_qDjEorYrjLELsps8o> (171.75.0.46) 0.0% 100 21.2 21.2 21.1 27.0 0.6
13. AS3356 ae48.48.ebr3.lon2.neo.colt.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht4%2F%2Fet%3Ar88..eb34.o2locne.n.ten.tlo&s=FgBVNcouxNeefID11Pvo4CVCh7I> (171.75.0.59) 0.0% 100 21.3 21.3 21.1 23.5 0.2
14. AS3356 ae43.43.ebr1.nyc6.level3.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht4%2F%2Fet%3Ar33..eb14.v6nylleec.ten.3&s=cg9sqzTnTbYH55OWNzYDnL5s66s> (171.75.16.1) 0.0% 100 90.3 90.3 90.1 90.6 0.1
15. AS3356 ae9.9.ebr3.NewYork6.level3.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht9%2F%2Fet%3A.e.9er3n.blrwyv6.eokten.3le&s=iJ2Hho1oUUIT-OarVl_c5E8pDlw> (4.69.226.222) 0.0% 100 90.7 91.7 90.3 104.3 2.5
16. AS3356 ae1.11.ebr4.NewYork6.level3.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht1%2F%2Fet%3A4..1nbr.1e.oewek6lyrten.3lev&s=AdrzHMpY4tnoB9EYO0QT3ExzOfU> (4.69.209.38) 0.0% 100 105.4 91.7 90.3 105.5 2.6
17. AS3356 ae10.10.ebr1.SanJose1.level3.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht1%2F%2Fet%3Ar00..eb11.1jsalse.noten.3leve&s=yYgFmKz5WsmUhsGu6IBKFW1RJqs> (4.69.160.78) 1.0% 100 804.5 800.9 669.2 871.6 46.6
18. AS3356 ae1.0.ebr2.SanJose1.level3.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht1%2F%2Fet%3A.e.0ar2s.blsnjv1.eoeten.3le&s=zM84PLGELMdmUMx6E7ZUUQtVGoc> (4.69.142.70) 0.0% 100 819.3 802.1 667.4 879.3 47.5
19. AS3356 ae2.30.bear2.Sacramento1.level3.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht2%2F%2Fet%3Ar..3.ea20bnrsaometca3e1.nel.lvte&s=gzqPYSqc17FtpgaKe5VbgiNNmSY> (4.69.141.77) 11.0% 100 825.1 801.5 668.0 865.0 45.1
20. AS3356 system.0.bear1.Sacramento1.level3.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=sphts%2F%2Fyt%3Ae.ter.bam0ma1.nraesce.to3evl1lten.&s=HgYEJnrRdawfz5Wi45acR276RGw> (4.69.183.132) 0.0% 100 821.3 800.5 662.7 865.9 47.1
Thanks.
--
Anurag Bhatia
anuragbhatia.com<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=%2Fphtn%3A%2Fatstgurahaiabmoc.&s=S-51H0xHJ-Z8oewcounpVrWuXp0>
--
Anurag Bhatia
anuragbhatia.com<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=%2Fphtn%3A%2Fatstgurahaiabmoc.&s=S-51H0xHJ-Z8oewcounpVrWuXp0>
________________________________
This email has been scanned for spam & viruses. If you believe this email should have been stopped by our filters, click here<https://portal.mailanyone.net/index.html#/outer/reportspam?token=dXNlcj1kb3J0aXpAenRoZXJuZXQuY29tO3RzPTE3MjA1MTc0NTM7dXVpZD02NjhEMDM0NkJDNDY3MDY0QzBERTkxRjUzNzU3RjY2RTt0b2tlbj04MjAwYzBmMTJmZTU4YzY5YzkzNDZlZDIxODU4YmI3NDVlYjEzMWExOw%3D%3D> to report it.
_______________________________________________
Outages mailing list
Outages at outages.org<mailto:Outages at outages.org>
https://puck.nether.net/mailman/listinfo/outages
_______________________________________________
Outages mailing list
Outages at outages.org
https://puck.nether.net/mailman/listinfo/outages
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20240709/f8c153da/attachment-0001.htm>
More information about the Outages
mailing list