[outages] Lumen issues between East Coast and West Coast?

Tue Jul 9 10:14:53 EDT 2024

From lumen regarding the Generator outages at the Houston Gateway site:

2024-07-09 12:58:07 GMT - Upon evaluation, technicians have identified that the chillers are not effectively cooling the equipment as intended. Efforts are underway to diagnose and resolve the issue. In support of these efforts, Lumen Management has reached out to the equipment vendors for additional assistance as technicians continue to work towards restoring the nodes.

2024-07-09 11:36:09 GMT - The chillers have begun to cool the equipment and Field Operations continues in efforts to begin powering up priority network elements.

2024-07-09 10:20:14 GMT - The 2.5 Mega Watt generator has been brought online and field teams are diligently working to activate the chillers, which will facilitate the cooling of our equipment. As the cooling process commences, technicians are sequentially restoring critical core equipment. Additionally, we have procured several replacement cards in anticipation of potential equipment damage caused by elevated temperatures

2024-07-09 08:57:26 GMT - Crews have advised that a 2-megawatt generator has arrived at the gateway site in addition to the initial generator - rated at 25-kilowatts, and wiring efforts are preparing to commence at this time. Teams have advised that wiring is underway and anticipated to be completed on the 2-megawatt generator in approximately 1 hour, and the completion of wiring on the 25-kilowatt generator will follow. Once the installation of the 2-megawatt generator is completed, crews have advised that the expanded load capacity will be used to power 2 chillers which will be used to begin cooling site temperatures and power priority equipment as additional installation efforts are completed. Additionally, an existing generator onsite has been recovered, and DC power has been stabilized. The next update will be provided upon the installation of the 2-megawatt generator, or as new information becomes available.

2024-07-09 08:40:13 GMT - The first generator has arrived onsite and Lumen teams have advised that preparations are underway to commence with the wiring process to install the generator. While the single generator is not anticipated to power the site singlehandedly, it is anticipated that the addition will power some equipment necessary to cool the site and assist with balancing the power load. Lumen teams are closely monitoring conditions and completing assessments on a regular basis. The next update will be provided upon the completed wiring of the first generator, or as new information becomes available.

2024-07-09 07:41:25 GMT - Investigations by the Critical Infrastructure Technical Support (CITS) Team revealed the two on-site generators at a Lumen Gateway Facility in Houston, TX were experiencing power loading issues. As a result, the site was running on one on-site generator for most of the day and the generator was able to support the load. At approximately 01:00 GMT, the second on-site generator began experiencing issues as well. The generators will turn on and begin to run which begins to power the three on-site Direct Current (DC) power plants; however, after about 30 minutes, the generator is unable to regulate itself and then shuts off. As a result of this, teams were having to toggle between charging the DC plant via the generators while running, and cooling the site via the cooling system that is powered by the on-site DC batteries. At 02:24 GMT, teams began to gracefully shut down shelves within an Infinera ring to help conserve power to the affected site and to protect any equipment that may be bouncing as a result of the generator failures. Other equipment at the site is being evaluated as the outage progresses.

There are currently two replacement generators en route to the site. One generator is coming from Dallas, TX and is expected to be on site and wired up within 2 hours. The second generator is en route from Midland, TX and should be arriving on site within the next 6-12 hours. Both generators are needed to power the site due to the size of the gateway site. The next update will be provided by 16:00 GMT or as new information becomes available.

2024-07-09 03:53:10 GMT - On July 09, 2024, at 03:40 GMT, Lumen was alerted to service impact in Houston, TX caused by severe area weather conditions resulting from Hurricane Beryl. As this network fault is impacting multiple clients, the event has increased visibility with Lumen leadership; however, as local authorities have issued a state of emergency requiring civilian parties to shelter in place, Lumen has postponed further action until the order has been lifted to ensure the safety of personnel. In the interim, Lumen is actively monitoring Hurricane Beryl's path and will remain in close contact with local, state, and federal offices of emergency management to provide close coordination and timely information. Each outage situation will be evaluated, and a determination made if a remote restoration action is viable or if on-site support is necessary. For the latter, and if evacuations occur, Field Operations personnel will dispatch once government authorization is received. The safety of our employees is of utmost concern; we are taking every step to ensure their safety, which may delay our network repair response times. Updates will continue to be provided every 4 hours or as new information is received.

Dustin Ortiz | VP Systems Engineering
214-945-2201 | dortiz at zthernet.com<mailto:dortiz at zthernet.com>

From: Outages <outages-bounces at outages.org> On Behalf Of Anurag Bhatia via Outages
Sent: Tuesday, July 9, 2024 4:29 AM
To: Outage Mailing list <outages at outages.org>
Subject: Re: [outages] Lumen issues between East Coast and West Coast?

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

(Seems like my last message is on hold for moderation)

I see the issue is resolved just now. It was there from 1:30 am to 9:16 am UTC.

On Tue, Jul 9, 2024 at 2:11 PM Anurag Bhatia <me at anuragbhatia.com<mailto:me at anuragbhatia.com>> wrote:
Since 1:30am UTC - I am getting massive 800ms latency on EU -> US East > US West routing over Lumen AS3356.

Wonder if it's just me (my endpoints) or anyone else getting these issues as well?

mtr from my server in Germany to Lumen in California:

  1. AS51167  ip-1-75-136-213.static.contabo.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=ipht-%2F%2Fpt%3A651-213-7-is13.atc.t.tcoebonnat&s=78n42c16MVZCatawSzOAbXiuLzg> (213.136.75.1)       1.0%   100    6.3   2.0   0.3  12.5   2.7
  2. AS???    10.0.50.3                                               1.0%   100    6.9   2.4   0.3  17.1   3.0
  3. AS3356   lag-104-951.ear3.Frankfurt1.Level3.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=lphtg%2F%2Fat%3A14-1e95.0-n.arfrak3fv1urlleet.ten.3&s=sn-UEMQHVUU9XYarXpFr600Vth8> (62.67.36.137)   0.0%   100    8.8  10.2   8.1  29.9   4.0
  4. AS3356   ae43.3221.ebr2.frf1.neo.colt.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht4%2F%2Fet%3Ae23.r1.b32nr2.o1.efften.tloc.&s=PoNcXIFn9faAV4OJRwJzBwaqQpI> (171.75.10.150)        0.0%   100    8.9  10.2   8.3  29.1   3.8
  5. AS3356   ae3.3.ebr4.frf1.neo.colt.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht3%2F%2Fet%3A.e.3rr4f.bcnf1lo.o.eten.t&s=SVDSBsti6BgCQHm2LSCIS7M5wxY> (171.75.0.43)              0.0%   100   10.7  10.9   8.3  28.7   4.5
  6. AS3356   ae49.49.ebr1.dus1.neo.colt.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht4%2F%2Fet%3Ar99..eb14.o1ducne.s.ten.tlo&s=lLj74I1dI3EAbec-FjCS60M3eaM> (171.75.0.79)            0.0%   100   12.1  13.5  11.7  33.5   3.9
  7. AS3356   ae10.10.ebr3.dus1.neo.colt.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht1%2F%2Fet%3Ar00..eb31.o1ducne.s.ten.tlo&s=aeogCMeSk7LOs3CzvlygqcT_D2Q> (171.75.0.29)            0.0%   100   11.7  11.7  11.5  11.9   0.1
  8. AS3356   ae1.100.ebr4.dus1.neo.colt.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht1%2F%2Fet%3Ar0.1.eb40.o1ducne.s.ten.tlo&s=YoKtpI5IOo16ZEqIZUKOjdBaCWg> (171.75.0.45)            0.0%   100   11.7  11.7  11.6  12.1   0.1
  9. AS3356   ae49.49.ebr3.ams1.neo.colt.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht4%2F%2Fet%3Ar99..eb34.o1amcne.s.ten.tlo&s=SoLhdJw3zMtgWdDnr1UcZsLdRFo> (171.75.0.121)           0.0%   100   14.7  14.6  14.5  14.9   0.1
 10. AS3356   ae1.100.ebr4.ams1.neo.colt.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht1%2F%2Fet%3Ar0.1.eb40.o1amcne.s.ten.tlo&s=GDTwRVBe8SplBHqrbPoU3b6-0ho> (171.75.0.15)            0.0%   100   14.7  14.6  14.6  15.0   0.1
 11. AS3356   ae47.47.ebr4.lon1.neo.colt.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht4%2F%2Fet%3Ar77..eb44.o1locne.n.ten.tlo&s=FeULy0zOgR7KfT3DPHvgNUUs53E> (171.75.0.54)            0.0%   100   21.3  21.2  21.1  21.4   0.1
 12. AS3356   ae1.100.ebr3.lon1.neo.colt.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht1%2F%2Fet%3Ar0.1.eb30.o1locne.n.ten.tlo&s=dFakOg6Eq_qDjEorYrjLELsps8o> (171.75.0.46)            0.0%   100   21.2  21.2  21.1  27.0   0.6
 13. AS3356   ae48.48.ebr3.lon2.neo.colt.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht4%2F%2Fet%3Ar88..eb34.o2locne.n.ten.tlo&s=FgBVNcouxNeefID11Pvo4CVCh7I> (171.75.0.59)            0.0%   100   21.3  21.3  21.1  23.5   0.2
 14. AS3356   ae43.43.ebr1.nyc6.level3.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht4%2F%2Fet%3Ar33..eb14.v6nylleec.ten.3&s=cg9sqzTnTbYH55OWNzYDnL5s66s> (171.75.16.1)              0.0%   100   90.3  90.3  90.1  90.6   0.1
 15. AS3356   ae9.9.ebr3.NewYork6.level3.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht9%2F%2Fet%3A.e.9er3n.blrwyv6.eokten.3le&s=iJ2Hho1oUUIT-OarVl_c5E8pDlw> (4.69.226.222)           0.0%   100   90.7  91.7  90.3 104.3   2.5
 16. AS3356   ae1.11.ebr4.NewYork6.level3.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht1%2F%2Fet%3A4..1nbr.1e.oewek6lyrten.3lev&s=AdrzHMpY4tnoB9EYO0QT3ExzOfU> (4.69.209.38)           0.0%   100  105.4  91.7  90.3 105.5   2.6
 17. AS3356   ae10.10.ebr1.SanJose1.level3.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht1%2F%2Fet%3Ar00..eb11.1jsalse.noten.3leve&s=yYgFmKz5WsmUhsGu6IBKFW1RJqs> (4.69.160.78)          1.0%   100  804.5 800.9 669.2 871.6  46.6
 18. AS3356   ae1.0.ebr2.SanJose1.level3.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht1%2F%2Fet%3A.e.0ar2s.blsnjv1.eoeten.3le&s=zM84PLGELMdmUMx6E7ZUUQtVGoc> (4.69.142.70)            0.0%   100  819.3 802.1 667.4 879.3  47.5
 19. AS3356   ae2.30.bear2.Sacramento1.level3.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht2%2F%2Fet%3Ar..3.ea20bnrsaometca3e1.nel.lvte&s=gzqPYSqc17FtpgaKe5VbgiNNmSY> (4.69.141.77)      11.0%   100  825.1 801.5 668.0 865.0  45.1
 20. AS3356   system.0.bear1.Sacramento1.level3.net<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=sphts%2F%2Fyt%3Ae.ter.bam0ma1.nraesce.to3evl1lten.&s=HgYEJnrRdawfz5Wi45acR276RGw> (4.69.183.132)    0.0%   100  821.3 800.5 662.7 865.9  47.1

Thanks.

--
Anurag Bhatia
anuragbhatia.com<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=%2Fphtn%3A%2Fatstgurahaiabmoc.&s=S-51H0xHJ-Z8oewcounpVrWuXp0>

--
Anurag Bhatia
anuragbhatia.com<https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=%2Fphtn%3A%2Fatstgurahaiabmoc.&s=S-51H0xHJ-Z8oewcounpVrWuXp0>
________________________________

This email has been scanned for spam & viruses. If you believe this email should have been stopped by our filters, click here<https://portal.mailanyone.net/index.html#/outer/reportspam?token=dXNlcj1kb3J0aXpAenRoZXJuZXQuY29tO3RzPTE3MjA1MTc0NTM7dXVpZD02NjhEMDM0NkJDNDY3MDY0QzBERTkxRjUzNzU3RjY2RTt0b2tlbj04MjAwYzBmMTJmZTU4YzY5YzkzNDZlZDIxODU4YmI3NDVlYjEzMWExOw%3D%3D> to report it.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20240709/a8a42c4e/attachment-0001.htm>