[outages] Lumen issues between East Coast and West Coast?
Automattic NOC
noc at automattic.com
Tue Jul 9 10:26:33 EDT 2024
We observed roughly 50% of Lumen paths going east<>west in the USA
encountering substantial congestion on the order of Anurag's MTR results.
The problem has come and gone a couple of times. Given that 50% of
their paths looked totally un-congested, there was probably a lot of room
for traffic engineering to reduce the impact here.
See: https://imgur.com/a/IBjvecN
(36 monitoring flows from NYC<>LA. Green lines are 50th percentile RTT
and little multicoloured lines on the bottom are loss. All times UTC).
Cheers,
Tyler
Automattic Network Operations
On Tue, Jul 9, 2024 at 2:16 PM Dustin Ortiz via Outages <outages at outages.org>
wrote:
> From lumen regarding the Generator outages at the Houston Gateway site:
>
>
>
> 2024-07-09 12:58:07 GMT - Upon evaluation, technicians have identified
> that the chillers are not effectively cooling the equipment as intended.
> Efforts are underway to diagnose and resolve the issue. In support of these
> efforts, Lumen Management has reached out to the equipment vendors for
> additional assistance as technicians continue to work towards restoring the
> nodes.
>
> 2024-07-09 11:36:09 GMT - The chillers have begun to cool the equipment
> and Field Operations continues in efforts to begin powering up priority
> network elements.
>
> 2024-07-09 10:20:14 GMT - The 2.5 Mega Watt generator has been brought
> online and field teams are diligently working to activate the chillers,
> which will facilitate the cooling of our equipment. As the cooling process
> commences, technicians are sequentially restoring critical core equipment.
> Additionally, we have procured several replacement cards in anticipation of
> potential equipment damage caused by elevated temperatures
>
> 2024-07-09 08:57:26 GMT - Crews have advised that a 2-megawatt generator
> has arrived at the gateway site in addition to the initial generator -
> rated at 25-kilowatts, and wiring efforts are preparing to commence at this
> time. Teams have advised that wiring is underway and anticipated to be
> completed on the 2-megawatt generator in approximately 1 hour, and the
> completion of wiring on the 25-kilowatt generator will follow. Once the
> installation of the 2-megawatt generator is completed, crews have advised
> that the expanded load capacity will be used to power 2 chillers which will
> be used to begin cooling site temperatures and power priority equipment as
> additional installation efforts are completed. Additionally, an existing
> generator onsite has been recovered, and DC power has been stabilized. The
> next update will be provided upon the installation of the 2-megawatt
> generator, or as new information becomes available.
>
> 2024-07-09 08:40:13 GMT - The first generator has arrived onsite and Lumen
> teams have advised that preparations are underway to commence with the
> wiring process to install the generator. While the single generator is not
> anticipated to power the site singlehandedly, it is anticipated that the
> addition will power some equipment necessary to cool the site and assist
> with balancing the power load. Lumen teams are closely monitoring
> conditions and completing assessments on a regular basis. The next update
> will be provided upon the completed wiring of the first generator, or as
> new information becomes available.
>
> 2024-07-09 07:41:25 GMT - Investigations by the Critical Infrastructure
> Technical Support (CITS) Team revealed the two on-site generators at a
> Lumen Gateway Facility in Houston, TX were experiencing power loading
> issues. As a result, the site was running on one on-site generator for most
> of the day and the generator was able to support the load. At approximately
> 01:00 GMT, the second on-site generator began experiencing issues as well.
> The generators will turn on and begin to run which begins to power the
> three on-site Direct Current (DC) power plants; however, after about 30
> minutes, the generator is unable to regulate itself and then shuts off. As
> a result of this, teams were having to toggle between charging the DC plant
> via the generators while running, and cooling the site via the cooling
> system that is powered by the on-site DC batteries. At 02:24 GMT, teams
> began to gracefully shut down shelves within an Infinera ring to help
> conserve power to the affected site and to protect any equipment that may
> be bouncing as a result of the generator failures. Other equipment at the
> site is being evaluated as the outage progresses.
>
> There are currently two replacement generators en route to the site. One
> generator is coming from Dallas, TX and is expected to be on site and wired
> up within 2 hours. The second generator is en route from Midland, TX and
> should be arriving on site within the next 6-12 hours. Both generators are
> needed to power the site due to the size of the gateway site. The next
> update will be provided by 16:00 GMT or as new information becomes
> available.
>
> 2024-07-09 03:53:10 GMT - On July 09, 2024, at 03:40 GMT, Lumen was
> alerted to service impact in Houston, TX caused by severe area weather
> conditions resulting from Hurricane Beryl. As this network fault is
> impacting multiple clients, the event has increased visibility with Lumen
> leadership; however, as local authorities have issued a state of emergency
> requiring civilian parties to shelter in place, Lumen has postponed further
> action until the order has been lifted to ensure the safety of personnel.
> In the interim, Lumen is actively monitoring Hurricane Beryl's path and
> will remain in close contact with local, state, and federal offices of
> emergency management to provide close coordination and timely information.
> Each outage situation will be evaluated, and a determination made if a
> remote restoration action is viable or if on-site support is necessary. For
> the latter, and if evacuations occur, Field Operations personnel will
> dispatch once government authorization is received. The safety of our
> employees is of utmost concern; we are taking every step to ensure their
> safety, which may delay our network repair response times. Updates will
> continue to be provided every 4 hours or as new information is received.
>
>
>
>
>
>
>
> *Dustin Ortiz | *VP Systems Engineering
> *214-945-2201 | dortiz at zthernet.com <dortiz at zthernet.com>*
>
>
>
> *From:* Outages <outages-bounces at outages.org> *On Behalf Of *Anurag
> Bhatia via Outages
> *Sent:* Tuesday, July 9, 2024 4:29 AM
> *To:* Outage Mailing list <outages at outages.org>
> *Subject:* Re: [outages] Lumen issues between East Coast and West Coast?
>
>
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
>
>
>
> (Seems like my last message is on hold for moderation)
>
> I see the issue is resolved just now. It was there from 1:30 am to 9:16 am
> UTC.
>
>
>
>
>
>
>
>
>
> On Tue, Jul 9, 2024 at 2:11 PM Anurag Bhatia <me at anuragbhatia.com> wrote:
>
> Since 1:30am UTC - I am getting massive 800ms latency on EU -> US East >
> US West routing over Lumen AS3356.
>
>
>
> Wonder if it's just me (my endpoints) or anyone else getting these issues
> as well?
>
>
>
>
> mtr from my server in Germany to Lumen in California:
>
>
>
> 1. AS51167 ip-1-75-136-213.static.contabo.net
> <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=ipht-%2F%2Fpt%3A651-213-7-is13.atc.t.tcoebonnat&s=78n42c16MVZCatawSzOAbXiuLzg>
> (213.136.75.1) 1.0% 100 6.3 2.0 0.3 12.5 2.7
> 2. AS??? 10.0.50.3 1.0%
> 100 6.9 2.4 0.3 17.1 3.0
> 3. AS3356 lag-104-951.ear3.Frankfurt1.Level3.net
> <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=lphtg%2F%2Fat%3A14-1e95.0-n.arfrak3fv1urlleet.ten.3&s=sn-UEMQHVUU9XYarXpFr600Vth8>
> (62.67.36.137) 0.0% 100 8.8 10.2 8.1 29.9 4.0
> 4. AS3356 ae43.3221.ebr2.frf1.neo.colt.net
> <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht4%2F%2Fet%3Ae23.r1.b32nr2.o1.efften.tloc.&s=PoNcXIFn9faAV4OJRwJzBwaqQpI>
> (171.75.10.150) 0.0% 100 8.9 10.2 8.3 29.1 3.8
> 5. AS3356 ae3.3.ebr4.frf1.neo.colt.net
> <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht3%2F%2Fet%3A.e.3rr4f.bcnf1lo.o.eten.t&s=SVDSBsti6BgCQHm2LSCIS7M5wxY>
> (171.75.0.43) 0.0% 100 10.7 10.9 8.3 28.7 4.5
> 6. AS3356 ae49.49.ebr1.dus1.neo.colt.net
> <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht4%2F%2Fet%3Ar99..eb14.o1ducne.s.ten.tlo&s=lLj74I1dI3EAbec-FjCS60M3eaM>
> (171.75.0.79) 0.0% 100 12.1 13.5 11.7 33.5 3.9
> 7. AS3356 ae10.10.ebr3.dus1.neo.colt.net
> <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht1%2F%2Fet%3Ar00..eb31.o1ducne.s.ten.tlo&s=aeogCMeSk7LOs3CzvlygqcT_D2Q>
> (171.75.0.29) 0.0% 100 11.7 11.7 11.5 11.9 0.1
> 8. AS3356 ae1.100.ebr4.dus1.neo.colt.net
> <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht1%2F%2Fet%3Ar0.1.eb40.o1ducne.s.ten.tlo&s=YoKtpI5IOo16ZEqIZUKOjdBaCWg>
> (171.75.0.45) 0.0% 100 11.7 11.7 11.6 12.1 0.1
> 9. AS3356 ae49.49.ebr3.ams1.neo.colt.net
> <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht4%2F%2Fet%3Ar99..eb34.o1amcne.s.ten.tlo&s=SoLhdJw3zMtgWdDnr1UcZsLdRFo>
> (171.75.0.121) 0.0% 100 14.7 14.6 14.5 14.9 0.1
> 10. AS3356 ae1.100.ebr4.ams1.neo.colt.net
> <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht1%2F%2Fet%3Ar0.1.eb40.o1amcne.s.ten.tlo&s=GDTwRVBe8SplBHqrbPoU3b6-0ho>
> (171.75.0.15) 0.0% 100 14.7 14.6 14.6 15.0 0.1
> 11. AS3356 ae47.47.ebr4.lon1.neo.colt.net
> <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht4%2F%2Fet%3Ar77..eb44.o1locne.n.ten.tlo&s=FeULy0zOgR7KfT3DPHvgNUUs53E>
> (171.75.0.54) 0.0% 100 21.3 21.2 21.1 21.4 0.1
> 12. AS3356 ae1.100.ebr3.lon1.neo.colt.net
> <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht1%2F%2Fet%3Ar0.1.eb30.o1locne.n.ten.tlo&s=dFakOg6Eq_qDjEorYrjLELsps8o>
> (171.75.0.46) 0.0% 100 21.2 21.2 21.1 27.0 0.6
> 13. AS3356 ae48.48.ebr3.lon2.neo.colt.net
> <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht4%2F%2Fet%3Ar88..eb34.o2locne.n.ten.tlo&s=FgBVNcouxNeefID11Pvo4CVCh7I>
> (171.75.0.59) 0.0% 100 21.3 21.3 21.1 23.5 0.2
> 14. AS3356 ae43.43.ebr1.nyc6.level3.net
> <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht4%2F%2Fet%3Ar33..eb14.v6nylleec.ten.3&s=cg9sqzTnTbYH55OWNzYDnL5s66s>
> (171.75.16.1) 0.0% 100 90.3 90.3 90.1 90.6 0.1
> 15. AS3356 ae9.9.ebr3.NewYork6.level3.net
> <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht9%2F%2Fet%3A.e.9er3n.blrwyv6.eokten.3le&s=iJ2Hho1oUUIT-OarVl_c5E8pDlw>
> (4.69.226.222) 0.0% 100 90.7 91.7 90.3 104.3 2.5
> 16. AS3356 ae1.11.ebr4.NewYork6.level3.net
> <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht1%2F%2Fet%3A4..1nbr.1e.oewek6lyrten.3lev&s=AdrzHMpY4tnoB9EYO0QT3ExzOfU>
> (4.69.209.38) 0.0% 100 105.4 91.7 90.3 105.5 2.6
> 17. AS3356 ae10.10.ebr1.SanJose1.level3.net
> <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht1%2F%2Fet%3Ar00..eb11.1jsalse.noten.3leve&s=yYgFmKz5WsmUhsGu6IBKFW1RJqs>
> (4.69.160.78) 1.0% 100 804.5 800.9 669.2 871.6 46.6
> 18. AS3356 ae1.0.ebr2.SanJose1.level3.net
> <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht1%2F%2Fet%3A.e.0ar2s.blsnjv1.eoeten.3le&s=zM84PLGELMdmUMx6E7ZUUQtVGoc>
> (4.69.142.70) 0.0% 100 819.3 802.1 667.4 879.3 47.5
> 19. AS3356 ae2.30.bear2.Sacramento1.level3.net
> <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=apht2%2F%2Fet%3Ar..3.ea20bnrsaometca3e1.nel.lvte&s=gzqPYSqc17FtpgaKe5VbgiNNmSY>
> (4.69.141.77) 11.0% 100 825.1 801.5 668.0 865.0 45.1
> 20. AS3356 system.0.bear1.Sacramento1.level3.net
> <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=sphts%2F%2Fyt%3Ae.ter.bam0ma1.nraesce.to3evl1lten.&s=HgYEJnrRdawfz5Wi45acR276RGw>
> (4.69.183.132) 0.0% 100 821.3 800.5 662.7 865.9 47.1
>
>
>
>
>
>
>
>
>
> Thanks.
>
>
>
>
>
>
> --
>
> Anurag Bhatia
>
> anuragbhatia.com
> <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=%2Fphtn%3A%2Fatstgurahaiabmoc.&s=S-51H0xHJ-Z8oewcounpVrWuXp0>
>
>
>
>
> --
>
> Anurag Bhatia
>
> anuragbhatia.com
> <https://url3.mailanyone.net/scanner?m=1sR7BO-00HMKo-4v&d=4%7Cmail%2F90%2F1720517400%2F1sR7BO-00HMKo-4v%7Cin3j%7C57e1b682%7C17620982%7C13443422%7C668D0346BC467064C0DE91F53757F66E&o=%2Fphtn%3A%2Fatstgurahaiabmoc.&s=S-51H0xHJ-Z8oewcounpVrWuXp0>
> ------------------------------
>
> This email has been scanned for spam & viruses. If you believe this email
> should have been stopped by our filters, click here
> <https://portal.mailanyone.net/index.html#/outer/reportspam?token=dXNlcj1kb3J0aXpAenRoZXJuZXQuY29tO3RzPTE3MjA1MTc0NTM7dXVpZD02NjhEMDM0NkJDNDY3MDY0QzBERTkxRjUzNzU3RjY2RTt0b2tlbj04MjAwYzBmMTJmZTU4YzY5YzkzNDZlZDIxODU4YmI3NDVlYjEzMWExOw%3D%3D>
> to report it.
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20240709/63957588/attachment-0001.htm>
More information about the Outages
mailing list