[outages] flapping issues reaching East Coast AWS

Mon Jul 31 16:49:00 EDT 2023

We are seeing similar issues from Santa Monica trying to reach our sites in the DC area today. I made a ticket with our east coast ISP Allied and they mentioned that AT&T was having peering issues with some ISPs in California. We are BGP'd with AT&T and Frontier and usually our outbound traffic rides on AT&T and inbound traffic picks whichever route is better between the two ISPs.  I moved traffic from AT&T to Frontier and then our others sites started to drop so I backed off. I'll try removing the Frontier routes later and see if things improve.

Jose Gomez

-----Original Message-----
From: Outages <outages-bounces at outages.org> On Behalf Of William Kern via Outages
Sent: Monday, July 31, 2023 12:23 PM
To: outages at outages.org
Subject: [outages] flapping issues reaching East Coast AWS

We have seen flapping all morning reaching our AWS East Coast servers from our Thousand Oaks location. They drop off for a minute or so and then come back.

It seems to occur at 3-5 minute intervals for several cycles, then everything is fine for 30 minutes or so. Then the alarms start to come in again so wash rinse and repeat.

The issue is definately not local, as our local monitoring to our peer connections are fine as are our connections to AWS WestCoast servers. 
The only complaint I got so far was from a customer on the East cost trying to VPN to a server here and getting booted out a lot.

Traceroutes imply there may be an issue with Frontier as the few times I have caught the problem while logged into the remote East Coast server the path outbound from AWS East goes to Dallas and then seems to have issue reaching Los Angeles. I also show those traceroute pages moving around a bit between those locations.

The connection from here to the the East Cost show weirdness involving Charter so maybe the issue is on the reverse trip.

this is pretty typical.

  8  lag-401.dllstx976iw-bcr00.netops.charter.com (66.109.5.229) 36.002 ms  36.316 ms lag-16.dllstx976iw-bcr00.netops.charter.com (66.109.6.1)
36.451 ms
  9  lag-0.pr3.dfw10.netops.charter.com (66.109.5.121)  39.187 ms lag-302.pr3.dfw10.netops.charter.com (209.18.43.77)  42.279 ms 45.338 ms
10  99.83.71.242 (99.83.71.242)  35.427 ms 99.83.71.240 (99.83.71.240)
36.389 ms 99.82.176.170 (99.82.176.170)  35.339 ms
11  * 150.222.206.169 (150.222.206.169)  34.071 ms *
12  * * 15.230.48.42 (15.230.48.42)  33.717 ms

Finally its not within AWS as our WestCoast servers can maintain a steady ping with the affected East Coast sites.

So is anyone else seeing weirdness on cross country trips?

William Kern

PixelGate

_______________________________________________
Outages mailing list
Outages at outages.org
https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman_listinfo_outages&d=DwIGaQ&c=UtbViGLMoQq17uDUqpM_9A&r=UuApoUz30cXKbJIUtc8kezBFqg4H3SVzFz95ksh7T7M&m=p_NqNaE60PttM3wrSQvpWgTX5jNSyUB04sP6Z1mRGHOtGfr76RMTdXCd8Wve4_Ow&s=kDELerzTlOJQypprUN7vkLHLcTQNzksKVnAjBEF6Pac&e= 