[outages] [CloudOps] Major loss/latency on Twelve99 around NY/SJO
Phil Lavin
phil.lavin at vonage.com
Thu Aug 15 06:30:20 EDT 2024
See traffic on Twelve99 re-routed via LAX now. Seems to be stable, for the moment:
My traceroute [v0.95]
VONMVPCVXYVFD (172.20.0.219) -> 216.147.3.209 (216.147.3.209) 2024-08-15T11:29:35+0100
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. 172.20.0.1 0.0% 68 0.7 1.0 0.7 1.9 0.2
2. lo0-0.bng2.ixn-lon.zen.net.uk 0.0% 67 12.3 8.4 5.1 43.1 6.1
3. lag-5.p2.ixn-lon.zen.net.uk 0.0% 67 5.6 5.9 5.4 7.8 0.4
4. ldn-b13-link.ip.twelve99.net 45.5% 67 5.6 5.5 4.9 6.1 0.3
5. ldn-bb2-link.ip.twelve99.net 0.0% 67 6.1 6.3 5.4 8.9 0.6
6. nyk-bb1-link.ip.twelve99.net 0.0% 67 74.7 75.0 74.1 77.0 0.5
7. rest-bb1-link.ip.twelve99.net 1.5% 67 81.5 81.5 80.8 83.9 0.5
8. lax-b22-link.ip.twelve99.net 0.0% 67 137.5 138.4 136.7 175.7 5.4
9. (waiting for reply)
10. (waiting for reply)
11. 15.230.16.155 0.0% 67 141.6 143.9 140.3 167.9 5.4
12. (waiting for reply)
13. 52.93.14.118 0.0% 67 216.6 178.8 165.8 267.3 20.9
14. 52.93.14.105 0.0% 67 171.5 171.4 170.2 175.7 1.0
15. 52.93.15.28 0.0% 67 169.2 179.0 169.0 238.9 15.5
16. 52.93.246.191 0.0% 67 166.1 167.8 165.3 194.0 4.4
17. 52.93.246.81 0.0% 67 172.3 172.4 171.3 179.7 1.4
18. (waiting for reply)
19. (waiting for reply)
20. (waiting for reply)
21. (waiting for reply)
22. (waiting for reply)
23. (waiting for reply)
24. 216.147.3.209 0.0% 67 166.4 167.1 166.0 169.1 0.6
> On 15 Aug 2024, at 11:09, Phil Lavin <phil.lavin at vonage.com> wrote:
>
> Fwiw, still ongoing
>
> I raised an AWS ticket and they’re looking into it. If there’s any friendly AWS Net Engineers on list, your help removing Twelve99 from route would be appreciated. Cross-posting to Nanog for wider reach.
>
>
>> On 15 Aug 2024, at 09:44, Matt Almgren <malmgren at backblaze.com> wrote:
>>
>> There's a planned SJC maintenance happening now. Expected end time is 2am PST.
>>
>> On Thu, Aug 15, 2024, 1:38 AM Phil Lavin via Outages <outages at outages.org <mailto:outages at outages.org>> wrote:
>>> It’s a good week for the Internet. Third outages list post this week :(
>>>
>>> Seeing major loss on Twelve99 around SJO:
>>>
>>> UK to AWS USW2:
>>>
>>> My traceroute [v0.95]
>>> VONMVPCVXYVFD (172.20.0.14) -> 216.147.3.209 (216.147.3.209) 2024-08-15T09:29:13+0100
>>> Keys: Help Display mode Restart statistics Order of fields quit
>>> Packets Pings
>>> Host Loss% Snt Last Avg Best Wrst StDev
>>> 1. 172.20.0.1 0.0% 9 1.1 1.2 1.0 1.7 0.2
>>> 2. lo0-0.bng2.ixn-lon.zen.net.uk <http://lo0-0.bng2.ixn-lon.zen.net.uk/> 0.0% 9 5.5 6.8 5.4 11.2 1.9
>>> 3. lag-5.p2.ixn-lon.zen.net.uk <http://lag-5.p2.ixn-lon.zen.net.uk/> 0.0% 8 5.5 6.0 5.2 6.5 0.5
>>> 4. ldn-b13-link.ip.twelve99.net <http://ldn-b13-link.ip.twelve99.net/> 25.0% 8 5.8 5.5 5.0 6.0 0.4
>>> 5. ldn-bb2-link.ip.twelve99.net <http://ldn-bb2-link.ip.twelve99.net/> 0.0% 8 6.6 6.8 5.9 7.5 0.7
>>> 6. nyk-bb1-link.ip.twelve99.net <http://nyk-bb1-link.ip.twelve99.net/> 0.0% 8 75.2 75.2 74.5 75.9 0.5
>>> 7. sjo-b23-link.ip.twelve99.net <http://sjo-b23-link.ip.twelve99.net/> 57.1% 8 296.2 312.5 296.2 343.7 27.0
>>> 8. (waiting for reply)
>>> 9. 15.230.28.67 62.5% 8 294.4 295.7 294.4 296.8 1.2
>>> 10. 54.240.242.119 25.0% 8 337.1 305.7 294.6 337.1 16.2
>>> 11. (waiting for reply)
>>> 12. 52.93.15.36 57.1% 8 398.2 400.8 395.4 408.8 7.1
>>> 13. 52.93.15.31 42.9% 8 388.7 395.5 388.7 403.3 6.2
>>> 14. 52.93.14.34 28.6% 8 431.8 432.2 393.2 488.2 35.8
>>> 15. 52.93.246.177 42.9% 8 397.7 399.3 397.7 401.1 1.4
>>> 16. 52.93.246.81 25.0% 8 393.4 396.9 393.2 400.6 3.0
>>> 17. (waiting for reply)
>>> 18. (waiting for reply)19. (waiting for re 28.6% 8 394.3 397.5 394.3 400.2 3.0
>>>
>>> AWS USW2 to UK:
>>>
>>> Host Loss% Snt Last Avg Best Wrst StDev
>>> 1. ip-10-96-50-153.us-west-2.compute.internal 0.0% 87 0.2 0.1 0.1 0.2 0.0
>>> 2. ec2-44-233-118-55.us-west-2.compute.amazonaws.com <http://ec2-44-233-118-55.us-west-2.compute.amazonaws.com/> 0.0% 87 7.3 11.9 1.0 66.9 16.4
>>> ec2-34-221-151-47.us-west-2.compute.amazonaws.com <http://ec2-34-221-151-47.us-west-2.compute.amazonaws.com/>
>>> ec2-44-233-117-187.us-west-2.compute.amazonaws.com <http://ec2-44-233-117-187.us-west-2.compute.amazonaws.com/>
>>> 3. 240.0.212.4 0.0% 87 0.5 0.4 0.3 1.7 0.2
>>> 240.0.72.7
>>> 4. 242.2.110.95 0.0% 87 0.8 1.8 0.4 20.6 3.4
>>> 5. 240.2.144.13 0.0% 87 5.8 5.8 5.7 5.9 0.1
>>> 6. amazon-ic-375418.ip.twelve99-cust.net <http://amazon-ic-375418.ip.twelve99-cust.net/> 0.0% 87 5.7 5.6 5.6 6.1 0.1
>>> 7. port-b4-link.ip.twelve99.net <http://port-b4-link.ip.twelve99.net/> 0.0% 87 6.2 6.1 5.9 6.5 0.1
>>> 8. port-b3-link.ip.twelve99.net <http://port-b3-link.ip.twelve99.net/> 0.0% 86 5.6 6.3 5.6 18.5 1.8
>>> 9. palo-b24-link.ip.twelve99.net <http://palo-b24-link.ip.twelve99.net/> 0.0% 86 21.4 21.3 21.1 21.6 0.1
>>> 10. nyk-bb2-link.ip.twelve99.net <http://nyk-bb2-link.ip.twelve99.net/> 19.8% 86 165.3 172.2 165.3 206.1 4.7
>>> 11. ldn-bb1-link.ip.twelve99.net <http://ldn-bb1-link.ip.twelve99.net/> 68.2% 86 391.5 401.2 387.3 411.8 6.3
>>> 12. ldn-b3-link.ip.twelve99.net <http://ldn-b3-link.ip.twelve99.net/> 57.0% 86 397.1 401.4 395.5 409.0 2.7
>>> 13. zeninternet-ic-363994.ip.twelve99-cust.net <http://zeninternet-ic-363994.ip.twelve99-cust.net/> 40.7% 86 397.0 412.3 397.0 473.3 18.0
>>> 14. lag-1.bng2.ixn-lon.zen.net.uk <http://lag-1.bng2.ixn-lon.zen.net.uk/> 63.5% 86 387.2 402.5 382.5 422.8 9.0
>>> 15. 62-3-100-30.dsl.in-addr.zen.co.uk <http://62-3-100-30.dsl.in-addr.zen.co.uk/> 55.8% 86 391.6 404.5 391.6 412.5 6.1
>>> 16. (waiting for reply)
>>> _______________________________________________
>>> Outages mailing list
>>> Outages at outages.org <mailto:Outages at outages.org>
>>> https://puck.nether.net/mailman/listinfo/outages
>>
>> This email, including its contents and any attachment(s), may contain confidential and/or proprietary information and is solely for the review and use of the intended recipient(s). If you have received this email in error, please notify the sender and permanently delete this email, its content, and any attachment(s). Any disclosure, copying, or taking of any action in reliance on an email received in error is strictly prohibited.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20240815/890966da/attachment.htm>
More information about the Outages
mailing list