From sean at donelan.com Fri Aug 5 16:21:27 2022 From: sean at donelan.com (Sean Donelan) Date: Fri, 5 Aug 2022 16:21:27 -0400 (EDT) Subject: [outages] Wireless Emergency Alerts (weather) outage Message-ID: Always have alternate ways to receive alerts (broadcast radio & TV, weather radio, etc.) SENIOR DUTY METEOROLOGIST NWS ALERT ADMINISTRATIVE MESSAGE NWS NCEP CENTRAL OPERATIONS COLLEGE PARK MD 2006Z FRI AUG 05 2022 ...WEATHER EMERGENCY ALERT OUTAGE... Since 1558Z...a system issue at FEMA is preventing Weather Emergency Alert (WEA) messages from getting out to cell phones. Until this issue is corrected, users should use other methods to receive NWS warnings. Handel/SDM/NCO/NCEP From sean at donelan.com Fri Aug 5 21:53:20 2022 From: sean at donelan.com (Sean Donelan) Date: Fri, 5 Aug 2022 21:53:20 -0400 (EDT) Subject: [outages] Wireless Emergency Alerts (weather) outage In-Reply-To: References: Message-ID: Wireless Emergency Alert problems resolved. A digital certificate problem with the vendor was fixed. From jschauma at netmeister.org Mon Aug 8 22:22:38 2022 From: jschauma at netmeister.org (Jan Schaumann) Date: Mon, 8 Aug 2022 22:22:38 -0400 Subject: [outages] Google failures Message-ID: <20220809022237.GA11880@netmeister.org> I'm seeing intermittent failures for www.google.com. $ curl -I https://www.google.com HTTP/2 502 content-type: text/html; charset=UTF-8 referrer-policy: no-referrer content-length: 1613 date: Tue, 09 Aug 2022 01:41:35 GMT Previously saw a 500 Internal Server Error. https://downdetector.com/status/google/ and social media show increased reports as well, though things are back to working for me in NYC. Still, it's been a while that I've seen www.google.com fail, so thought I'd post it here... From robertsdoug at gmail.com Mon Aug 8 22:25:01 2022 From: robertsdoug at gmail.com (Doug Roberts) Date: Mon, 8 Aug 2022 19:25:01 -0700 Subject: [outages] Google failures In-Reply-To: <20220809022237.GA11880@netmeister.org> References: <20220809022237.GA11880@netmeister.org> Message-ID: I have been getting 500 server errors on various Google services for several hours now. On Mon, Aug 8, 2022 at 7:24 PM Jan Schaumann via Outages < outages at outages.org> wrote: > I'm seeing intermittent failures for www.google.com. > > $ curl -I https://www.google.com > HTTP/2 502 > content-type: text/html; charset=UTF-8 > referrer-policy: no-referrer > content-length: 1613 > date: Tue, 09 Aug 2022 01:41:35 GMT > > Previously saw a 500 Internal Server Error. > > https://downdetector.com/status/google/ and social > media show increased reports as well, though things > are back to working for me in NYC. > > Still, it's been a while that I've seen www.google.com > fail, so thought I'd post it here... > > _______________________________________________ > Outages mailing list > Outages at outages.org > https://puck.nether.net/mailman/listinfo/outages > -- -- *Doug Roberts* Doug at DougR.com 702-846-5000 (Las Vegas) 480-401-5000 (Phoenix) 212-220-0000 (New York) 702-874-4428 (Fax) CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended for the addressee(s) named in this message. This communication may contain confidential and/or privileged information and may be legally protected from disclosure. If you are not the intended recipient of this message, or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and its attachments. Do not disclose the contents or take any action in reliance upon the information contained in this communication or any attachments. Do not deliver, distribute or copy this message and/or any attachments and, if you are not the intended recipient, you are hereby notified that any use, dissemination, copying, or storage of this message or its attachments is strictly prohibited. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ryan at digicana.com Mon Aug 8 22:26:27 2022 From: ryan at digicana.com (Ryan McGinnis) Date: Tue, 09 Aug 2022 02:26:27 +0000 Subject: [outages] Google failures In-Reply-To: <20220809022237.GA11880@netmeister.org> References: <20220809022237.GA11880@netmeister.org> Message-ID: https://www.ketv.com/article/council-bluffs-google-data-center-electrical-event-three-people-critically-injured/40838924 Appears to be back up and running normal again. Perhaps someone asked Google?s AI chatbot ?what is love?? -Ryan McGinnis ryan at digicana.com [https://bigstormpicture.com](http://bigstormpicture.com) GPG: 5C73 8727 EE58 786A 777C 4F1D B5AA 3FA3 486E D7AD On Mon, Aug 8, 2022 at 21:22, Jan Schaumann via Outages wrote: > I'm seeing intermittent failures for www.google.com. > > $ curl -I https://www.google.com > HTTP/2 502 > content-type: text/html; charset=UTF-8 > referrer-policy: no-referrer > content-length: 1613 > date: Tue, 09 Aug 2022 01:41:35 GMT > > Previously saw a 500 Internal Server Error. > > https://downdetector.com/status/google/ and social > media show increased reports as well, though things > are back to working for me in NYC. > > Still, it's been a while that I've seen www.google.com > fail, so thought I'd post it here... > > _______________________________________________ > Outages mailing list > Outages at outages.org > https://puck.nether.net/mailman/listinfo/outages -------------- next part -------------- An HTML attachment was scrubbed... URL: From jhost at vultr.com Mon Aug 8 22:28:29 2022 From: jhost at vultr.com (James Host) Date: Mon, 8 Aug 2022 19:28:29 -0700 Subject: [outages] Google failures In-Reply-To: References: <20220809022237.GA11880@netmeister.org> Message-ID: >From what I know, there was a DC explosion in IA when doing electrical maint and some guys got hurt. Hope they're ok. James Host Architect - CDN Platform E: jhost at vultr.com On Mon, Aug 8, 2022 at 7:26 PM Doug Roberts via Outages wrote: > I have been getting 500 server errors on various Google services for > several hours now. > > On Mon, Aug 8, 2022 at 7:24 PM Jan Schaumann via Outages < > outages at outages.org> wrote: > >> I'm seeing intermittent failures for www.google.com. >> >> $ curl -I https://www.google.com >> HTTP/2 502 >> content-type: text/html; charset=UTF-8 >> referrer-policy: no-referrer >> content-length: 1613 >> date: Tue, 09 Aug 2022 01:41:35 GMT >> >> Previously saw a 500 Internal Server Error. >> >> https://downdetector.com/status/google/ and social >> media show increased reports as well, though things >> are back to working for me in NYC. >> >> Still, it's been a while that I've seen www.google.com >> fail, so thought I'd post it here... >> >> _______________________________________________ >> Outages mailing list >> Outages at outages.org >> https://puck.nether.net/mailman/listinfo/outages >> > -- > > -- > *Doug Roberts* > Doug at DougR.com > 702-846-5000 (Las Vegas) > 480-401-5000 (Phoenix) > 212-220-0000 (New York) > 702-874-4428 (Fax) > > CONFIDENTIALITY NOTICE: The contents of this e-mail message and any > attachments are intended for the addressee(s) named in this message. This > communication may contain confidential and/or privileged information and > may be legally protected from disclosure. If you are not the intended > recipient of this message, or if this message has been addressed to you in > error, please immediately alert the sender by reply e-mail and then delete > this message and its attachments. Do not disclose the contents or take any > action in reliance upon the information contained in this communication or > any attachments. Do not deliver, distribute or copy this message and/or any > attachments and, if you are not the intended recipient, you are hereby > notified that any use, dissemination, copying, or storage of this message > or its attachments is strictly prohibited. > _______________________________________________ > Outages mailing list > Outages at outages.org > https://puck.nether.net/mailman/listinfo/outages > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tls at panix.com Tue Aug 9 00:24:51 2022 From: tls at panix.com (Thor Lancelot Simon) Date: Tue, 9 Aug 2022 00:24:51 -0400 Subject: [outages] Google failures In-Reply-To: <20220809022237.GA11880@netmeister.org> References: <20220809022237.GA11880@netmeister.org> Message-ID: On Mon, Aug 08, 2022 at 10:22:38PM -0400, Jan Schaumann via Outages wrote: > I'm seeing intermittent failures for www.google.com. Also seeing 500 errors for Google searches. Seemed to peak about an hour ago. Thor From jonathan.selea at portsgroup.com Tue Aug 9 01:42:05 2022 From: jonathan.selea at portsgroup.com (=?utf-8?B?Sm9uYXRoYW4gU8OpbGVh?=) Date: Tue, 9 Aug 2022 05:42:05 +0000 Subject: [outages] Google failures In-Reply-To: References: <20220809022237.GA11880@netmeister.org> Message-ID: NetActuate reported a partial disruption at their IAD-3 location. Seems like both incidents are related to each other? Med v?nlig h?lsning / Best Regards?? Jonathan S?lea +46 70 726 00 50 jonathan.selea at portsgroup.com G?teborg, Kungsgatan 42 From: Outages On Behalf Of James Host via Outages Sent: den 9 augusti 2022 04:28 To: Doug Roberts Cc: outages at outages.org Subject: Re: [outages] Google failures >From what I know, there was a DC explosion in IA when doing electrical maint and some guys got hurt. Hope they're ok. James Host Architect - CDN Platform E: jhost at vultr.com On Mon, Aug 8, 2022 at 7:26 PM Doug Roberts via Outages > wrote: I have been getting 500 server errors on various Google services for several hours now. On Mon, Aug 8, 2022 at 7:24 PM Jan Schaumann via Outages > wrote: I'm seeing intermittent failures for www.google.com . $ curl -I https://www.google.com HTTP/2 502 content-type: text/html; charset=UTF-8 referrer-policy: no-referrer content-length: 1613 date: Tue, 09 Aug 2022 01:41:35 GMT Previously saw a 500 Internal Server Error. https://downdetector.com/status/google/ and social media show increased reports as well, though things are back to working for me in NYC. Still, it's been a while that I've seen www.google.com fail, so thought I'd post it here... _______________________________________________ Outages mailing list Outages at outages.org https://puck.nether.net/mailman/listinfo/outages -- -- Doug Roberts Doug at DougR.com 702-846-5000 (Las Vegas) 480-401-5000 (Phoenix) 212-220-0000 (New York) 702-874-4428 (Fax) CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended for the addressee(s) named in this message. This communication may contain confidential and/or privileged information and may be legally protected from disclosure. If you are not the intended recipient of this message, or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and its attachments. Do not disclose the contents or take any action in reliance upon the information contained in this communication or any attachments. Do not deliver, distribute or copy this message and/or any attachments and, if you are not the intended recipient, you are hereby notified that any use, dissemination, copying, or storage of this message or its attachments is strictly prohibited. _______________________________________________ Outages mailing list Outages at outages.org https://puck.nether.net/mailman/listinfo/outages -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 272 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.png Type: image/png Size: 291 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image003.png Type: image/png Size: 278 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image004.png Type: image/png Size: 2352 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4968 bytes Desc: not available URL: From beckman at angryox.com Tue Aug 9 14:02:04 2022 From: beckman at angryox.com (Peter Beckman) Date: Tue, 9 Aug 2022 14:02:04 -0400 Subject: [outages] Google failures In-Reply-To: References: <20220809022237.GA11880@netmeister.org> Message-ID: On Tue, 9 Aug 2022, Ryan McGinnis via Outages wrote: > https://www.ketv.com/article/council-bluffs-google-data-center-electrical-event-three-people-critically-injured/40838924 This is the kind of event that needs the Five Whys asked and a deep dive into what processes were not followed or what processes need to be put in place to avoid a similar situation. I wonder if this is a US Consumer Safety Board worthy investigation. Electricity is scary stuff if you make mistakes. I hope the team injured are able to recover fully. Beckman --------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/ --------------------------------------------------------------------------- -------------- next part -------------- _______________________________________________ Outages mailing list Outages at outages.org https://puck.nether.net/mailman/listinfo/outages From chris at object-zone.net Tue Aug 9 14:42:18 2022 From: chris at object-zone.net (Chris Dickens) Date: Tue, 9 Aug 2022 14:42:18 -0400 Subject: [outages] Google failures In-Reply-To: References: <20220809022237.GA11880@netmeister.org> Message-ID: <003301d8ac1f$c0fd9ec0$42f8dc40$@object-zone.net> I have monitoring systems from Paessler known as PRTG. These systems have sensors known as "Common SaaS" where a variety of online services like Google, Facebook and Youtube amongst others are checked. This morning I've been seeing on and off alarms that a Host is not found, or "Read timeout". The reports are from discrete datacenter connections. I suspect these are related - does anyone here know for sure? --Chris -----Original Message----- From: Outages On Behalf Of Thor Lancelot Simon via Outages Sent: Tuesday, August 9, 2022 12:25 AM To: Jan Schaumann Cc: outages at outages.org Subject: Re: [outages] Google failures On Mon, Aug 08, 2022 at 10:22:38PM -0400, Jan Schaumann via Outages wrote: > I'm seeing intermittent failures for www.google.com. Also seeing 500 errors for Google searches. Seemed to peak about an hour ago. Thor _______________________________________________ Outages mailing list Outages at outages.org https://puck.nether.net/mailman/listinfo/outages From josh at imaginenetworksllc.com Tue Aug 9 14:46:57 2022 From: josh at imaginenetworksllc.com (Josh Luthman) Date: Tue, 9 Aug 2022 14:46:57 -0400 Subject: [outages] Google failures In-Reply-To: <003301d8ac1f$c0fd9ec0$42f8dc40$@object-zone.net> References: <20220809022237.GA11880@netmeister.org> <003301d8ac1f$c0fd9ec0$42f8dc40$@object-zone.net> Message-ID: Can we discuss it in please? On Tue, Aug 9, 2022 at 2:43 PM Chris Dickens via Outages < outages at outages.org> wrote: > I have monitoring systems from Paessler known as PRTG. These systems have > sensors known as "Common SaaS" where a variety of online services like > Google, Facebook and Youtube amongst others are checked. This morning I've > been seeing on and off alarms that a Host is not found, or "Read timeout". > The reports are from discrete datacenter connections. > > I suspect these are related - does anyone here know for sure? > > --Chris > > -----Original Message----- > From: Outages On Behalf Of Thor Lancelot > Simon > via Outages > Sent: Tuesday, August 9, 2022 12:25 AM > To: Jan Schaumann > Cc: outages at outages.org > Subject: Re: [outages] Google failures > > On Mon, Aug 08, 2022 at 10:22:38PM -0400, Jan Schaumann via Outages wrote: > > I'm seeing intermittent failures for www.google.com. > > Also seeing 500 errors for Google searches. Seemed to peak about an hour > ago. > > Thor > _______________________________________________ > Outages mailing list > Outages at outages.org > https://puck.nether.net/mailman/listinfo/outages > > > _______________________________________________ > Outages mailing list > Outages at outages.org > https://puck.nether.net/mailman/listinfo/outages > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aoconner at tnacomp.com Tue Aug 9 14:48:46 2022 From: aoconner at tnacomp.com (Aaron OConner) Date: Tue, 9 Aug 2022 18:48:46 +0000 Subject: [outages] Google failures In-Reply-To: <003301d8ac1f$c0fd9ec0$42f8dc40$@object-zone.net> References: <20220809022237.GA11880@netmeister.org> <003301d8ac1f$c0fd9ec0$42f8dc40$@object-zone.net> Message-ID: I think they are having problems, or a transit problem getting to them. I can get to all of the sensor monitored sites locally. I have two separate systems, one licensed and one on the free license, that are both having problems. The licensed system says it failed activation, and I'm getting alerts on the same sensor you are on the free one. I have a ticket open with them, but no response yet. -----Original Message----- From: Outages On Behalf Of Chris Dickens via Outages Sent: Tuesday, August 9, 2022 1:42 PM To: 'Thor Lancelot Simon' ; 'Jan Schaumann' Cc: outages at outages.org Subject: Re: [outages] Google failures I have monitoring systems from Paessler known as PRTG. These systems have sensors known as "Common SaaS" where a variety of online services like Google, Facebook and Youtube amongst others are checked. This morning I've been seeing on and off alarms that a Host is not found, or "Read timeout". The reports are from discrete datacenter connections. I suspect these are related - does anyone here know for sure? --Chris -----Original Message----- From: Outages On Behalf Of Thor Lancelot Simon via Outages Sent: Tuesday, August 9, 2022 12:25 AM To: Jan Schaumann Cc: outages at outages.org Subject: Re: [outages] Google failures On Mon, Aug 08, 2022 at 10:22:38PM -0400, Jan Schaumann via Outages wrote: > I'm seeing intermittent failures for www.google.com. Also seeing 500 errors for Google searches. Seemed to peak about an hour ago. Thor _______________________________________________ Outages mailing list Outages at outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages at outages.org https://puck.nether.net/mailman/listinfo/outages This email may contain confidential and privileged material for the sole use of the intended recipient. Any review, use, distribution or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive for the recipient), please contact the sender by reply email and delete all copies of this message. From etieseler at metrolist.net Mon Aug 15 13:22:58 2022 From: etieseler at metrolist.net (Eduard Tieseler) Date: Mon, 15 Aug 2022 10:22:58 -0700 Subject: [outages] Intellepeer SIP outage? Message-ID: An HTML attachment was scrubbed... URL: From bclark at peregrinenetworks.net Mon Aug 15 13:25:58 2022 From: bclark at peregrinenetworks.net (Bret Clark) Date: Mon, 15 Aug 2022 17:25:58 +0000 Subject: [outages] Intellepeer SIP outage? In-Reply-To: References: Message-ID: Yes, started around 1PM, downdector shows that COX is having phone issues too! Bret ________________________________ From: Outages on behalf of Eduard Tieseler via Outages Sent: Monday, August 15, 2022 1:22 PM To: outages at outages.org Subject: [outages] Intellepeer SIP outage? Anyone else having an outage on their SIP trunk with intellepeer? Our phones went down and their phones appear to be down as well. -------------- next part -------------- An HTML attachment was scrubbed... URL: From shripal.daphtary at pressone.net Mon Aug 15 13:26:42 2022 From: shripal.daphtary at pressone.net (Shripal Daphtary) Date: Mon, 15 Aug 2022 13:26:42 -0400 Subject: [outages] Intellepeer SIP outage? In-Reply-To: References: Message-ID: yes.. we're down from Ipeer. Shripal Daphtary shripal.daphtary at pressone.net D: 718.360.1222 C: 973.432.1440 F: 929.369.0900 On Mon, Aug 15, 2022 at 1:23 PM Eduard Tieseler via Outages < outages at outages.org> wrote: > Anyone else having an outage on their SIP trunk with intellepeer? > Our phones went down and their phones appear to be down as well. > _______________________________________________ > Outages mailing list > Outages at outages.org > https://puck.nether.net/mailman/listinfo/outages > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bclark at peregrinenetworks.net Mon Aug 15 13:40:22 2022 From: bclark at peregrinenetworks.net (Bret Clark) Date: Mon, 15 Aug 2022 17:40:22 +0000 Subject: [outages] Intellepeer SIP outage? In-Reply-To: References: Message-ID: I opened a ticket with Intelepeer, but haven't heard anything back. I tried calling them of course there support DID isn't working right now. Anyone hear of what the issue might be? ________________________________ From: Outages on behalf of Shripal Daphtary via Outages Sent: Monday, August 15, 2022 1:26 PM To: Eduard Tieseler Cc: outages at outages.org Subject: Re: [outages] Intellepeer SIP outage? yes.. we're down from Ipeer. [https://ci3.googleusercontent.com/mail-sig/AIorK4z9pYm7j4k4gsrUnswgrW5gHXwM5DkYgIbACcbbLQXAfG3Wk5bem0ubGNIpFW7SYgm-S941JL50Z1nMckCgvM5FfGHA3ntra1oyO_rkVg] Shripal Daphtary shripal.daphtary at pressone.net D: 718.360.1222 C: 973.432.1440 F: 929.369.0900 On Mon, Aug 15, 2022 at 1:23 PM Eduard Tieseler via Outages > wrote: Anyone else having an outage on their SIP trunk with intellepeer? Our phones went down and their phones appear to be down as well. _______________________________________________ Outages mailing list Outages at outages.org https://puck.nether.net/mailman/listinfo/outages -------------- next part -------------- An HTML attachment was scrubbed... URL: From etieseler at metrolist.net Mon Aug 15 13:44:11 2022 From: etieseler at metrolist.net (Eduard Tieseler) Date: Mon, 15 Aug 2022 10:44:11 -0700 Subject: [outages] Intellepeer SIP outage? Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- We did the same. No response at this time and I haven't been able to find any further details about the issue. On Aug 15, 2022 10:40 AM, Bret Clark via Outages wrote: I opened a ticket with Intelepeer, but haven't heard anything back. I tried calling them of course there support DID isn't working right now. Anyone hear of what the issue might be? ________________________________ From: Outages on behalf of Shripal Daphtary via Outages Sent: Monday, August 15, 2022 1:26 PM To: Eduard Tieseler Cc: outages at outages.org Subject: Re: [outages] Intellepeer SIP outage? yes.. we're down from Ipeer. [https://ci3.googleusercontent.com/mail-sig/AIorK4z9pYm7j4k4gsrUnswgrW5gHXwM5DkYgIbACcbbLQXAfG3Wk5bem0ubGNIpFW7SYgm-S941JL50Z1nMckCgvM5FfGHA3ntra1oyO_rkVg] Shripal Daphtary shripal.daphtary at pressone.net D: 718.360.1222 C: 973.432.1440 F: 929.369.0900 On Mon, Aug 15, 2022 at 1:23 PM Eduard Tieseler via Outages > wrote: Anyone else having an outage on their SIP trunk with intellepeer? Our phones went down and their phones appear to be down as well. _______________________________________________ Outages mailing list Outages at outages.org https://puck.nether.net/mailman/listinfo/outages From etieseler at metrolist.net Mon Aug 15 13:49:49 2022 From: etieseler at metrolist.net (Eduard Tieseler) Date: Mon, 15 Aug 2022 10:49:49 -0700 Subject: [outages] Intellepeer SIP outage? Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- We just came back up.? On Aug 15, 2022 10:44 AM, Eduard Tieseler via Outages wrote: We did the same. No response at this time and I haven't been able to find any further details about the issue. On Aug 15, 2022 10:40 AM, Bret Clark via Outages wrote: I opened a ticket with Intelepeer, but haven't heard anything back. I tried calling them of course there support DID isn't working right now. Anyone hear of what the issue might be? ________________________________ From: Outages on behalf of Shripal Daphtary via Outages Sent: Monday, August 15, 2022 1:26 PM To: Eduard Tieseler Cc: outages at outages.org Subject: Re: [outages] Intellepeer SIP outage? yes.. we're down from Ipeer. [https://ci3.googleusercontent.com/mail-sig/AIorK4z9pYm7j4k4gsrUnswgrW5gHXwM5DkYgIbACcbbLQXAfG3Wk5bem0ubGNIpFW7SYgm-S941JL50Z1nMckCgvM5FfGHA3ntra1oyO_rkVg] Shripal Daphtary shripal.daphtary at pressone.net D: 718.360.1222 C: 973.432.1440 F: 929.369.0900 On Mon, Aug 15, 2022 at 1:23 PM Eduard Tieseler via Outages > wrote: Anyone else having an outage on their SIP trunk with intellepeer? Our phones went down and their phones appear to be down as well. _______________________________________________ Outages mailing list Outages at outages.org https://puck.nether.net/mailman/listinfo/outages From bclark at peregrinenetworks.net Mon Aug 15 13:52:08 2022 From: bclark at peregrinenetworks.net (Bret Clark) Date: Mon, 15 Aug 2022 17:52:08 +0000 Subject: [outages] Intellepeer SIP outage? In-Reply-To: References: Message-ID: Yes, we are seeing calls again. Bret ________________________________ From: Eduard Tieseler Sent: Monday, August 15, 2022 1:49 PM To: Bret Clark Cc: outages at outages.org Subject: Re: [outages] Intellepeer SIP outage? We just came back up. On Aug 15, 2022 10:44 AM, Eduard Tieseler via Outages wrote: We did the same. No response at this time and I haven't been able to find any further details about the issue. On Aug 15, 2022 10:40 AM, Bret Clark via Outages wrote: I opened a ticket with Intelepeer, but haven't heard anything back. I tried calling them of course there support DID isn't working right now. Anyone hear of what the issue might be? ________________________________ From: Outages on behalf of Shripal Daphtary via Outages Sent: Monday, August 15, 2022 1:26 PM To: Eduard Tieseler Cc: outages at outages.org Subject: Re: [outages] Intellepeer SIP outage? yes.. we're down from Ipeer. [https://ci3.googleusercontent.com/mail-sig/AIorK4z9pYm7j4k4gsrUnswgrW5gHXwM5DkYgIbACcbbLQXAfG3Wk5bem0ubGNIpFW7SYgm-S941JL50Z1nMckCgvM5FfGHA3ntra1oyO_rkVg] Shripal Daphtary shripal.daphtary at pressone.net D: 718.360.1222 C: 973.432.1440 F: 929.369.0900 On Mon, Aug 15, 2022 at 1:23 PM Eduard Tieseler via Outages > wrote: Anyone else having an outage on their SIP trunk with intellepeer? Our phones went down and their phones appear to be down as well. _______________________________________________ Outages mailing list Outages at outages.org https://puck.nether.net/mailman/listinfo/outages -------------- next part -------------- An HTML attachment was scrubbed... URL: From bclark at peregrinenetworks.net Mon Aug 15 14:14:16 2022 From: bclark at peregrinenetworks.net (Bret Clark) Date: Mon, 15 Aug 2022 18:14:16 +0000 Subject: [outages] Intellepeer SIP outage? In-Reply-To: References: Message-ID: Actually still seeing sporadic issues, was working for awhile, but now having trouble with calls again. ________________________________ From: Outages on behalf of Bret Clark via Outages Sent: Monday, August 15, 2022 1:52 PM To: Eduard Tieseler Cc: outages at outages.org Subject: Re: [outages] Intellepeer SIP outage? Yes, we are seeing calls again. Bret ________________________________ From: Eduard Tieseler Sent: Monday, August 15, 2022 1:49 PM To: Bret Clark Cc: outages at outages.org Subject: Re: [outages] Intellepeer SIP outage? We just came back up. On Aug 15, 2022 10:44 AM, Eduard Tieseler via Outages wrote: We did the same. No response at this time and I haven't been able to find any further details about the issue. On Aug 15, 2022 10:40 AM, Bret Clark via Outages wrote: I opened a ticket with Intelepeer, but haven't heard anything back. I tried calling them of course there support DID isn't working right now. Anyone hear of what the issue might be? ________________________________ From: Outages on behalf of Shripal Daphtary via Outages Sent: Monday, August 15, 2022 1:26 PM To: Eduard Tieseler Cc: outages at outages.org Subject: Re: [outages] Intellepeer SIP outage? yes.. we're down from Ipeer. [https://ci3.googleusercontent.com/mail-sig/AIorK4z9pYm7j4k4gsrUnswgrW5gHXwM5DkYgIbACcbbLQXAfG3Wk5bem0ubGNIpFW7SYgm-S941JL50Z1nMckCgvM5FfGHA3ntra1oyO_rkVg] Shripal Daphtary shripal.daphtary at pressone.net D: 718.360.1222 C: 973.432.1440 F: 929.369.0900 On Mon, Aug 15, 2022 at 1:23 PM Eduard Tieseler via Outages > wrote: Anyone else having an outage on their SIP trunk with intellepeer? Our phones went down and their phones appear to be down as well. _______________________________________________ Outages mailing list Outages at outages.org https://puck.nether.net/mailman/listinfo/outages -------------- next part -------------- An HTML attachment was scrubbed... URL: From johnpaulable at gmail.com Mon Aug 15 14:27:27 2022 From: johnpaulable at gmail.com (John-Paul Reed) Date: Mon, 15 Aug 2022 13:27:27 -0500 Subject: [outages] Outages Digest, Vol 171, Issue 4 In-Reply-To: References: Message-ID: Denver based trunk down. No inbound or outbound calls. Also cant get through to Intelepeer Support as their phones are down. Regards, John-Paul Reed On Mon, Aug 15, 2022, 12:50 PM wrote: > Send Outages mailing list submissions to > outages at outages.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://puck.nether.net/mailman/listinfo/outages > or, via email, send a message with subject or body 'help' to > outages-request at outages.org > > You can reach the person managing the list at > outages-owner at outages.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Outages digest..." > > > Today's Topics: > > 1. Re: Intellepeer SIP outage? (Shripal Daphtary) > 2. Re: Intellepeer SIP outage? (Bret Clark) > 3. Re: Intellepeer SIP outage? (Eduard Tieseler) > 4. Re: Intellepeer SIP outage? (Eduard Tieseler) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Mon, 15 Aug 2022 13:26:42 -0400 > From: Shripal Daphtary > To: Eduard Tieseler > Cc: outages at outages.org > Subject: Re: [outages] Intellepeer SIP outage? > Message-ID: > vQMnRC55WonYyCiHBbdNt7Q at mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > yes.. we're down from Ipeer. > > > > Shripal Daphtary > shripal.daphtary at pressone.net > D: 718.360.1222 > C: 973.432.1440 > F: 929.369.0900 > > > On Mon, Aug 15, 2022 at 1:23 PM Eduard Tieseler via Outages < > outages at outages.org> wrote: > > > Anyone else having an outage on their SIP trunk with intellepeer? > > Our phones went down and their phones appear to be down as well. > > _______________________________________________ > > Outages mailing list > > Outages at outages.org > > https://puck.nether.net/mailman/listinfo/outages > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > https://puck.nether.net/pipermail/outages/attachments/20220815/f53d4ff0/attachment-0001.htm > > > > ------------------------------ > > Message: 2 > Date: Mon, 15 Aug 2022 17:40:22 +0000 > From: Bret Clark > Cc: "outages at outages.org" > Subject: Re: [outages] Intellepeer SIP outage? > Message-ID: > < > BL1PR12MB52545FA41F0952456E91B8D2C6689 at BL1PR12MB5254.namprd12.prod.outlook.com > > > > Content-Type: text/plain; charset="us-ascii" > > I opened a ticket with Intelepeer, but haven't heard anything back. I > tried calling them of course there support DID isn't working right now. > > Anyone hear of what the issue might be? > > > ________________________________ > From: Outages on behalf of Shripal Daphtary > via Outages > Sent: Monday, August 15, 2022 1:26 PM > To: Eduard Tieseler > Cc: outages at outages.org > Subject: Re: [outages] Intellepeer SIP outage? > > yes.. we're down from Ipeer. > > [ > https://ci3.googleusercontent.com/mail-sig/AIorK4z9pYm7j4k4gsrUnswgrW5gHXwM5DkYgIbACcbbLQXAfG3Wk5bem0ubGNIpFW7SYgm-S941JL50Z1nMckCgvM5FfGHA3ntra1oyO_rkVg > ] > > Shripal Daphtary > shripal.daphtary at pressone.net > D: 718.360.1222 > C: 973.432.1440 > F: 929.369.0900 > > > On Mon, Aug 15, 2022 at 1:23 PM Eduard Tieseler via Outages < > outages at outages.org> wrote: > Anyone else having an outage on their SIP trunk with intellepeer? > Our phones went down and their phones appear to be down as well. > _______________________________________________ > Outages mailing list > Outages at outages.org > https://puck.nether.net/mailman/listinfo/outages > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > https://puck.nether.net/pipermail/outages/attachments/20220815/b03bda0f/attachment-0001.htm > > > > ------------------------------ > > Message: 3 > Date: Mon, 15 Aug 2022 10:44:11 -0700 > From: Eduard Tieseler > To: Bret Clark > Cc: outages at outages.org > Subject: Re: [outages] Intellepeer SIP outage? > Message-ID: > Content-Type: text/plain; charset="utf-8" > > An HTML attachment was scrubbed... > URL: < > https://puck.nether.net/pipermail/outages/attachments/20220815/0b5d83f3/attachment-0001.htm > > > -------------- next part -------------- > > We did the same. No response at this time and I haven't been able to find > any further details about the issue. > > > On Aug 15, 2022 10:40 AM, Bret Clark via Outages > wrote: > I opened a ticket with Intelepeer, but haven't heard anything back. I > tried calling them of course there support DID isn't working right now. > > Anyone hear of what the issue might be? > > > ________________________________ > From: Outages on behalf of Shripal Daphtary > via Outages > Sent: Monday, August 15, 2022 1:26 PM > To: Eduard Tieseler > Cc: outages at outages.org > Subject: Re: [outages] Intellepeer SIP outage? > > yes.. we're down from Ipeer. > > [ > https://ci3.googleusercontent.com/mail-sig/AIorK4z9pYm7j4k4gsrUnswgrW5gHXwM5DkYgIbACcbbLQXAfG3Wk5bem0ubGNIpFW7SYgm-S941JL50Z1nMckCgvM5FfGHA3ntra1oyO_rkVg > ] > > Shripal Daphtary > shripal.daphtary at pressone.net > D: 718.360.1222 > C: 973.432.1440 > F: 929.369.0900 > > > On Mon, Aug 15, 2022 at 1:23 PM Eduard Tieseler via Outages < > outages at outages.org> wrote: > Anyone else having an outage on their SIP trunk with intellepeer? > Our phones went down and their phones appear to be down as well. > _______________________________________________ > Outages mailing list > Outages at outages.org > https://puck.nether.net/mailman/listinfo/outages > > > ------------------------------ > > Message: 4 > Date: Mon, 15 Aug 2022 10:49:49 -0700 > From: Eduard Tieseler > To: Bret Clark > Cc: outages at outages.org > Subject: Re: [outages] Intellepeer SIP outage? > Message-ID: > Content-Type: text/plain; charset="utf-8" > > An HTML attachment was scrubbed... > URL: < > https://puck.nether.net/pipermail/outages/attachments/20220815/37744c63/attachment.htm > > > -------------- next part -------------- > > We just came back up.? > > > On Aug 15, 2022 10:44 AM, Eduard Tieseler via Outages > wrote: > > We did the same. No response at this time and I haven't been able to find > any further details about the issue. > > > On Aug 15, 2022 10:40 AM, Bret Clark via Outages > wrote: > I opened a ticket with Intelepeer, but haven't heard anything back. I > tried calling them of course there support DID isn't working right now. > > Anyone hear of what the issue might be? > > > ________________________________ > From: Outages on behalf of Shripal Daphtary > via Outages > Sent: Monday, August 15, 2022 1:26 PM > To: Eduard Tieseler > Cc: outages at outages.org > Subject: Re: [outages] Intellepeer SIP outage? > > yes.. we're down from Ipeer. > > [ > https://ci3.googleusercontent.com/mail-sig/AIorK4z9pYm7j4k4gsrUnswgrW5gHXwM5DkYgIbACcbbLQXAfG3Wk5bem0ubGNIpFW7SYgm-S941JL50Z1nMckCgvM5FfGHA3ntra1oyO_rkVg > ] > > Shripal Daphtary > shripal.daphtary at pressone.net > D: 718.360.1222 > C: 973.432.1440 > F: 929.369.0900 > > > On Mon, Aug 15, 2022 at 1:23 PM Eduard Tieseler via Outages < > outages at outages.org> wrote: > Anyone else having an outage on their SIP trunk with intellepeer? > Our phones went down and their phones appear to be down as well. > _______________________________________________ > Outages mailing list > Outages at outages.org > https://puck.nether.net/mailman/listinfo/outages > > > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > Outages mailing list > Outages at outages.org > https://puck.nether.net/mailman/listinfo/outages > > > ------------------------------ > > End of Outages Digest, Vol 171, Issue 4 > *************************************** > -------------- next part -------------- An HTML attachment was scrubbed... URL: From craigslist4md at gmail.com Mon Aug 15 14:46:39 2022 From: craigslist4md at gmail.com (Mike) Date: Mon, 15 Aug 2022 14:46:39 -0400 Subject: [outages] Intellepeer SIP outage? In-Reply-To: References: Message-ID: Found a status page for them - http://statuspage.intelepeer.com/ On Mon, Aug 15, 2022 at 2:15 PM Bret Clark via Outages wrote: > Actually still seeing sporadic issues, was working for awhile, but now > having trouble with calls again. > > ------------------------------ > *From:* Outages on behalf of Bret Clark via > Outages > *Sent:* Monday, August 15, 2022 1:52 PM > *To:* Eduard Tieseler > *Cc:* outages at outages.org > *Subject:* Re: [outages] Intellepeer SIP outage? > > Yes, we are seeing calls again. > Bret > ------------------------------ > *From:* Eduard Tieseler > *Sent:* Monday, August 15, 2022 1:49 PM > *To:* Bret Clark > *Cc:* outages at outages.org > *Subject:* Re: [outages] Intellepeer SIP outage? > > We just came back up. > > On Aug 15, 2022 10:44 AM, Eduard Tieseler via Outages > wrote: > We did the same. No response at this time and I haven't been able to find > any further details about the issue. > > On Aug 15, 2022 10:40 AM, Bret Clark via Outages > wrote: > I opened a ticket with Intelepeer, but haven't heard anything back. I > tried calling them of course there support DID isn't working right now. > > Anyone hear of what the issue might be? > > > ------------------------------ > *From:* Outages on behalf of Shripal > Daphtary via Outages > *Sent:* Monday, August 15, 2022 1:26 PM > *To:* Eduard Tieseler > *Cc:* outages at outages.org > *Subject:* Re: [outages] Intellepeer SIP outage? > > yes.. we're down from Ipeer. > > > > Shripal Daphtary > shripal.daphtary at pressone.net > D: 718.360.1222 > C: 973.432.1440 > F: 929.369.0900 > > > On Mon, Aug 15, 2022 at 1:23 PM Eduard Tieseler via Outages < > outages at outages.org> wrote: > > Anyone else having an outage on their SIP trunk with intellepeer? > Our phones went down and their phones appear to be down as well. > _______________________________________________ > Outages mailing list > Outages at outages.org > https://puck.nether.net/mailman/listinfo/outages > > _______________________________________________ > Outages mailing list > Outages at outages.org > https://puck.nether.net/mailman/listinfo/outages > -------------- next part -------------- An HTML attachment was scrubbed... URL: From adam.gregory at amnetsystems.com Mon Aug 15 15:28:42 2022 From: adam.gregory at amnetsystems.com (Adam Gregory) Date: Mon, 15 Aug 2022 19:28:42 +0000 Subject: [outages] Intellepeer SIP outage? In-Reply-To: References: Message-ID: Unfortunately, IntelePeer is still running in an impaired stated. We are restoring our routing engine, but we still have impairment on some of the nodes as we are seeing conflicts with external services (external feeds attributed to Stir Shaken affective stability) which we are trying to disassociate. . As of this juncture, we expect that we will be running in an impaired state until 5 pm ET. More to come. From: Outages On Behalf Of Bret Clark via Outages Sent: Monday, August 15, 2022 2:14 PM To: Eduard Tieseler Cc: outages at outages.org Subject: Re: [outages] Intellepeer SIP outage? Actually still seeing sporadic issues, was working for awhile, but now having trouble with calls again. ________________________________ From: Outages > on behalf of Bret Clark via Outages > Sent: Monday, August 15, 2022 1:52 PM To: Eduard Tieseler > Cc: outages at outages.org > Subject: Re: [outages] Intellepeer SIP outage? Yes, we are seeing calls again. Bret ________________________________ From: Eduard Tieseler > Sent: Monday, August 15, 2022 1:49 PM To: Bret Clark > Cc: outages at outages.org > Subject: Re: [outages] Intellepeer SIP outage? We just came back up. On Aug 15, 2022 10:44 AM, Eduard Tieseler via Outages > wrote: We did the same. No response at this time and I haven't been able to find any further details about the issue. On Aug 15, 2022 10:40 AM, Bret Clark via Outages > wrote: I opened a ticket with Intelepeer, but haven't heard anything back. I tried calling them of course there support DID isn't working right now. Anyone hear of what the issue might be? ________________________________ From: Outages > on behalf of Shripal Daphtary via Outages > Sent: Monday, August 15, 2022 1:26 PM To: Eduard Tieseler > Cc: outages at outages.org > Subject: Re: [outages] Intellepeer SIP outage? yes.. we're down from Ipeer. [https://ci3.googleusercontent.com/mail-sig/AIorK4z9pYm7j4k4gsrUnswgrW5gHXwM5DkYgIbACcbbLQXAfG3Wk5bem0ubGNIpFW7SYgm-S941JL50Z1nMckCgvM5FfGHA3ntra1oyO_rkVg] Shripal Daphtary shripal.daphtary at pressone.net D: 718.360.1222 C: 973.432.1440 F: 929.369.0900 On Mon, Aug 15, 2022 at 1:23 PM Eduard Tieseler via Outages > wrote: Anyone else having an outage on their SIP trunk with intellepeer? Our phones went down and their phones appear to be down as well. _______________________________________________ Outages mailing list Outages at outages.org https://puck.nether.net/mailman/listinfo/outages -------------- next part -------------- An HTML attachment was scrubbed... URL: From cody at codywilson.co Thu Aug 18 12:09:30 2022 From: cody at codywilson.co (Cody Wilson) Date: Thu, 18 Aug 2022 12:09:30 -0400 Subject: [outages] TCP errors observed in AWS US-West-2 Message-ID: I'm noting a significant increase in TCP errors across multiple accounts in AWS us-west-2. Curious if anyone else with accounts are noting this too, or if it's localized to some hardware we happen to be on.? -------------- next part -------------- An HTML attachment was scrubbed... URL: From ktamas at ktamas.com Thu Aug 18 12:11:36 2022 From: ktamas at ktamas.com (=?UTF-8?B?VGFtw6FzIEvDoWTDoXIgKEtUYW1hcyk=?=) Date: Thu, 18 Aug 2022 17:11:36 +0100 Subject: [outages] TCP errors observed in AWS US-West-2 In-Reply-To: References: Message-ID: I had some users complain about weirdly slow queries in RDS and connection issues so... maybe? On Thu, Aug 18, 2022 at 5:10 PM Cody Wilson via Outages wrote: > I'm noting a significant increase in TCP errors across multiple accounts > in AWS us-west-2. Curious if anyone else with accounts are noting this too, > or if it's localized to some hardware we happen to be on. > _______________________________________________ > Outages mailing list > Outages at outages.org > https://puck.nether.net/mailman/listinfo/outages > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.wright at commnetbroadband.com Thu Aug 18 12:13:49 2022 From: chris.wright at commnetbroadband.com (Chris Wright) Date: Thu, 18 Aug 2022 16:13:49 +0000 Subject: [outages] TCP errors observed in AWS US-West-2 In-Reply-To: References: Message-ID: AWS Health Dashboard looks clean. Is there a common carrier between your hosts and AWS? https://health.aws.amazon.com/health/status Chris Wright | Network Data Operations Engineer Network Operations and Integrations Commnet Broadband E: chris.wright at commnetbroadband.com M: 423-584-9578 [cid:image001.jpg at 01D8B2FB.F9C19B20] From: Outages On Behalf Of Cody Wilson via Outages Sent: Thursday, August 18, 2022 12:10 PM To: outages.org Subject: [outages] TCP errors observed in AWS US-West-2 I'm noting a significant increase in TCP errors across multiple accounts in AWS us-west-2. Curious if anyone else with accounts are noting this too, or if it's localized to some hardware we happen to be on. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 1894 bytes Desc: image001.jpg URL: From cody at codywilson.co Thu Aug 18 12:25:39 2022 From: cody at codywilson.co (Cody Wilson) Date: Thu, 18 Aug 2022 12:25:39 -0400 Subject: [outages] TCP errors observed in AWS US-West-2 In-Reply-To: Message-ID: While we can see and replicate these issues externally, we're also seeing it intra-AWS networks, so it's certainly something within us- west-2 that's amiss here. Moreso looking for any corroboration with anyone else's issues at this point. Thanks Chris! On August 18, 2022, Chris Wright wrote: > AWS Health Dashboard looks clean. Is there a common carrier between > your hosts and AWS? > > ? > > > > ? > > Chris Wright? | Network Data Operations Engineer > > Network Operations and Integrations > > Commnet Broadband > > E: chris.wright at commnetbroadband.com > M: 423-584-9578 > > > > > ? > > From: Outages On Behalf Of Cody Wilson > via Outages > Sent: Thursday, August 18, 2022 12:10 PM > To: outages.org > Subject: [outages] TCP errors observed in AWS US-West-2 > > ? > > I'm noting a significant increase in TCP errors across multiple > accounts in AWS us-west-2. Curious if anyone else with accounts are > noting this too, or if it's localized to some hardware we happen to be > on.? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From me at grg.pw Thu Aug 18 13:05:10 2022 From: me at grg.pw (=?UTF-8?Q?Giorgio_Bonfiglio?=) Date: Thu, 18 Aug 2022 17:05:10 +0000 Subject: [outages] TCP errors observed in AWS US-West-2 In-Reply-To: References: <8B983F81-BF81-4B81-BADB-18B942B70D67@grg.pw> Message-ID: <01020182b1ea7c69-e5bb5c7f-687d-4512-8020-5f387d9a8738-000000@eu-west-1.amazonses.com> Hey Cody, can you please throw what you have in a support case and share its id with me? If you don?t have a support plan then some details direct to my Amazon email (bonfigg @ amazon com) would be appreciated too but I might not be able to respond/share findings directly (privacy data security etc etc). Thanks! Giorgio On 18 Aug 2022, at 17:25, Cody Wilson via Outages > wrote: While we can see and replicate these issues externally, we're also seeing it intra-AWS networks, so it's certainly something within us-west-2 that's amiss here. Moreso looking for any corroboration with anyone else's issues at this point. Thanks Chris! On August 18, 2022, Chris Wright > wrote: AWS Health Dashboard looks clean. Is there a common carrier between your hosts and AWS? ? https://health.aws.amazon.com/health/status ? Chris Wright??| Network Data Operations Engineer Network Operations and Integrations Commnet Broadband E:?chris.wright at commnetbroadband.com M:?423-584-9578 ? From:?Outages >?On Behalf Of?Cody Wilson via Outages Sent:?Thursday, August 18, 2022 12:10 PM To:?outages.org > Subject:?[outages] TCP errors observed in AWS US-West-2 ? I'm noting a significant increase in TCP errors across multiple accounts in AWS us-west-2. Curious if anyone else with accounts are noting this too, or if it's localized to some hardware we happen to be on.? _______________________________________________ Outages mailing list Outages at outages.org https://puck.nether.net/mailman/listinfo/outages -------------- next part -------------- An HTML attachment was scrubbed... URL: From cody at codywilson.co Thu Aug 18 14:37:10 2022 From: cody at codywilson.co (Cody Wilson) Date: Thu, 18 Aug 2022 14:37:10 -0400 Subject: [outages] TCP errors observed in AWS US-West-2 In-Reply-To: <01020182b1ea7c69-e5bb5c7f-687d-4512-8020-5f387d9a8738-000000@eu-west-1.amazonses.com> Message-ID: <4070027cb1aa09b74bb9ea8978f713f772afbba1@hey.com> Thanks Giorgio! Sending case ID directly to you via your Amazon address.? On August 18, 2022, Giorgio Bonfiglio wrote: > Hey Cody, can you please throw what you have in a support case and > share its id with me? If you don?t have a support plan then some > details direct to my Amazon email (bonfigg @ amazon com) would be > appreciated too but I might not be able to respond/share findings > directly (privacy data security etc etc). > > Thanks! > Giorgio > > > On 18 Aug 2022, at 17:25, Cody Wilson via Outages > > > wrote: > > > > While we can see and replicate these issues externally, we're also > > seeing it intra-AWS networks, so it's certainly something within us- > > west-2 that's amiss here. Moreso looking for any corroboration with > > anyone else's issues at this point. Thanks Chris! > > > > On August 18, 2022, Chris Wright > > wrote: > > > AWS Health Dashboard looks clean. Is there a common carrier > > > between your hosts and AWS? > > > ? > > > > > > > > > ? > > > > > > Chris Wright??| Network Data Operations Engineer > > > Network Operations and Integrations > > > Commnet Broadband > > > E:?chris.wright at commnetbroadband.com > > > > > > M:?423-584-9578 > > > > > > > > > > > > ? > > > > > > From:?Outages > > bounces at outages.org>>?On Behalf Of?Cody Wilson via Outages > > > Sent:?Thursday, August 18, 2022 12:10 PM > > > To:?outages.org > > > > > > Subject:?[outages] TCP errors observed in AWS US-West-2 > > > ? > > > > > > I'm noting a significant increase in TCP errors across multiple > > > accounts in AWS us-west-2. Curious if anyone else with accounts > > > are noting this too, or if it's localized to some hardware we > > > happen to be on.? > > > > _______________________________________________ > > Outages mailing list > > Outages at outages.org > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeffreybelcher at gmail.com Sat Aug 20 16:26:37 2022 From: jeffreybelcher at gmail.com (Jeff Belcher) Date: Sat, 20 Aug 2022 16:26:37 -0400 Subject: [outages] TCP errors observed in AWS US-West-2 In-Reply-To: <4070027cb1aa09b74bb9ea8978f713f772afbba1@hey.com> References: <01020182b1ea7c69-e5bb5c7f-687d-4512-8020-5f387d9a8738-000000@eu-west-1.amazonses.com> <4070027cb1aa09b74bb9ea8978f713f772afbba1@hey.com> Message-ID: I believe this was on the right track, as we are still seeing issues. I believe that it's specifically Frontier (at least in FL) that is having issues contacting AWS resources. We use a loyalty program called Astro that is on AWS and our point of sale (non AWS) is still timing out when the API call is being made. Giorgio/Cody, did you all come up with a resolution? Would love a solution, as our retail stores are doing most of their processes manually right now to get around this. -Jeff On Thu, Aug 18, 2022 at 2:38 PM Cody Wilson via Outages wrote: > Thanks Giorgio! Sending case ID directly to you via your Amazon address. > > On August 18, 2022, Giorgio Bonfiglio wrote: > > Hey Cody, can you please throw what you have in a support case and share > its id with me? If you don?t have a support plan then some details direct > to my Amazon email (bonfigg @ amazon com) would be appreciated too but I > might not be able to respond/share findings directly (privacy data security > etc etc). > > Thanks! > Giorgio > > On 18 Aug 2022, at 17:25, Cody Wilson via Outages > wrote: > > While we can see and replicate these issues externally, we're also seeing > it intra-AWS networks, so it's certainly something within us-west-2 that's > amiss here. Moreso looking for any corroboration with anyone else's issues > at this point. Thanks Chris! > > On August 18, 2022, Chris Wright > wrote: > > AWS Health Dashboard looks clean. Is there a common carrier between your > hosts and AWS? > > > https://health.aws.amazon.com/health/status > > > *Chris Wright* | Network Data Operations Engineer > Network Operations and Integrations > Commnet Broadband > *E:* chris.wright at commnetbroadband.com > *M: *423-584-9578 > > > > > *From:* Outages *On Behalf Of *Cody Wilson > via Outages > *Sent:* Thursday, August 18, 2022 12:10 PM > *To:* outages.org > *Subject:* [outages] TCP errors observed in AWS US-West-2 > > > I'm noting a significant increase in TCP errors across multiple accounts > in AWS us-west-2. Curious if anyone else with accounts are noting this too, > or if it's localized to some hardware we happen to be on. > > _______________________________________________ > Outages mailing list > Outages at outages.org > https://puck.nether.net/mailman/listinfo/outages > > > _______________________________________________ > Outages mailing list > Outages at outages.org > https://puck.nether.net/mailman/listinfo/outages > -------------- next part -------------- An HTML attachment was scrubbed... URL: From me at grg.pw Sun Aug 21 11:01:36 2022 From: me at grg.pw (=?UTF-8?Q?Giorgio_Bonfiglio?=) Date: Sun, 21 Aug 2022 15:01:36 +0000 Subject: [outages] TCP errors observed in AWS US-West-2 In-Reply-To: References: Message-ID: <01020182c0ec6f63-5e8d3184-97de-4b11-b282-1905f7b24059-000000@eu-west-1.amazonses.com> Not really yet, two more people have reached out but symptoms and timing are completely off so I?m wondering whether we are looking at a single event or multiple customer specific details. Can you share with me some of your source IPs please and the endpoints (better if they are already resolved to IPs) timing out? I?ll try to narrow down to this specific access network and see if something stands out. Thanks Giorgio On 20 Aug 2022, at 22:26, Jeff Belcher wrote: ? I believe this was on the right track, as we are still seeing issues. I believe that it's specifically Frontier (at least in FL) that is having issues contacting AWS resources. We use a loyalty program called Astro that is on AWS and our point of sale (non AWS) is still timing out when the API call is being made. Giorgio/Cody, did you all come up with a resolution? Would love a solution, as our retail stores are doing most of their processes manually right now to get around this. -Jeff On Thu, Aug 18, 2022 at 2:38 PM Cody Wilson via Outages > wrote: Thanks Giorgio! Sending case ID directly to you via your Amazon address.? On August 18, 2022, Giorgio Bonfiglio > wrote: Hey Cody, can you please throw what you have in a support case and share its id with me? If you don?t have a support plan then some details direct to my Amazon email (bonfigg @ amazon com) would be appreciated too but I might not be able to respond/share findings directly (privacy data security etc etc). Thanks! Giorgio On 18 Aug 2022, at 17:25, Cody Wilson via Outages > wrote: While we can see and replicate these issues externally, we're also seeing it intra-AWS networks, so it's certainly something within us-west-2 that's amiss here. Moreso looking for any corroboration with anyone else's issues at this point. Thanks Chris! On August 18, 2022, Chris Wright > wrote: AWS Health Dashboard looks clean. Is there a common carrier between your hosts and AWS? ? https://health.aws.amazon.com/health/status ? Chris Wright??| Network Data Operations Engineer Network Operations and Integrations Commnet Broadband E:?chris.wright at commnetbroadband.com M:?423-584-9578 ? From:?Outages >?On Behalf Of?Cody Wilson via Outages Sent:?Thursday, August 18, 2022 12:10 PM To:?outages.org > Subject:?[outages] TCP errors observed in AWS US-West-2 ? I'm noting a significant increase in TCP errors across multiple accounts in AWS us-west-2. Curious if anyone else with accounts are noting this too, or if it's localized to some hardware we happen to be on.? _______________________________________________ Outages mailing list Outages at outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages at outages.org https://puck.nether.net/mailman/listinfo/outages -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeffreybelcher at gmail.com Sun Aug 21 11:14:38 2022 From: jeffreybelcher at gmail.com (Jeff Belcher) Date: Sun, 21 Aug 2022 11:14:38 -0400 Subject: [outages] TCP errors observed in AWS US-West-2 In-Reply-To: <01020182c0ec6f63-5e8d3184-97de-4b11-b282-1905f7b24059-000000@eu-west-1.amazonses.com> References: <01020182c0ec6f63-5e8d3184-97de-4b11-b282-1905f7b24059-000000@eu-west-1.amazonses.com> Message-ID: Absolutely. I can also do live testing, if that would help at all. On Sun, Aug 21, 2022, 11:01 AM Giorgio Bonfiglio wrote: > Not really yet, two more people have reached out but symptoms and timing > are completely off so I?m wondering whether we are looking at a single > event or multiple customer specific details. > > Can you share with me some of your source IPs please and the endpoints > (better if they are already resolved to IPs) timing out? I?ll try to narrow > down to this specific access network and see if something stands out. > > Thanks > Giorgio > > On 20 Aug 2022, at 22:26, Jeff Belcher wrote: > > ? > I believe this was on the right track, as we are still seeing issues. I > believe that it's specifically Frontier (at least in FL) that is having > issues contacting AWS resources. We use a loyalty program called Astro that > is on AWS and our point of sale (non AWS) is still timing out when the API > call is being made. > > Giorgio/Cody, did you all come up with a resolution? Would love a > solution, as our retail stores are doing most of their processes manually > right now to get around this. > > -Jeff > > > On Thu, Aug 18, 2022 at 2:38 PM Cody Wilson via Outages < > outages at outages.org> wrote: > >> Thanks Giorgio! Sending case ID directly to you via your Amazon address. >> >> On August 18, 2022, Giorgio Bonfiglio wrote: >> >> Hey Cody, can you please throw what you have in a support case and share >> its id with me? If you don?t have a support plan then some details direct >> to my Amazon email (bonfigg @ amazon com) would be appreciated too but I >> might not be able to respond/share findings directly (privacy data security >> etc etc). >> >> Thanks! >> Giorgio >> >> On 18 Aug 2022, at 17:25, Cody Wilson via Outages >> wrote: >> >> While we can see and replicate these issues externally, we're also seeing >> it intra-AWS networks, so it's certainly something within us-west-2 that's >> amiss here. Moreso looking for any corroboration with anyone else's issues >> at this point. Thanks Chris! >> >> On August 18, 2022, Chris Wright >> wrote: >> >> AWS Health Dashboard looks clean. Is there a common carrier between your >> hosts and AWS? >> >> >> https://health.aws.amazon.com/health/status >> >> >> *Chris Wright* | Network Data Operations Engineer >> Network Operations and Integrations >> Commnet Broadband >> *E:* chris.wright at commnetbroadband.com >> *M: *423-584-9578 >> >> >> >> >> *From:* Outages *On Behalf Of *Cody Wilson >> via Outages >> *Sent:* Thursday, August 18, 2022 12:10 PM >> *To:* outages.org >> *Subject:* [outages] TCP errors observed in AWS US-West-2 >> >> >> I'm noting a significant increase in TCP errors across multiple accounts >> in AWS us-west-2. Curious if anyone else with accounts are noting this too, >> or if it's localized to some hardware we happen to be on. >> >> _______________________________________________ >> Outages mailing list >> Outages at outages.org >> https://puck.nether.net/mailman/listinfo/outages >> >> >> _______________________________________________ >> Outages mailing list >> Outages at outages.org >> https://puck.nether.net/mailman/listinfo/outages >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Michael.Williams at glexia.com Sun Aug 21 11:30:41 2022 From: Michael.Williams at glexia.com (Michael B. Williams) Date: Mon, 22 Aug 2022 01:30:41 +1000 Subject: [outages] TCP errors observed in AWS US-West-2 In-Reply-To: References: <01020182c0ec6f63-5e8d3184-97de-4b11-b282-1905f7b24059-000000@eu-west-1.amazonses.com> Message-ID: We?re also seeing issues in AWS US-West-2. Any thoughts or things we should try? On Mon, Aug 22, 2022 at 01:16 Jeff Belcher via Outages wrote: > Absolutely. I can also do live testing, if that would help at all. > > On Sun, Aug 21, 2022, 11:01 AM Giorgio Bonfiglio wrote: > >> Not really yet, two more people have reached out but symptoms and timing >> are completely off so I?m wondering whether we are looking at a single >> event or multiple customer specific details. >> >> Can you share with me some of your source IPs please and the endpoints >> (better if they are already resolved to IPs) timing out? I?ll try to narrow >> down to this specific access network and see if something stands out. >> >> Thanks >> Giorgio >> >> On 20 Aug 2022, at 22:26, Jeff Belcher wrote: >> >> ? >> I believe this was on the right track, as we are still seeing issues. I >> believe that it's specifically Frontier (at least in FL) that is having >> issues contacting AWS resources. We use a loyalty program called Astro that >> is on AWS and our point of sale (non AWS) is still timing out when the API >> call is being made. >> >> Giorgio/Cody, did you all come up with a resolution? Would love a >> solution, as our retail stores are doing most of their processes manually >> right now to get around this. >> >> -Jeff >> >> >> On Thu, Aug 18, 2022 at 2:38 PM Cody Wilson via Outages < >> outages at outages.org> wrote: >> >>> Thanks Giorgio! Sending case ID directly to you via your Amazon address. >>> >>> On August 18, 2022, Giorgio Bonfiglio wrote: >>> >>> Hey Cody, can you please throw what you have in a support case and share >>> its id with me? If you don?t have a support plan then some details direct >>> to my Amazon email (bonfigg @ amazon com) would be appreciated too but I >>> might not be able to respond/share findings directly (privacy data security >>> etc etc). >>> >>> Thanks! >>> Giorgio >>> >>> On 18 Aug 2022, at 17:25, Cody Wilson via Outages >>> wrote: >>> >>> While we can see and replicate these issues externally, we're also >>> seeing it intra-AWS networks, so it's certainly something within us-west-2 >>> that's amiss here. Moreso looking for any corroboration with anyone else's >>> issues at this point. Thanks Chris! >>> >>> On August 18, 2022, Chris Wright >>> wrote: >>> >>> AWS Health Dashboard looks clean. Is there a common carrier between your >>> hosts and AWS? >>> >>> >>> https://health.aws.amazon.com/health/status >>> >>> >>> *Chris Wright* | Network Data Operations Engineer >>> Network Operations and Integrations >>> Commnet Broadband >>> *E:* chris.wright at commnetbroadband.com >>> *M: *423-584-9578 >>> >>> >>> >>> >>> *From:* Outages *On Behalf Of *Cody >>> Wilson via Outages >>> *Sent:* Thursday, August 18, 2022 12:10 PM >>> *To:* outages.org >>> *Subject:* [outages] TCP errors observed in AWS US-West-2 >>> >>> >>> I'm noting a significant increase in TCP errors across multiple accounts >>> in AWS us-west-2. Curious if anyone else with accounts are noting this too, >>> or if it's localized to some hardware we happen to be on. >>> >>> _______________________________________________ >>> Outages mailing list >>> Outages at outages.org >>> https://puck.nether.net/mailman/listinfo/outages >>> >>> >>> _______________________________________________ >>> Outages mailing list >>> Outages at outages.org >>> https://puck.nether.net/mailman/listinfo/outages >>> >> _______________________________________________ > Outages mailing list > Outages at outages.org > https://puck.nether.net/mailman/listinfo/outages > -- Sent from Gmail Mobile -------------- next part -------------- An HTML attachment was scrubbed... URL: From me at grg.pw Sun Aug 21 11:55:40 2022 From: me at grg.pw (=?UTF-8?Q?Giorgio_Bonfiglio?=) Date: Sun, 21 Aug 2022 15:55:40 +0000 Subject: [outages] TCP errors observed in AWS US-West-2 In-Reply-To: References: Message-ID: <01020182c11deef3-3c39bd1b-f83b-4e8e-8dad-753cac2e5d61-000000@eu-west-1.amazonses.com> Can you please share some src/dst IPs, timestamps and description of the issues? On 21 Aug 2022, at 17:30, Michael B. Williams wrote: ? We?re also seeing issues in AWS US-West-2. Any thoughts or things we should try? On Mon, Aug 22, 2022 at 01:16 Jeff Belcher via Outages > wrote: Absolutely. I can also do live testing, if that would help at all. On Sun, Aug 21, 2022, 11:01 AM Giorgio Bonfiglio > wrote: Not really yet, two more people have reached out but symptoms and timing are completely off so I?m wondering whether we are looking at a single event or multiple customer specific details. Can you share with me some of your source IPs please and the endpoints (better if they are already resolved to IPs) timing out? I?ll try to narrow down to this specific access network and see if something stands out. Thanks Giorgio On 20 Aug 2022, at 22:26, Jeff Belcher > wrote: ? I believe this was on the right track, as we are still seeing issues. I believe that it's specifically Frontier (at least in FL) that is having issues contacting AWS resources. We use a loyalty program called Astro that is on AWS and our point of sale (non AWS) is still timing out when the API call is being made. Giorgio/Cody, did you all come up with a resolution? Would love a solution, as our retail stores are doing most of their processes manually right now to get around this. -Jeff On Thu, Aug 18, 2022 at 2:38 PM Cody Wilson via Outages > wrote: Thanks Giorgio! Sending case ID directly to you via your Amazon address.? On August 18, 2022, Giorgio Bonfiglio > wrote: Hey Cody, can you please throw what you have in a support case and share its id with me? If you don?t have a support plan then some details direct to my Amazon email (bonfigg @ amazon com) would be appreciated too but I might not be able to respond/share findings directly (privacy data security etc etc). Thanks! Giorgio On 18 Aug 2022, at 17:25, Cody Wilson via Outages > wrote: While we can see and replicate these issues externally, we're also seeing it intra-AWS networks, so it's certainly something within us-west-2 that's amiss here. Moreso looking for any corroboration with anyone else's issues at this point. Thanks Chris! On August 18, 2022, Chris Wright > wrote: AWS Health Dashboard looks clean. Is there a common carrier between your hosts and AWS? ? https://health.aws.amazon.com/health/status ? Chris Wright??| Network Data Operations Engineer Network Operations and Integrations Commnet Broadband E:?chris.wright at commnetbroadband.com M:?423-584-9578 ? From:?Outages >?On Behalf Of?Cody Wilson via Outages Sent:?Thursday, August 18, 2022 12:10 PM To:?outages.org > Subject:?[outages] TCP errors observed in AWS US-West-2 ? I'm noting a significant increase in TCP errors across multiple accounts in AWS us-west-2. Curious if anyone else with accounts are noting this too, or if it's localized to some hardware we happen to be on.? _______________________________________________ Outages mailing list Outages at outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages at outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages at outages.org https://puck.nether.net/mailman/listinfo/outages -- Sent from Gmail Mobile -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlos at initiatel.com Tue Aug 30 14:59:37 2022 From: carlos at initiatel.com (Carlos Alvarez) Date: Tue, 30 Aug 2022 11:59:37 -0700 Subject: [outages] Call completion errors (503) from multiple carriers Message-ID: We are suddenly seeing a lot of this seemingly nationwide from Bandwidth and thinQ (which is a multi-carrier aggregator). -- Carlos Alvarez 602-368-6403 -------------- next part -------------- An HTML attachment was scrubbed... URL: From shawn.begarowicz at cit-4u.com Tue Aug 30 15:03:27 2022 From: shawn.begarowicz at cit-4u.com (Begarowicz, Shawn) Date: Tue, 30 Aug 2022 19:03:27 +0000 Subject: [outages] [EXTERNAL] Call completion errors (503) from multiple carriers In-Reply-To: References: Message-ID: Not sure if this is part of it Bandwidth Messaging Impairment with T-Mobile New incident: Identified We?ve identified an issue affecting a subset of messaging traffic with T-Mobile. Relevant teams are engaged and we?ll provide updates as they become available. Time posted Aug 30, 14:11 EDT Components affected [Partial Outage] United States Messaging Services - SM... [Partial Outage] United States Messaging Services - RE... View full incident details Thank You, Shawn Begarowicz (Technology/Service Manager) Complete Interactive Technologies 44265 Groesbeck Highway Clinton Township, MI 48036 586-439-2000 Ext. 120 | 586-439-2001 Fax 586-913-2420 Direct Tel | 586-203-4940 Direct Fax 586-883-4037 Cell shawn.begarowicz at cit-4u.com www.cit-4u.com For Technical Support: support at cit-4u.com Website: www.cit-4u.com [cid:image001.png at 01D8BC82.184C8CF0] Celebrating 30 Years of your VISION our SOLUTION The information contained in this message is confidential and, if addressed to a client or certain counsel, is subject to the attorney-client or work-product privilege, and is intended only for the use of the individual or entity named above. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please reply or notify us immediately by email or telephone, and destroy or erase the message. Please think twice before printing out this email! From: Outages On Behalf Of Carlos Alvarez via Outages Sent: Tuesday, August 30, 2022 3:00 PM To: Robert Blayzor via Outages Subject: [EXTERNAL] [outages] Call completion errors (503) from multiple carriers CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. We are suddenly seeing a lot of this seemingly nationwide from Bandwidth and thinQ (which is a multi-carrier aggregator). -- Carlos Alvarez 602-368-6403 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 21679 bytes Desc: image001.png URL: From carlos at initiatel.com Tue Aug 30 15:12:48 2022 From: carlos at initiatel.com (Carlos Alvarez) Date: Tue, 30 Aug 2022 12:12:48 -0700 Subject: [outages] [EXTERNAL] Call completion errors (503) from multiple carriers In-Reply-To: References: Message-ID: We've seen failures to destinations other than T-Mobile, such as TW Telecom and Comcast. On Tue, Aug 30, 2022 at 12:03 PM Begarowicz, Shawn < shawn.begarowicz at cit-4u.com> wrote: > Not sure if this is part of it > > > > Bandwidth > > > > > > Messaging Impairment with T-Mobile > > New incident: Identified > > We?ve identified an issue affecting a subset of messaging traffic with > T-Mobile. Relevant teams are engaged and we?ll provide updates as they > become available. > > Time posted > > Aug 30, 14:11 EDT > > Components affected > > > > [image: Partial Outage] > > United States Messaging Services - SM... > > > > [image: Partial Outage] > > United States Messaging Services - RE... > > > > View full incident details > > > > > > > Thank You, > Shawn Begarowicz (Technology/Service Manager) > Complete Interactive Technologies > 44265 Groesbeck Highway > Clinton Township, MI 48036 > 586-439-2000 Ext. 120 | 586-439-2001 Fax > 586-913-2420 Direct Tel | 586-203-4940 Direct Fax > 586-883-4037 Cell > shawn.begarowicz at cit-4u.com > www.cit-4u.com > For Technical Support: > support at cit-4u.com > Website: > www.cit-4u.com > > Celebrating 30 Years of your VISION our SOLUTION > > > > > > The information contained in this message is confidential and, if > addressed to a client or certain counsel, is subject to the attorney-client > or work-product privilege, and is intended only for the use of the > individual or entity named above. If the reader of this message is not the > intended recipient, you are hereby notified that any dissemination, > distribution or copying of this communication is strictly prohibited. If > you have received this communication in error, please reply or notify us > immediately by email or telephone, and destroy or erase the message. > > > > Please think twice before printing out this email! > > > > *From:* Outages *On Behalf Of *Carlos > Alvarez via Outages > *Sent:* Tuesday, August 30, 2022 3:00 PM > *To:* Robert Blayzor via Outages > *Subject:* [EXTERNAL] [outages] Call completion errors (503) from > multiple carriers > > > > CAUTION: This email originated from outside of the organization. Do not > click links or open attachments unless you recognize the sender and know > the content is safe. > > > > We are suddenly seeing a lot of this seemingly nationwide from Bandwidth > and thinQ (which is a multi-carrier aggregator). > > > > -- > > Carlos Alvarez > > 602-368-6403 > > > -- Carlos Alvarez 602-368-6403 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 21679 bytes Desc: not available URL: