<div dir="ltr">I work for a company that services VOIP phones in the Philly/South Jersey area. We saw a various ammount of Verizon FioS/IDE connections lose the ability to route to vendors on various places on the internet around this time. We failed the ones we could over to a back up connection and corrected issues. Suspected higher end routing change/issue. </div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Mar 11, 2015 at 12:00 PM, <span dir="ltr"><<a href="mailto:outages-request@outages.org" target="_blank">outages-request@outages.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Send Outages mailing list submissions to<br>
<a href="mailto:outages@outages.org">outages@outages.org</a><br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br>
or, via email, send a message with subject or body 'help' to<br>
<a href="mailto:outages-request@outages.org">outages-request@outages.org</a><br>
<br>
You can reach the person managing the list at<br>
<a href="mailto:outages-owner@outages.org">outages-owner@outages.org</a><br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of Outages digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. Verizon Northern NJ (James Cornman)<br>
2. Re: Verizon Northern NJ (Dan Langille)<br>
3. Issues reaching Verion NJ/NY (Dan Spataro)<br>
4. Re: Verizon Northern NJ (Jeremy Chadwick)<br>
5. Re: Verizon Northern NJ (joel jaeggli)<br>
6. Re: Verizon Northern NJ (Charles Sprickman)<br>
7. Re: Verizon Northern NJ (Jason Lixfeld)<br>
<br>
<br>
----------------------------------------------------------------------<br>
<br>
Message: 1<br>
Date: Tue, 10 Mar 2015 17:48:26 -0400<br>
From: James Cornman <<a href="mailto:james@atlanticmetro.net">james@atlanticmetro.net</a>><br>
To: "<a href="mailto:outages@outages.org">outages@outages.org</a>" <<a href="mailto:outages@outages.org">outages@outages.org</a>><br>
Subject: [outages] Verizon Northern NJ<br>
Message-ID: <-7282723577044655635@unknownmsgid><br>
Content-Type: text/plain; charset=UTF-8<br>
<br>
We've had a lot of customers that are having problems with Verizon<br>
FiOS and other IP services within the past 40 minutes. Mainly<br>
concentrated in the northern New Jersey area. Are there any other<br>
reports of this?<br>
<br>
-James<br>
<br>
<br>
------------------------------<br>
<br>
Message: 2<br>
Date: Tue, 10 Mar 2015 18:06:30 -0400<br>
From: Dan Langille <<a href="mailto:dan@langille.org">dan@langille.org</a>><br>
To: James Cornman <<a href="mailto:james@atlanticmetro.net">james@atlanticmetro.net</a>><br>
Cc: "<a href="mailto:outages@outages.org">outages@outages.org</a>" <<a href="mailto:outages@outages.org">outages@outages.org</a>><br>
Subject: Re: [outages] Verizon Northern NJ<br>
Message-ID: <<a href="mailto:C9F41142-CBCA-426C-A84B-3C2E65BAE0D4@langille.org">C9F41142-CBCA-426C-A84B-3C2E65BAE0D4@langille.org</a>><br>
Content-Type: text/plain; charset=utf-8<br>
<br>
<br>
> On Mar 10, 2015, at 5:48 PM, James Cornman via Outages <<a href="mailto:outages@outages.org">outages@outages.org</a>> wrote:<br>
><br>
> We've had a lot of customers that are having problems with Verizon<br>
> FiOS and other IP services within the past 40 minutes. Mainly<br>
> concentrated in the northern New Jersey area. Are there any other<br>
> reports of this?<br>
<br>
I'm a Verizon customer and from home between 5:16pm to about 5:41pm I had no connection to various services<br>
HE.net and various other places. I'm in Media, PA. All good now.<br>
<br>
?<br>
Dan Langille<br>
<a href="http://langille.org/" target="_blank">http://langille.org/</a><br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
------------------------------<br>
<br>
Message: 3<br>
Date: Tue, 10 Mar 2015 22:06:30 +0000<br>
From: Dan Spataro <<a href="mailto:dspataro@corp.nac.net">dspataro@corp.nac.net</a>><br>
To: "<a href="mailto:outages@outages.org">outages@outages.org</a>" <<a href="mailto:outages@outages.org">outages@outages.org</a>><br>
Subject: [outages] Issues reaching Verion NJ/NY<br>
Message-ID: <<a href="mailto:9c786d409dda4f7abebc5e27e801fff2@exch2013-1.hq.nac.net">9c786d409dda4f7abebc5e27e801fff2@exch2013-1.hq.nac.net</a>><br>
Content-Type: text/plain; charset="us-ascii"<br>
<br>
At about 5:06pm we completely lost connectivity to Verizon via Telia. We since routed Verizon through NTT and we are now seeing 70ms+ latency in Verizons network. One of our end users is reporting problems reaching twitter and other sites from their NJ FIOS. Anyone else seeing problems?<br>
<br>
<br>
4. <a href="http://ae-30.r06.nycmny01.us.bb.gin.ntt.net" target="_blank">ae-30.r06.nycmny01.us.bb.gin.ntt.net</a> 0.0% 15 3.0 2.4 1.9 3.0 0.3<br>
5. <a href="http://0.xe-7-1-0.BR3.NYC4.ALTER.NET" target="_blank">0.xe-7-1-0.BR3.NYC4.ALTER.NET</a> 0.0% 15 1.6 2.9 1.4 12.1 3.2<br>
<a href="http://xe-0.verizonbusiness.nycmny01.us.bb.gin.ntt.net" target="_blank">xe-0.verizonbusiness.nycmny01.us.bb.gin.ntt.net</a><br>
<a href="http://0.xe-11-3-0.BR3.NYC4.ALTER.NET" target="_blank">0.xe-11-3-0.BR3.NYC4.ALTER.NET</a><br>
<a href="http://0.xe-11-3-0.BR3.NYC4.ALTER.NET" target="_blank">0.xe-11-3-0.BR3.NYC4.ALTER.NET</a><br>
6. <a href="http://Bundle-Ether1100.NWRKNJ-LCR-22.verizon-gni.NET" target="_blank">Bundle-Ether1100.NWRKNJ-LCR-22.verizon-gni.NET</a> 0.0% 15 78.5 81.1 78.2 84.5 2.1<br>
<a href="http://Bundle-Ether1100.NWRKNJ-LCR-21.verizon-gni.NET" target="_blank">Bundle-Ether1100.NWRKNJ-LCR-21.verizon-gni.NET</a><br>
<a href="http://Bundle-Ether1100.NWRKNJ-LCR-21.verizon-gni.NET" target="_blank">Bundle-Ether1100.NWRKNJ-LCR-21.verizon-gni.NET</a><br>
7. ???<br>
8. <a href="http://pool-173-70-118-168.nwrknj.fios.verizon.net" target="_blank">pool-173-70-118-168.nwrknj.fios.verizon.net</a> 0.0% 14 89.9 87.4 82.3 90.8 2.5<br>
<br>
<br>
<br>
<br>
Daniel Spataro<br>
Net Access Corporation<br>
(v) 973-590-5144<br>
(e) <a href="mailto:dspataro@corp.nac.net">dspataro@corp.nac.net</a><br>
<br>
<br>
<br>
<br>
------------------------------<br>
<br>
Message: 4<br>
Date: Tue, 10 Mar 2015 15:46:56 -0700<br>
From: Jeremy Chadwick <<a href="mailto:jdc@koitsu.org">jdc@koitsu.org</a>><br>
To: James Cornman <<a href="mailto:james@atlanticmetro.net">james@atlanticmetro.net</a>><br>
Cc: "<a href="mailto:outages@outages.org">outages@outages.org</a>" <<a href="mailto:outages@outages.org">outages@outages.org</a>><br>
Subject: Re: [outages] Verizon Northern NJ<br>
Message-ID: <20150310224656.GA33954@icarus.home.lan><br>
Content-Type: text/plain; charset=us-ascii<br>
<br>
This just got posted (Email seems delayed but the web archive[1] seems<br>
fine): <a href="https://puck.nether.net/pipermail/outages/2015-March/007661.html" target="_blank">https://puck.nether.net/pipermail/outages/2015-March/007661.html</a><br>
<br>
A colleague of mine on Verizon FiOS in NY reported to me the exact same<br>
problems as being reported here. In other words: it wasn't just you.<br>
<br>
<br>
[1]: <a href="https://puck.nether.net/pipermail/outages/2015-March/thread.html" target="_blank">https://puck.nether.net/pipermail/outages/2015-March/thread.html</a><br>
<br>
--<br>
| Jeremy Chadwick <a href="mailto:jdc@koitsu.org">jdc@koitsu.org</a> |<br>
| UNIX Systems Administrator <a href="http://jdc.koitsu.org/" target="_blank">http://jdc.koitsu.org/</a> |<br>
| Making life hard for others since 1977. PGP 4BD6C0CB |<br>
<br>
On Tue, Mar 10, 2015 at 05:48:26PM -0400, James Cornman via Outages wrote:<br>
> We've had a lot of customers that are having problems with Verizon<br>
> FiOS and other IP services within the past 40 minutes. Mainly<br>
> concentrated in the northern New Jersey area. Are there any other<br>
> reports of this?<br>
><br>
> -James<br>
> _______________________________________________<br>
> Outages mailing list<br>
> <a href="mailto:Outages@outages.org">Outages@outages.org</a><br>
> <a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br>
<br>
<br>
------------------------------<br>
<br>
Message: 5<br>
Date: Tue, 10 Mar 2015 16:16:31 -0700<br>
From: joel jaeggli <<a href="mailto:joelja@bogus.com">joelja@bogus.com</a>><br>
To: James Cornman via Outages <<a href="mailto:outages@outages.org">outages@outages.org</a>><br>
Subject: Re: [outages] Verizon Northern NJ<br>
Message-ID: <<a href="mailto:54FF7B4F.9070605@bogus.com">54FF7B4F.9070605@bogus.com</a>><br>
Content-Type: text/plain; charset="windows-1252"<br>
<br>
On 3/10/15 2:48 PM, James Cornman via Outages wrote:<br>
> OS and other IP services within the past 40 minutes. Mainly<br>
> concentrated in the northern New Jersey area. Are there any other<br>
> reports of this?<br>
<br>
we have customer who can't reach us for there yeah.<br>
<br>
> -James<br>
<br>
<br>
-------------- next part --------------<br>
A non-text attachment was scrubbed...<br>
Name: signature.asc<br>
Type: application/pgp-signature<br>
Size: 243 bytes<br>
Desc: OpenPGP digital signature<br>
URL: <<a href="https://puck.nether.net/pipermail/outages/attachments/20150310/a6e1ef9d/attachment-0001.sig" target="_blank">https://puck.nether.net/pipermail/outages/attachments/20150310/a6e1ef9d/attachment-0001.sig</a>><br>
<br>
------------------------------<br>
<br>
Message: 6<br>
Date: Tue, 10 Mar 2015 20:00:24 -0400<br>
From: Charles Sprickman <<a href="mailto:spork@bway.net">spork@bway.net</a>><br>
To: joel jaeggli <<a href="mailto:joelja@bogus.com">joelja@bogus.com</a>><br>
Cc: James Cornman via Outages <<a href="mailto:outages@outages.org">outages@outages.org</a>><br>
Subject: Re: [outages] Verizon Northern NJ<br>
Message-ID: <<a href="mailto:D6D46EA5-6D92-470D-A188-5A4C1DFCC90D@bway.net">D6D46EA5-6D92-470D-A188-5A4C1DFCC90D@bway.net</a>><br>
Content-Type: text/plain; charset="windows-1252"<br>
<br>
>From Optonline in Morristown I?m seeing drops to a server off business FiOS outside Philly.<br>
<br>
Looks like there?s no intermediate between OOL and VZ either:<br>
<br>
1 10.240.176.233 (10.240.176.233) 8.719 ms 7.099 ms 7.766 ms<br>
2 67.59.225.220 (67.59.225.220) 8.379 ms 10.610 ms 7.925 ms<br>
3 <a href="http://rtr2-ge1-10.mhe.prnynj.cv.net" target="_blank">rtr2-ge1-10.mhe.prnynj.cv.net</a> (67.83.242.5) 9.210 ms<br>
<a href="http://ool-4353f20d.dyn.optonline.net" target="_blank">ool-4353f20d.dyn.optonline.net</a> (67.83.242.13) 9.598 ms 9.476 ms<br>
4 64.15.7.49 (64.15.7.49) 9.244 ms<br>
65.19.119.193 (65.19.119.193) 9.222 ms<br>
64.15.7.41 (64.15.7.41) 8.662 ms<br>
5 <a href="http://451be0d2.cst.lightpath.net" target="_blank">451be0d2.cst.lightpath.net</a> <a href="tel:%2865.19.120.210" value="+16519120210">(65.19.120.210</a>) 11.150 ms<br>
64.15.0.52 (64.15.0.52) 11.240 ms<br>
<a href="http://451be0d2.cst.lightpath.net" target="_blank">451be0d2.cst.lightpath.net</a> <a href="tel:%2865.19.120.210" value="+16519120210">(65.19.120.210</a>) 11.470 ms<br>
6 <a href="http://0.xe-2-3-0.GW13.NYC1.ALTER.NET" target="_blank">0.xe-2-3-0.GW13.NYC1.ALTER.NET</a> (152.179.220.57) 9.715 ms<br>
<a href="http://0.TenGigE0-0-0-6.GW8.EWR6.ALTER.NET" target="_blank">0.TenGigE0-0-0-6.GW8.EWR6.ALTER.NET</a> (152.179.61.85) 16.414 ms<br>
<a href="http://0.xe-2-3-0.GW13.NYC1.ALTER.NET" target="_blank">0.xe-2-3-0.GW13.NYC1.ALTER.NET</a> (152.179.220.57) 10.204 ms<br>
7 * <a href="http://B400.PHLAPA-LCR-21.verizon-gni.net" target="_blank">B400.PHLAPA-LCR-21.verizon-gni.net</a> (130.81.163.133) 21.193 ms<br>
<a href="http://P0-4-0-0.PHLAPA-LCR-22.verizon-gni.net" target="_blank">P0-4-0-0.PHLAPA-LCR-22.verizon-gni.net</a> (130.81.22.61) 19.623 ms<br>
8 * * *<br>
9 static-xxxx 25.310 ms 20.195 ms 24.948 ms<br>
<br>
VZ has been having some interesting issues the past few months. Maybe the last of the UUNet people finally left. :)<br>
<br>
Charles<br>
<br>
On Mar 10, 2015, at 7:16 PM, joel jaeggli via Outages <<a href="mailto:outages@outages.org">outages@outages.org</a>> wrote:<br>
<br>
> On 3/10/15 2:48 PM, James Cornman via Outages wrote:<br>
>> OS and other IP services within the past 40 minutes. Mainly<br>
>> concentrated in the northern New Jersey area. Are there any other<br>
>> reports of this?<br>
><br>
> we have customer who can't reach us for there yeah.<br>
><br>
>> -James<br>
><br>
><br>
> _______________________________________________<br>
> Outages mailing list<br>
> <a href="mailto:Outages@outages.org">Outages@outages.org</a><br>
> <a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br>
<br>
-------------- next part --------------<br>
A non-text attachment was scrubbed...<br>
Name: signature.asc<br>
Type: application/pgp-signature<br>
Size: 495 bytes<br>
Desc: Message signed with OpenPGP using GPGMail<br>
URL: <<a href="https://puck.nether.net/pipermail/outages/attachments/20150310/0c3b93e8/attachment-0001.sig" target="_blank">https://puck.nether.net/pipermail/outages/attachments/20150310/0c3b93e8/attachment-0001.sig</a>><br>
<br>
------------------------------<br>
<br>
Message: 7<br>
Date: Tue, 10 Mar 2015 21:02:19 -0400<br>
From: Jason Lixfeld <<a href="mailto:jason@lixfeld.ca">jason@lixfeld.ca</a>><br>
To: Charles Sprickman <<a href="mailto:spork@bway.net">spork@bway.net</a>><br>
Cc: James Cornman via Outages <<a href="mailto:outages@outages.org">outages@outages.org</a>><br>
Subject: Re: [outages] Verizon Northern NJ<br>
Message-ID: <<a href="mailto:DFE03362-9C32-4CD9-9D53-6B7D8B518D9B@lixfeld.ca">DFE03362-9C32-4CD9-9D53-6B7D8B518D9B@lixfeld.ca</a>><br>
Content-Type: text/plain; charset=windows-1252<br>
<br>
Has this cleared up for folks? I have one customer who reported issues reaching Verizon Toronto via Level 3 around the time when it was first reported here (just after 5pm EDT). I?m waiting for confirmation that the issue has cleared for them; I don?t see it myself from the same ASN as the customer who reported the problem, but I figured I?d ask anyway.<br>
<br>
> On Mar 10, 2015, at 8:00 PM, Charles Sprickman via Outages <<a href="mailto:outages@outages.org">outages@outages.org</a>> wrote:<br>
><br>
> From Optonline in Morristown I?m seeing drops to a server off business FiOS outside Philly.<br>
><br>
> Looks like there?s no intermediate between OOL and VZ either:<br>
><br>
> 1 10.240.176.233 (10.240.176.233) 8.719 ms 7.099 ms 7.766 ms<br>
> 2 67.59.225.220 (67.59.225.220) 8.379 ms 10.610 ms 7.925 ms<br>
> 3 <a href="http://rtr2-ge1-10.mhe.prnynj.cv.net" target="_blank">rtr2-ge1-10.mhe.prnynj.cv.net</a> (67.83.242.5) 9.210 ms<br>
> <a href="http://ool-4353f20d.dyn.optonline.net" target="_blank">ool-4353f20d.dyn.optonline.net</a> (67.83.242.13) 9.598 ms 9.476 ms<br>
> 4 64.15.7.49 (64.15.7.49) 9.244 ms<br>
> 65.19.119.193 (65.19.119.193) 9.222 ms<br>
> 64.15.7.41 (64.15.7.41) 8.662 ms<br>
> 5 <a href="http://451be0d2.cst.lightpath.net" target="_blank">451be0d2.cst.lightpath.net</a> <a href="tel:%2865.19.120.210" value="+16519120210">(65.19.120.210</a>) 11.150 ms<br>
> 64.15.0.52 (64.15.0.52) 11.240 ms<br>
> <a href="http://451be0d2.cst.lightpath.net" target="_blank">451be0d2.cst.lightpath.net</a> <a href="tel:%2865.19.120.210" value="+16519120210">(65.19.120.210</a>) 11.470 ms<br>
> 6 <a href="http://0.xe-2-3-0.GW13.NYC1.ALTER.NET" target="_blank">0.xe-2-3-0.GW13.NYC1.ALTER.NET</a> (152.179.220.57) 9.715 ms<br>
> <a href="http://0.TenGigE0-0-0-6.GW8.EWR6.ALTER.NET" target="_blank">0.TenGigE0-0-0-6.GW8.EWR6.ALTER.NET</a> (152.179.61.85) 16.414 ms<br>
> <a href="http://0.xe-2-3-0.GW13.NYC1.ALTER.NET" target="_blank">0.xe-2-3-0.GW13.NYC1.ALTER.NET</a> (152.179.220.57) 10.204 ms<br>
> 7 * <a href="http://B400.PHLAPA-LCR-21.verizon-gni.net" target="_blank">B400.PHLAPA-LCR-21.verizon-gni.net</a> (130.81.163.133) 21.193 ms<br>
> <a href="http://P0-4-0-0.PHLAPA-LCR-22.verizon-gni.net" target="_blank">P0-4-0-0.PHLAPA-LCR-22.verizon-gni.net</a> (130.81.22.61) 19.623 ms<br>
> 8 * * *<br>
> 9 static-xxxx 25.310 ms 20.195 ms 24.948 ms<br>
><br>
> VZ has been having some interesting issues the past few months. Maybe the last of the UUNet people finally left. :)<br>
><br>
> Charles<br>
><br>
> On Mar 10, 2015, at 7:16 PM, joel jaeggli via Outages <<a href="mailto:outages@outages.org">outages@outages.org</a>> wrote:<br>
><br>
>> On 3/10/15 2:48 PM, James Cornman via Outages wrote:<br>
>>> OS and other IP services within the past 40 minutes. Mainly<br>
>>> concentrated in the northern New Jersey area. Are there any other<br>
>>> reports of this?<br>
>><br>
>> we have customer who can't reach us for there yeah.<br>
>><br>
>>> -James<br>
>><br>
>><br>
>> _______________________________________________<br>
>> Outages mailing list<br>
>> <a href="mailto:Outages@outages.org">Outages@outages.org</a><br>
>> <a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br>
><br>
> _______________________________________________<br>
> Outages mailing list<br>
> <a href="mailto:Outages@outages.org">Outages@outages.org</a><br>
> <a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br>
<br>
<br>
<br>
------------------------------<br>
<br>
Subject: Digest Footer<br>
<br>
_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br>
<br>
<br>
------------------------------<br>
<br>
End of Outages Digest, Vol 82, Issue 6<br>
**************************************<br>
</blockquote></div><br></div>