[outages] Newark Verizon Issues....

[Jeremy Chadwick]

On Thu, Sep 23, 2010 at 06:24:03PM +0000, john at quonix.net wrote:
> Many of my customers cant reach me from behind FioS. Traceroute shows a 
> problem spot in Newark in Verizon backbone. 256ms latency at the last hop, 
> then dies. 
> 
> It several hops within Verizon, so it cant be an Abovenet peering issue.
> 
> If anyone from Verizon is listening out there, please look into this. Fios 
> support refuses to troubleshoot. I know at least 20 people who have called 
> in to complain, there are Verizon (DSL or Fios) through Jersey and 
> Delaware. 
> 
>  1  gw-238-225.quonix.net (208.72.238.225)  0.256 ms   0.268 ms   0.270 ms
>  2  ge-11-1-2.mpr3.phl2.us.above.net (209.249.122.165)  0.174 ms   0.195 
> ms   0. 
> 166 ms
>  3  xe-2-3-0.cr1.lga5.us.above.net (64.125.31.34)  2.422 ms   2.357 ms   
> 2.412 m 
> s
>  4  xe-0-1-0.er1.lga5.us.above.net (64.125.27.61)  2.222 ms   2.200 ms   
> 2.228 m 
> s
>  5  0.ge-3-2-0.BR3.NYC4.ALTER.NET (204.255.168.25)  2.254 ms   2.297 ms   
> 2.264 
> ms
>  6  0.ge-4-2-0.NY5030-BB-RTR2.verizon-gni.net (152.63.10.54)  2.573 ms   
> 2.605 m 
> s   2.566 ms
>  7  P15-0-0.NWRKNJ-LCR-04.verizon-gni.net (130.81.29.195)  4.767 ms   
> 4.701 ms 
>  4.714 ms
>  8  P12-0-0.NWRKNJ-LCR-06.verizon-gni.net (130.81.27.7)  5.717 ms   5.348 
> ms   5 
> .262 ms
>  9  P14-0.NWRKNJ-LCR-08.verizon-gni.net (130.81.30.95)  5.265 ms   5.202 
> ms   5. 
> 259 ms
> 10  * * *
> 11  * * *
> 12  * * *
> 13  * * *
> 14  * * *

This isn't to say there's not a problem, but where is the "256ms latency
at the last hop" evidence?  All that's shown here is 3 probes at hop #9
with RTTs of 5.265, 5.202, and 5.259ms.  The copy-paste was done poorly
so the output is formatted badly.

You're going to need to provide traceroutes from both directions
(src->dst and dst->src), and should probably try UDP, TCP, and
ICMP-based traceroutes to rule out filtering at the endpoints.  Some
traceroute implementations also offer a flag (on BSD it's -e, to be used
alongside -p) which causes UDP and TCP destination packets to *not*
increment their port number per probe, which can help if the destination
uses a firewall (and the user has control over it).

When dealing with Verizon, you should probably also indicate if all
forms of packets are being filtered (meaning provide actual packet
traces on both the source and destination ends showing something like
the results of an telnet x.x.x.x yyyy towards you, where yyyy should be
a port you're definitely listening on + have permitted on your firewall
and/or port forwarded).  Basically you want to confirm if the remote end
receives a SYN+ACK response (from you) to their initial SYN.

Good luck.

-- 
| Jeremy Chadwick                                   jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.              PGP: 4BD6C0CB |