[outages] Cogent /ATT and Verizon Peering alternatives

Mon Feb 10 22:12:58 EST 2014

By total chance wearing my SomaFM T-shirt today...

If it's of any help, here are some mtrs coming from two different
locations/environments.

This is only half the picture though -- the return path to the below
sources is unknown.  I've provided source IPs in case you want to run
tests back (please use ICMP protocol).

src IP: 76.102.14.35  (Comcast; Mountain View, CA)
dst IP: 71.252.251.1  (Verizon; Dallas, TX)

                                                                      Packets               Pings
 Host                                                               Loss%   Snt   Rcv  Last   Avg  Best  Wrst
 1. gw.home.lan (192.168.1.1)                                        0.0%    91    91   0.2   0.3   0.2   0.4
 2. 76.102.12.1                                                      0.0%    91    91   8.0   9.1   7.9  25.2
 3. te-0-2-0-5-ur06.santaclara.ca.sfba.comcast.net (68.86.249.253)   0.0%    91    90  16.5   8.9   8.1  16.5
 4. te-0-5-0-2-ur05.santaclara.ca.sfba.comcast.net (69.139.198.41)   0.0%    90    90   8.6   9.0   8.3  16.6
 5. te-1-1-0-0-ar01.sfsutro.ca.sfba.comcast.net (69.139.198.82)      0.0%    90    90  14.7  12.2   9.9  17.1
    te-1-1-0-3-ar01.sfsutro.ca.sfba.comcast.net (68.85.155.62)
    te-1-1-0-9-ar01.sfsutro.ca.sfba.comcast.net (69.139.198.178)
    te-1-1-0-10-ar01.sfsutro.ca.sfba.comcast.net (68.87.226.118)
    te-1-1-0-4-ar01.sfsutro.ca.sfba.comcast.net (68.85.155.66)
    te-1-1-0-8-ar01.sfsutro.ca.sfba.comcast.net (68.85.155.146)
    te-1-1-0-13-ar01.sfsutro.ca.sfba.comcast.net (69.139.199.106)
    te-1-1-0-1-ar01.sfsutro.ca.sfba.comcast.net (69.139.198.90)
 6. he-1-7-0-0-cr01.sanjose.ca.ibone.comcast.net (68.86.90.153)      0.0%    90    90  14.1  13.4  10.8  21.8
 7. be-10-pe02.11greatoaks.ca.ibone.comcast.net (68.86.82.82)        0.0%    90    90  14.7  15.2  14.0  51.0
 8. 23.30.206.94                                                    11.1%    90    80  73.8  72.3  65.9  79.6
 9. B100.DLLSTX-LCR-21.verizon-gni.net (130.81.199.59)              10.0%    90    81 111.8 109.5 106.7 116.5
10. ???
11. L100.DLLSTX-VFTTP-64.verizon-gni.net (71.252.251.1)              6.7%    90    84 113.4 110.4 107.1 131.9

src IP: 208.79.90.130  (ARP Networks; Sylmar, CA)
dst IP: 71.252.251.1   (Verizon; Dallas, TX)

                                                                      Packets               Pings
 Host                                                               Loss%   Snt   Rcv  Last   Avg  Best  Wrst
 1. 208.79.90.129                                                    0.0%    91    91   1.0   4.9   0.9  69.8
 2. s7.lax.arpnetworks.com (208.79.88.135)                           0.0%    91    91   0.5  12.1   0.4 188.5
 3. ge-0-7-0-24.r04.lsanca03.us.bb.gin.ntt.net (129.250.198.185)     0.0%    91    91   0.8   0.9   0.7   1.4
 4. ae-3.r05.lsanca03.us.bb.gin.ntt.net (129.250.2.221)              0.0%    90    90   1.2   1.2   0.9   4.3
 5. ae-0.verizonbusiness.lsanca03.us.bb.gin.ntt.net (129.250.8.86)   7.8%    90    83  29.4  32.0  28.9  85.7
 6. P0-8-0-0.DLLSTX-LCR-21.verizon-gni.net (130.81.199.35)           2.2%    90    88  69.3  69.5  67.0  76.3
 7. ???
 8. L100.DLLSTX-VFTTP-64.verizon-gni.net (71.252.251.1)              3.3%    90    87  68.7  71.1  64.5 200.9

It's very hard to discern something useful from these single sets of
mtrs because ICMP deprioritisation has become so commonplace**.  For
example (going off gut feeling), in the 2nd mtr above, I'd say that the
NTT/Verio hop (#5) is probably ICMP deprio, while subsequent loss is
probably real.

Also, 23.30.206.94 lives within Comcast IP space, but it's impossible to
know what company (Verizon or Comcast) is responsible for it.

** -- Folks often forget that even when using classic UDP traceroute,
ICMP is still involved (specifically time exceeded).

-- 
| Jeremy Chadwick                                   jdc at koitsu.org |
| UNIX Systems Administrator                http://jdc.koitsu.org/ |
| Making life hard for others since 1977.             PGP 4BD6C0CB |

On Mon, Feb 10, 2014 at 05:02:40PM -0800, Rusty Hodge wrote:
> We are seeing lots of problems with our audio streaming traffic dropping lots of packets where our transit provider Cogent hands off to ATT & Verizon.
> 
> Right now, I'm seeing  43% packet loss
> 
> Cogent is blaming ATT/Verizon and says there is nothing they can do about it. We're the collateral damage.
> 
> Who else is having the same issue?
> 
> I threaten to cancel my service, Cogent says, "sorry to lose you as a customer", and go on their merry way.
> 
> Are other transit providers having issues peering / getting to ATT/Verizon? I need to know who to avoid.
> 
> 
> 
> traceroute to 71.252.251.1 (71.252.251.1), 30 hops max, 60 byte packets
>  1  gi3-11.ccr01.sfo02.atlas.cogentco.com (38.104.130.89)  0.982 ms  0.985 ms  1.195 ms
>  2  te0-0-0-28.ccr22.sfo01.atlas.cogentco.com (154.54.3.129)  0.531 ms te0-0-0-28.ccr21.sfo01.atlas.cogentco.com (154.54.3.121)  0.604 ms  0.636 ms
>  3  be2165.ccr22.sjc01.atlas.cogentco.com (154.54.28.66)  2.277 ms be2164.ccr21.sjc01.atlas.cogentco.com (154.54.28.34)  1.806 ms  2.449 ms
>  4  be2000.ccr21.sjc03.atlas.cogentco.com (154.54.6.106)  3.824 ms  2.504 ms  2.596 ms
>  5  * * verizon.sjc03.atlas.cogentco.com (154.54.12.170)  94.373 ms
>  6  * * *
>  7  * * *
>  8  L100.DLLSTX-VFTTP-64.verizon-gni.net (71.252.251.1)  135.242 ms  86.272 ms  86.231 ms
> 
> 
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages