[c-nsp] GigE woes

[Tim Durack]

After a month of denial, the providerjust reconfigured the GigE
circuit to be "clear channel", and it is now behaving itself |-\

Provider says we must be doing some weird encap/encryption. Nope. Just
two 6504s, SUP720-3BXL, GigE L3 interface. We replaced SFPs, tried
different 6504s, linecards, SUP720s. Tried a pair of simple HP L2
switches. No resolution.

Finally put some Anritsu GigE testers and ran RFC2544. Fails tests
with high loss at 64B frame, high loss on burst tests, and weird
latency results.

The fact that "clear channel" fixes this makes me think there is an
OEO element with a framing problem. At this point the circuit is
working, so it is academic. Like to know more for the next time
though...

Anyone else got ideas?

Tim:>

On Fri, May 14, 2010 at 2:53 PM, Tim Durack <tdurack at gmail.com> wrote:
> I've got a crazy GigE circuit that's having problems:
>
> RTR-1#ping vrf v10 10.241.1.10 repeat 10000 df-bit size 9000 timeout 1
>
> Type escape sequence to abort.
> Sending 10000, 9000-byte ICMP Echos to 10.241.1.10, timeout is 1 seconds:
> Packet sent with the DF bit set
> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
> !!!!!!!!!!!!!!!!!!!!!!!!!!!
> Success rate is 100 percent (587/587), round-trip min/avg/max = 8/18/408 ms
>
> RTR-1#ping vrf v10 10.241.1.10 repeat 10000 df-bit size 200 timeout 1
>
> Type escape sequence to abort.
> Sending 10000, 200-byte ICMP Echos to 10.241.1.10, timeout is 1 seconds:
> Packet sent with the DF bit set
> ..!..!..!.!....!..!..!.!!..!...!..!..!..!..!.!!...!.!..!..!....!..!..!
> ..!..!....!..!..!..!.!.!..!....!.!..!..!..!..!....!..!..!..!..!.!.!..!
> .!..!..!..!..!....!..!..!..!..!....!!..!.!..!....!.!!....!..!..!..!..!
> ....!.!..
> Success rate is 32 percent (72/219), round-trip min/avg/max = 4/160/1180 ms
>
> RTR-1#ping vrf v10 10.241.1.10 repeat 10000 df-bit size 9000 timeout 1
>
> Type escape sequence to abort.
> Sending 10000, 9000-byte ICMP Echos to 10.241.1.10, timeout is 1 seconds:
> Packet sent with the DF bit set
> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
> Success rate is 100 percent (255/255), round-trip min/avg/max = 12/16/228 ms
>
> Ping with large frame size appears to pass. Small frame size results
> in packet loss and unusually high rtt.
>
> Additionally, traffic must be present in both directions for any
> traffic to cross the link. I have to start the ping on both routers
> simultaneously. If the ping is stopped on one side, the ping from the
> other side stops returning.
>
> Have mostly ruled out my equipment, but the carrier doesn't believe
> there is anything wrong on their side. They strap a GigE tester across
> the path and of course it works just fine. But try convincing arp/ospf
> to work across such a link.
>
> The link appears to be a muxponder over the carriers DWDM network, so
> it is mostly optical, possibly only two OEO points at the customer
> hand off.
>
> Anyone seen anything like this before?
>
> --
> Tim:>
>



-- 
Tim:>