[c-nsp] GigE woes

[Tim Durack]

I've got a crazy GigE circuit that's having problems:

RTR-1#ping vrf v10 10.241.1.10 repeat 10000 df-bit size 9000 timeout 1

Type escape sequence to abort.
Sending 10000, 9000-byte ICMP Echos to 10.241.1.10, timeout is 1 seconds:
Packet sent with the DF bit set
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (587/587), round-trip min/avg/max = 8/18/408 ms

RTR-1#ping vrf v10 10.241.1.10 repeat 10000 df-bit size 200 timeout 1

Type escape sequence to abort.
Sending 10000, 200-byte ICMP Echos to 10.241.1.10, timeout is 1 seconds:
Packet sent with the DF bit set
..!..!..!.!....!..!..!.!!..!...!..!..!..!..!.!!...!.!..!..!....!..!..!
..!..!....!..!..!..!.!.!..!....!.!..!..!..!..!....!..!..!..!..!.!.!..!
.!..!..!..!..!....!..!..!..!..!....!!..!.!..!....!.!!....!..!..!..!..!
....!.!..
Success rate is 32 percent (72/219), round-trip min/avg/max = 4/160/1180 ms

RTR-1#ping vrf v10 10.241.1.10 repeat 10000 df-bit size 9000 timeout 1

Type escape sequence to abort.
Sending 10000, 9000-byte ICMP Echos to 10.241.1.10, timeout is 1 seconds:
Packet sent with the DF bit set
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (255/255), round-trip min/avg/max = 12/16/228 ms

Ping with large frame size appears to pass. Small frame size results
in packet loss and unusually high rtt.

Additionally, traffic must be present in both directions for any
traffic to cross the link. I have to start the ping on both routers
simultaneously. If the ping is stopped on one side, the ping from the
other side stops returning.

Have mostly ruled out my equipment, but the carrier doesn't believe
there is anything wrong on their side. They strap a GigE tester across
the path and of course it works just fine. But try convincing arp/ospf
to work across such a link.

The link appears to be a muxponder over the carriers DWDM network, so
it is mostly optical, possibly only two OEO points at the customer
hand off.

Anyone seen anything like this before?

-- 
Tim:>