[j-nsp] BGP apparent I/O throttling on MX960 (JUNOS 14.1R6)

[Adam Chappell]

Hello all.

Anyone any experience with situations where "show bgp neighbor X.X.X.X" on
JUNOS CLI produces a small appendix to the usual output stating: "Received
and buffered octets: 20".  20 in this case seems to vary between
invocations, but usually under 100.  Example pseudo-sanitised output at the
end of the mail for anyone interested.

It seems to suggest that rpd completed a short read or is otherwise still
waiting for a complete message from the remote peer. In this case, the
remote peer was aware of the situation because they monitor their XR
speaker's BGP OutQ instrumentation to watch for slow readers. Their
observation was a zero-size advertised receive window in TCP preventing
their BGP speaker from sending queued message).

Problem disappeared after several days with no obvious action taken by
either party. OutQ on peer's side returned to zero, and I could see no
further messages of partial reception or buffering on our side.

Beyond the instrumentation, I could find no obvious evidence of degraded
function. Peer's large OutQ was a cause for concern (their export policy to
us is one of full table), but it went unrealised. No other BGP peer on the
box exhibited similar symptoms. NSR in use (wondered if replication between
REs could slow down effective TCP receive rate). Link-layer to peer was
reliable and with low latency.

Probably in the X-Files, I realise, but I thought a stab-in-the-dark here
might be worthwhile.

-- Adam.

Peer: REMOTE_IP+179 AS REMOTE_AS Local: LOCAL+51007 AS MY_AS
  Description: Peer
  Type: External    State: Established    Flags: <Sync RSync>
  Last State: EstabSync     Last Event: RecvKeepAlive
  Last Error: Cease
  Export: [ OUT ] Import: [ IN ]
  Options: <Preference AdvertiseInactive AuthKey LogUpDown PeerAS Refresh>
  Options: <MtuDiscovery>
  Authentication key is configured
  Holdtime: 90 Preference: 170
  Number of flaps: 1
  Last flap event: Stop
  Error: 'Cease' Sent: 1 Recv: 0
  Peer ID: PEER-ROUTER-ID   Local ID: MY-ROUTER-ID     Active Holdtime: 90
  Keepalive Interval: 30         Group index: 9    Peer index: 0
  BFD: disabled, down
  Local Interface: ae12.0
  NLRI for restart configured on peer: inet-unicast
  NLRI advertised by peer: inet-unicast inet-multicast
  NLRI for this session: inet-unicast
  Peer supports Refresh capability (2)
  Stale routes from peer are kept for: 300
  Peer does not support Restarter functionality
  NLRI that restart is negotiated for: inet-unicast
  NLRI of received end-of-rib markers: inet-unicast
  NLRI of all end-of-rib markers sent: inet-unicast
  Peer supports 4 byte AS extension (peer-as REMOTE_AS)
  Peer does not support Addpath
  Table inet.0 Bit: 10007
    RIB State: BGP restart is complete
    Send state: in sync
    Active prefixes:              203815
    Received prefixes:            595715
    Accepted prefixes:            505370
    Suppressed due to damping:    0
    Advertised prefixes:          5755
  Last traffic (seconds): Received 27   Sent 2    Checked 57
  Input messages:  Total 13670093 Updates 13660814 Refreshes 41 Octets
1307202704
  Output messages: Total 299601 Updates 174126 Refreshes 0 Octets 15932309
  Output Queue[0]: 0
*  Received and buffered octets: 29*

adamc at router> show system connections extensive | find REMOTE
tcp4       0     38  LOCAL.51007                           REMOTE.179
                      ESTABLISHED
   sndsbcc:         38 sndsbmbcnt:        256  sndsbmbmax:     131072
sndsblowat:       2048 sndsbhiwat:      16384
   rcvsbcc:          0 rcvsbmbcnt:          0  rcvsbmbmax:     131072
rcvsblowat:          1 rcvsbhiwat:      16384
   proc id:       1873  proc name:        rpd
       iss: 3882968990      sndup: 3901286434
    snduna: 3901286434     sndnxt: 3901286472      sndwnd:      32195
    sndmax: 3901286472    sndcwnd:      12397 sndssthresh:          0
       irs:  811065039      rcvup: 2118576537
    rcvnxt: 2118576537     rcvadv: 1059075027      rcvwnd:      16384
       rtt: 1977181859       srtt:     556992        rttv:     388071
    rxtcur:      64000   rxtshift:          0       rtseq: 3901286434
    rttmin:          0  mss:       1404
     flags: REQ_SCALE RCVD_SCALE REQ_TSTMP [0x130000e0]