[c-nsp] Unusual outage scenario

[Rick Coloccia]

Hi Everyone,

First, thanks to everyone on this list for their great help - there are 
lots of very experienced folks out here who share vast amounts of 
knowledge. 


OK, to my strange and unique issue:

Customer router is a Cisco 7304 running 12.2(33)SB7.  ISP router is an 
Alcatel Service Router (Make and model are unknown, so I can't share them.)

Twice in 5 weeks the exact same thing has happened:

This appears in customer router logs:

Apr 25 22:06:54 router 1620: Apr 25 22:06:48 EDT: %BGP-5-ADJCHANGE: 
neighbor x.x.x.x Down BGP Notification sent
Apr 25 22:06:54 router 1621: Apr 25 22:06:48 EDT: %BGP-3-NOTIFICATION: 
sent to neighbor x.x.x.x 4/0 (hold time expired) 0 bytes
Apr 25 22:06:55 router 1622: Apr 25 22:06:48 EDT: 
%BGP_SESSION-5-ADJCHANGE: neighbor x.x.x.x IPv4 Unicast topology base 
removed from session  BGP Notification sent

And as of that moment, customer router is no longer able to ping x.x.x.x.

The relevant interface on the customer router is g2.  It is a copper 
interface connected to a mux, which connects to single mode fiber that 
ultimately lands in the ISPs MPLS network.  At the far end of the MPLS 
network somewhere is the Alcatel, connected with ip address x.x.x.x.

Naturally, once the BGP session drops, all traffic stops passing.

When I connect to the customer router, and issue the "shut" and then "no 
shut" commands to g2, the router is suddenly again able to ping x.x.x.x 
and the bgp session subsequently restores itself.

What sort of issue may be going on such that the solution to this outage 
is to issue "shut" and "no shut" on g2 on the customer router?

The customer router is never showing g2 going down as the mux connected 
to the router never does go down.

I suspect something is happening in the MPLS network at the beginning of 
the outage, but I can't understand why a "shut" and "no shut" is brining 
everything back to life.

Does anyone have an insight into this that I can use to prevent this 
from happening a third time?

Thanks in advance!

-Rick







-- 
Rick Coloccia, Jr.
Network Manager
State University of NY College at Geneseo
1 College Circle, 119 South Hall
Geneseo, NY 14454
V: 585-245-5577
F: 585-245-5579

CIT will never ask for your password or other confidential information via email.