[c-nsp] 3750-12G interfaces dropping out

Thu Feb 5 00:50:24 EST 2009

Hi all,

We're having an odd issue with a 3750 that's running as the core switch at one of our sites. It's been fine for years; suddenly, this week, it's decided that some of it's interfaces should reset randomly, disrupting access to staff phones and PCs in the buildings at the other end.

It's in a stack (as master) with a 3750-48. All interfaces are populated with 1000SX SFP modules (all genuine Cisco).

All edge switches are 3560s. None of the edge switches show anything in the log about the link loss (it's only about 3 seconds) but the core switch shows this:
Date

Time

Priority

Hostname

 Message

02-05-2009

16:30:44

Local7.Notice

10.2.115.182

640: 000609: Feb 5 16:30:44: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/9, changed state to up

02-05-2009

16:30:41

Local7.Notice

10.2.115.182

639: 000608: Feb 5 16:30:41: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/9, changed state to down

02-05-2009

16:28:31

Local7.Notice

10.2.115.182

638: 000607: Feb 5 16:28:31: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/10, changed state to up

02-05-2009

16:28:28

Local7.Notice

10.2.115.182

637: 000606: Feb 5 16:28:28: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/10, changed state to down

02-05-2009

16:24:41

Local7.Notice

10.2.115.182

635: 000604: Feb 5 16:24:41: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/5, changed state to up

02-05-2009

16:24:38

Local7.Notice

10.2.115.182

634: 000603: Feb 5 16:24:38: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/5, changed state to down

02-05-2009

16:16:42

Local7.Notice

10.2.115.182

633: 000602: Feb 5 16:16:42: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/9, changed state to up

02-05-2009

16:16:38

Local7.Notice

10.2.115.182

632: 000601: Feb 5 16:16:38: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/9, changed state to down

02-05-2009

16:14:36

Local7.Notice

10.2.115.182

631: 000600: Feb 5 16:14:36: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/9, changed state to up

02-05-2009

16:14:33

Local7.Notice

10.2.115.182

630: 000599: Feb 5 16:14:33: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/9, changed state to down

02-05-2009

16:12:03

Local7.Notice

10.2.115.182

629: 000598: Feb 5 16:12:03: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/9, changed state to up

02-05-2009

16:12:00

Local7.Notice

10.2.115.182

628: 000597: Feb 5 16:12:00: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/9, changed state to down

02-05-2009

16:01:35

Local7.Notice

10.2.115.182

627: 000596: Feb 5 16:01:35: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/2, changed state to up

02-05-2009

16:01:32

Local7.Notice

10.2.115.182

626: 000595: Feb 5 16:01:32: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/2, changed state to down

02-05-2009

15:49:02

Local7.Notice

10.2.115.182

625: 000594: Feb 5 15:49:01: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet2/0/21, changed state to up

02-05-2009

15:49:00

Local7.Notice

10.2.115.182

624: 000593: Feb 5 15:48:59: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet2/0/21, changed state to down

02-05-2009

15:43:24

Local7.Notice

10.2.115.182

623: 000592: Feb 5 15:43:24: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/2, changed state to up

02-05-2009

15:43:21

Local7.Notice

10.2.115.182

622: 000591: Feb 5 15:43:21: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/2, changed state to down

I note that it's only the line protocol dropping, not the actual interface. I have seen this happen once when I was looking at the switch - all the activity lights froze, some went out, and then the switch happily went back to normal.
It's only some of the interfaces, too - 1, 2, 5, 9 and 10 out of 12. None of the interfaces on the 3750-48 that's a stack member have been affected at all.

Has anyone seen this sort of behaviour before? It's hard to know where to look. Nothing much has happened this week; we did have a new link to new switches in a building patched in on Monday but that link isn't affected (I disabled the interface and the problem was still occurring while it was down). Some PCs and phones have been set up in one of the buildings that previously were at another site, so perhaps that could be related, although the edge switches in that building show no problems internally...

Regards,

Damien

**********************************************************************
This message is intended for the addressee named and may contain
privileged information or confidential information or both. If you
are not the intended recipient please delete it and notify the sender.
**********************************************************************