[c-nsp] WS-X6704-10GE to WS-X6708-10GE replacement

Artyom Viklenko artem at aws-net.org.ua
Mon May 23 07:51:44 EDT 2011


Hi, list!

We have CISCO 7604 router with the folowing:

Mod Ports Card Type                              Model 
Serial No.
--- ---- -------------------------------------- ------------------ 
-----------
   1   2  Route Switch Processor 720 (Active)    RSP720-3CXL-GE
   2   4  CEF720 4 port 10-Gigabit Ethernet      WS-X6704-10GE
   3  48  CEF720 48 port 1000mb SFP              WS-X6748-SFP
   4  48  CEF720 48 port 10/100/1000mb Ethernet  WS-X6748-GE-TX

Mod   Hw    Fw           Sw           Status
--- ------ ------------ ------------ -------
   1  5.2   12.2(33r)SRB 12.2(33)SRD5 Ok
   2  2.7   12.2(14r)S5  12.2(33)SRD5 Ok
   3  1.11  12.2(14r)S5  12.2(33)SRD5 Ok
   4  2.8   12.2(14r)S5  12.2(33)SRD5 Ok

Mod  Sub-Module                  Model               Hw     Status
---- --------------------------- ------------------ ------- -------
   1  Policy Feature Card 3       7600-PFC3CXL        1.0    Ok
   1  C7600 MSFC4 Daughterboard   7600-MSFC4          1.1    Ok
   2  Centralized Forwarding Card WS-F6700-CFC        4.0    Ok
   3  Centralized Forwarding Card WS-F6700-CFC        4.0    Ok
   4  Centralized Forwarding Card WS-F6700-CFC        4.0    Ok


Recently we try to replace 6704 line card with 6708:

Mod Ports Card Type                              Model
--- ----- -------------------------------------- ------------------
  2    8   CEF720 8 port 10GE with DFC            WS-X6708-10GE

Mod   Hw    Fw           Sw           Status
--- ------ ------------ ------------ -------
  2   1.4   12.2(18r)S1  12.2(33)SRD5 Ok


First of all, the folowing command was entered:
"no power enable mod 2"

10G interfaces goes down, status led on line card turns off.
After removing 6704 card and inserting 6708 router become unavail.
According to crashdump last evens was:

May 20 15:45:42.446 EEST: %C7600_PWR-SP-4-DISABLED: power to module in 
slot 2 set off (admin request)
May 20 15:45:42.490 EEST: %LINK-SP-3-UPDOWN: Interface 
TenGigabitEthernet2/1, changed state to down
May 20 15:45:42.490 EEST: %LINK-SP-3-UPDOWN: Interface 
TenGigabitEthernet2/2, changed state to down
May 20 15:45:42.490 EEST: %LINK-SP-3-UPDOWN: Interface 
TenGigabitEthernet2/3, changed state to down
May 20 15:45:42.490 EEST: %LINK-SP-3-UPDOWN: Interface 
TenGigabitEthernet2/4, changed state to down
May 20 15:45:42.494 EEST: %LINEPROTO-SP-5-UPDOWN: Line protocol on 
Interface TenGigabitEthernet2/1, changed state to down
May 20 15:45:42.494 EEST: %LINEPROTO-SP-5-UPDOWN: Line protocol on 
Interface TenGigabitEthernet2/2, changed state to down
May 20 15:45:42.494 EEST: %LINEPROTO-SP-5-UPDOWN: Line protocol on 
Interface TenGigabitEthernet2/3, changed state to down
May 20 15:45:42.494 EEST: %LINEPROTO-SP-5-UPDOWN: Line protocol on 
Interface TenGigabitEthernet2/4, changed state to down
May 20 15:46:18.414 EEST: %LINEPROTO-SP-5-UPDOWN: Line protocol on 
Interface GigabitEthernet3/3, changed state to down
May 20 15:46:18.418 EEST: %LINK-SP-3-UPDOWN: Interface 
GigabitEthernet3/3, changed state to down
May 20 15:46:19.610 EEST: %LINK-SP-3-UPDOWN: Interface 
GigabitEthernet3/3, changed state to up
May 20 15:46:20.058 EEST: %OIR-SP-6-REMCARD: Card removed from slot 2, 
interfaces disabled
May 20 15:46:22.602 EEST: %LINEPROTO-SP-5-UPDOWN: Line protocol on 
Interface GigabitEthernet3/3, changed state to up
May 20 15:46:24.078 EEST: %LINEPROTO-SP-5-UPDOWN: Line protocol on 
Interface GigabitEthernet3/3, changed state to down
May 20 15:46:24.086 EEST: %LINK-SP-3-UPDOWN: Interface 
GigabitEthernet3/3, changed state to down
May 20 15:46:27.930 EEST: %LINK-SP-3-UPDOWN: Interface 
GigabitEthernet3/3, changed state to up
May 20 15:46:30.922 EEST: %LINEPROTO-SP-5-UPDOWN: Line protocol on 
Interface GigabitEthernet3/3, changed state to up
May 20 15:47:14.491 EEST: %C7600_PWR-SP-4-DISABLED: power to module in 
slot 2 set off (admin request)
May 20 15:47:24.211 EEST: %EARL-SP-2-PATCH_INVOCATION_LIMIT: 10 Recovery 
patch invocations in the last 30 secs have been attempted. Max limit reached

%Software-forced reload


But after reboot we found extremly large rtt to this router 6-9 secons 
with typical 8 mS and near impossible ssh connection - too high CPU load.

In general, for example after IOS upgrade this router backs online 
withing 4 minutes. We wait 10 - no luck. We try to power cycle it.

After more than typical 4 mins. we start to receive syslog messages from
it (no console output, unfortunatelly). According to these messages,
different ports on two 48-port line cards (SFP and Copper) fails 
Loopback tests, and was err-disabled after they goes online.

For example:

*May 20 15:54:02.544 EEST: %DIAG-SP-3-MINOR: Module 4: Online 
Diagnostics detected a Minor Error. Please use 'show diagnostic result 
<target>' to see test results.
*May 20 15:54:02.544 EEST: %CONST_DIAG-SP-3-BOOTUP_TEST_FAIL: Module 4: 
TestLoopback failed on port(s) 
2,4-6,8-9,12-13,15,19,23,30,32,35,37-38,41,43,46,48
*May 20 15:54:02.544 EEST: %CONST_DIAG-SP-3-BOOTUP_TEST_FAIL: Module 4: 
TestL3VlanMet failed

*May 20 15:54:03.223 EEST: %LINK-3-UPDOWN: Interface GigabitEthernet4/1, 
changed state to down
*May 20 15:54:03.627 EEST: %LINK-3-UPDOWN: Interface GigabitEthernet4/2, 
changed state to down
*May 20 15:54:03.543 EEST: %PM-SP-4-ERR_DISABLE: diagnostics error 
detected on Gi4/2, putting Gi4/2 in err-disable state

*May 20 15:54:04.543 EEST: %PM-SP-4-ERR_DISABLE: diagnostics error 
detected on Gi4/4, putting Gi4/4 in err-disable state
*May 20 15:54:04.743 EEST: %PM-SP-4-ERR_DISABLE: diagnostics error 
detected on Gi4/6, putting Gi4/6 in err-disable state

And after another reboot

May 20 15:57:52.972 EEST: %DIAG-SP-3-MINOR: Module 3: Online Diagnostics 
detected a Minor Error. Please use 'show diagnostic result <target>' to 
see test results.
May 20 15:57:52.972 EEST: %CONST_DIAG-SP-3-BOOTUP_TEST_FAIL: Module 3: 
TestLoopback failed on port(s) 1-7,9-14,16-19,21,23-43,45-48
May 20 15:57:52.972 EEST: %CONST_DIAG-SP-3-BOOTUP_TEST_FAIL: Module 3: 
TestL3VlanMet failed
.May 20 15:57:54.416 EEST: %LINK-3-UPDOWN: Interface GigabitEthernet3/1, 
changed state to down
May 20 15:57:54.311 EEST: %PM_SCP-SP-3-TRANSCEIVER_UNSUPPORTED: 
Unsupported transceiver in LAN port 3/5
May 20 15:57:54.399 EEST: %PM_SCP-SP-3-TRANSCEIVER_UNSUPPORTED: 
Unsupported transceiver in LAN port 3/9
.May 20 15:57:55.024 EEST: %LINK-3-UPDOWN: Interface GigabitEthernet3/4, 
changed state to down
.May 20 15:57:55.092 EEST: %LINK-3-UPDOWN: Interface GigabitEthernet3/2, 
changed state to down
.May 20 15:57:55.292 EEST: %LINK-3-UPDOWN: Interface GigabitEthernet3/3, 
changed state to down
May 20 15:57:54.407 EEST: %PM-SP-4-ERR_DISABLE: diagnostics error 
detected on Gi3/1, putting Gi3/1 in err-disable state
May 20 15:57:54.607 EEST: %PM-SP-4-ERR_DISABLE: diagnostics error 
detected on Gi3/2, putting Gi3/2 in err-disable state
May 20 15:57:54.811 EEST: %PM-SP-4-ERR_DISABLE: diagnostics error 
detected on Gi3/3, putting Gi3/3 in err-disable state
May 20 15:57:55.011 EEST: %PM-SP-4-ERR_DISABLE: diagnostics error 
detected on Gi3/4, putting Gi3/4 in err-disable state


Finally, we return 6704 line card back and rotuer rebooted and
works fine as usual.

6708 line cad today was installed in another 6704 router with the
same RSP - all seems ok with it.

IOS on both routers c7600rsp72043-advipservicesk9-mz.122-33.SRD5.bin.

According to Cisco, 6708 line card can be installed in ANY slot of 7604
chassis, ISO SRD5 supports it as well. Similar 6708 line cars now
works in another router but 7606-S chassis.

Powe supply in problem router is 2700W. And now utilised only half from
awailable power and differens between 6704 and 6708 only 133W.

Does anybody have some glue what is wrong?

Is it possible that such behavior due to different ROMMON versions
on line cards in bundle or something wrong with chassis?

Thans in advance!


-- 
            Sincerely yours,
                             Artyom Viklenko.
-------------------------------------------------------
artem at aws-net.org.ua | http://www.aws-net.org.ua/~artem
artem at viklenko.net   | JID: artem at jabber.aws-net.org.ua
FreeBSD: The Power to Serve   -  http://www.freebsd.org


More information about the cisco-nsp mailing list