[c-nsp] 7507 GEIP Controller

Masood Ahmad Shah masood at nexlinx.net.pk
Tue Jul 31 09:18:32 EDT 2007 

The router has been upgraded to 12.4 (16). The router CPU is now back to
normal.  

The problem was the result of an issue on Ether-Channel process. We briefly
recall the configuration of Ether-Channel few days ago, and Ether-Channel
interrupts were getting sent to the RSP for some reason. I think there is a
bug in Ether-Channel process. If we configure the Ether-Channel at the
interface, and then revert it to a previous level. it consume some amount of
CPU and it's process stay active till the next reboot.


Regards,
Masood Ahmad Shah

-----Original Message-----
From: Rodney Dunn [mailto:rodunn at cisco.com] 
Sent: Tuesday, July 31, 2007 5:32 PM
To: Masood Ahmad Shah
Cc: 'Rodney Dunn'; cisco-nsp at puck.nether.net
Subject: Re: [c-nsp] 7507 GEIP Controller

On Thu, Jul 26, 2007 at 03:04:30AM +0500, Masood Ahmad Shah wrote:
> 
> Well, the CPU usage is high even without cable. I started generating 
> CPU graph and it shows the CPU usage is pretty high after inserting 
> GEIP controller in. I tried to get "sh stack" picture and according to 
> stats "Network Interrupt" is being called higher than others. Traffic 
> is being passed from this interface and there are no CRCs or input errors.

Huh? That seems whacked. What did 'sh int' snapshots say for a couple
outputs? 

Did you reload it after you inserted the GEIP?

> 
> The question is, how can I check the GEIP controller is faulty or GEIP
card?
> How can one check that faulty card is causing high cpu load while the 
> card is working good if we talk about traffic passing out/in from that
interface.


> 
> Is there any workaround for such issues? What do you suggest upgarding 
> from
> 12.4 (3a) to 12.4 (16) will have positive impact :)

It's a good move to be proactive given 12.4 is the bugfix train for the rest
of the support life of the 75xx.

> 
> Anyways once again thanks for your reply.
> 
> Regards,
> Masood Ahmad Shah
> 
> 
> -----Original Message-----
> From: Rodney Dunn [mailto:rodunn at cisco.com]
> Sent: Thursday, July 26, 2007 2:26 AM
> To: Masood Ahmad Shah
> Cc: 'Rodney Dunn'; cisco-nsp at puck.nether.net
> Subject: Re: [c-nsp] 7507 GEIP Controller
> 
> On Thu, Jul 26, 2007 at 01:26:46AM +0500, Masood Ahmad Shah wrote:
> > Thanks for your reply..I have not tried without cable, I will check it. 
> > 
> > Does cable effects even if the interface is shutdown?
> 
> For packet switching it shouldn't but if the driver is still sending 
> an interrupt it may.
> 
> > Why I can't see the process taking this high cpu usage while I do 
> > show proc cpu etcc.?
> 
> Because it's interrupt level work the CPU is doing. We could try 
> profiling the CPU and see what it says.
> 
> It may or may not help.
> 
> > 
> > Here is the output for show diag, show stacks and show align
> > 
> > Router#show diag 5 ( this is GEIP controller slot ) Slot 5:
> >         Physical slot 5, ~physical slot 0xA, logical slot 5, CBus 1
> >         Microcode Status 0x4
> >         Master Enable, LED, WCS Loaded
> >         Board is analyzed 
> >         Pending I/O Status: None
> >         EEPROM format version 1
> >         GEIP controller, HW rev 2.02, board revision A0
> >         Serial number: 27588119  Part number: 73-2167-05
> >         Test history: 0x00        RMA number: 00-00-00
> >         Flags: cisco 7000 board; 7500 compatible
> > 
> >         EEPROM contents (hex):
> >           0x20: 01 21 02 02 01 A4 F6 17 49 08 77 05 00 00 00 00
> >           0x30: 50 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00
> > 
> >         Slot database information:
> >         Flags: 0x4      Insertion time: 0x33C0 (4d00h ago)
> > 
> >         Controller Memory Size: 128 MBytes DRAM, 4096 KBytes SRAM
> > 
> >         PA Bay 0 Information:
> >                 Gigabit-Ethernet PA, 1 ports
> >                 EEPROM format version 1
> >                 HW rev 1.01, Board revision A0
> >                 Serial number: 29339093  Part number: 73-3144-05
> > 
> >         --Boot log begin--
> > 
> > Cisco IOS Software, VIP Software (SVIP-DW-M), Version 12.4(3a), 
> > RELEASE SOFTWARE (fc2) Technical Support:
> > http://www.cisco.com/techsupport Copyright (c) 1986-2005 by Cisco 
> > Systems, Inc.
> > Compiled Fri 30-Sep-05 07:50 by hqluong Image text-base: 0x6001104C,
> > data-base: 0x60720000
> > 
> > Port Statistics for unclassified packets is not turned on.
> > 
> >         --Boot log end--
> > 
> > 
> > 
> > Router# #show stacks
> > 
> > Minimum process stacks:
> >  Free/Size   Name
> >  2340/3000   RSP memory size check
> > 11280/12000  DHCPD Receive
> >  5340/6000   Inspect Init Msg
> > 59136/60000  script background loader
> >  5224/6000   CDP Protocol
> >  5028/6000   Clock Server
> > 11004/12000  Router Init
> >  3220/12000  Init
> >  4184/6000   Update prst
> >  4976/6000   DIB error message
> >  5152/6000   RADIUS INITCONFIG
> >  5188/6000   DRMI Master Reg. Slave Process
> >  5136/6000   CEF Reloader
> >  5312/6000   MDFS Reload
> > 34844/36000  TCP Command
> >  5332/6000   BGP Accepter
> >  4892/6000   BGP Open
> >  2276/3000   Rom Random Update Process
> >  9236/12000  IPS SDF Loader
> >  8220/12000  Virtual Exec
> >  8220/12000  Exec
> > 10884/12000  SSH Process
> > 
> > Interrupt level stacks:
> > Level    Called Unused/Size  Name
> >   1  2213677378   6892/9000  Network Interrupt
> >   2     5585183   7676/9000  Network Status Interrupt
> >   3           0   8692/9000  OIR interrupt
> >   4           0   9000/9000  PCMCIA Interrupt
> >   5      171491   8596/9000  Console Uart
> >   6           0   9000/9000  Error Interrupt
> >   7    86692017   8604/9000  NMI Interrupt Handler
> 
> 
> Get a couple of sh stacks and look at the interrupt level calls and 
> see which one is going up the most.
> 
> It may be a bad card.
> 
> > 
> > Router# #show align
> > 
> > No alignment data has been recorded.
> > 
> > No spurious memory references have been recorded.
> > 
> > 
> > Regards,
> > Masood Ahmad Shah
> > 
> > 
> > -----Original Message-----
> > From: Rodney Dunn [mailto:rodunn at cisco.com]
> > Sent: Thursday, July 26, 2007 1:14 AM
> > To: Masood Ahmad Shah
> > Cc: cisco-nsp at puck.nether.net
> > Subject: Re: [c-nsp] 7507 GEIP Controller
> > 
> > What does 'sh diag' say and sh stack and show align.
> > 
> > It's as if interrupts are getting sent to the RSP for some reason.
> > 
> > Does the same thing happen if no cables are attached.
> > 
> > 'sh diag'?
> > 
> > On Thu, Jul 26, 2007 at 12:47:43AM +0500, Masood Ahmad Shah wrote:
> > > I have strange issue with 7507.  I have added a new GEIP 
> > > controller into
> > it.
> > > By adding this GEIP controller the CPU usage has been increased up 
> > > to
> 20%.
> > > The strange thing is that CPU usage is always normal + 20% and 
> > > there is no traffic on this interface ( even if interface is 
> > > shutdown the usage remains same ). And if there is traffic it's 
> > > same J
> > > 
> > >  
> > > 
> > > I can't see the process taking this high CPU utilization if I look 
> > > into CPU usage stats.
> > > 
> > >  
> > > 
> > > What the heck has happened to this? J
> > > 
> > >  
> > > 
> > > Here are CPU stats:
> > > 
> > >  
> > > 
> > > Router# #show proc cpu | excl 0.00%__0.00%__0.00%
> > > 
> > > CPU utilization for five seconds: 15%/13%; one minute: 15%; five
> minutes:
> > > 13%
> > > 
> > >  PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

> > > 
> > >    5      518320     41974      12348  0.00%  0.11%  0.11%   0 Check
> heaps
> > > 
> > > 
> > >   20       66004    924869         71  0.08%  0.01%  0.00%   0 IPC
Seat
> > > Manager
> > > 
> > >   22      204088   1466063        139  0.00%  0.03%  0.02%   0 ARP
Input
> > > 
> > > 
> > >   31        7080      1609       4400  0.98%  1.44%  0.59%   2 Virtual
> > Exec
> > > 
> > > 
> > >   52      117672      5843      20138  0.00%  0.02%  0.00%   0
> Per-minute
> > > Jobs
> > > 
> > >   73     3586004  17990635        199  0.82%  0.60%  0.49%   0 IP
Input
> > > 
> > > 
> > >  189       38964    213398        182  0.00%  0.01%  0.00%   0 IP SNMP
> > > 
> > > 
> > >  191       56184    104952        535  0.00%  0.01%  0.00%   0 SNMP
> ENGINE
> > > 
> > > 
> > >  196      160560   1397067        114  0.08%  0.01%  0.00%   0 BGP
> Router
> > > 
> > > 
> > >  197       49200    250208        196  0.00%  0.01%  0.00%   0 BGP I/O
> > > 
> > >  
> > > 
> > > IOS version:
> > > 
> > >  
> > > 
> > > IOS Version 12.4(3a)
> > > 
> > >  
> > > 
> > > Controller
> > > 
> > > Router# sh ver | inc GEIP
> > > 
> > >  
> > > 
> > > 1 GEIP controller (1 GigabitEthernet).
> > > 
> > > 1 GEIP controller (1 GigabitEthernet)
> > > 
> > >  
> > > 
> > >  
> > > 
> > > Regards,
> > > 
> > > Masood Ahmad Shah
> > > 
> > >  
> > > 
> > > _______________________________________________
> > > cisco-nsp mailing list  cisco-nsp at puck.nether.net 
> > > https://puck.nether.net/mailman/listinfo/cisco-nsp
> > > archive at http://puck.nether.net/pipermail/cisco-nsp/

More information about the cisco-nsp mailing list