[c-nsp] GSR E2 linecard performance woes

Joe Maimon jmaimon at ttec.com
Wed Mar 16 08:19:23 EST 2005


Which can easily start stinking when you have to change every ingress 
filter instead of one egress.

Ah. Typical.

David Freedman wrote:
> Well, I removed the egress ACLs and now the utilisation is at around
> 
> 10%/10%
> 
> unfortunately I don't have and E3 cards in this chassis, and being a 
> 12012 can't put an E4+ in it.
> 
> So what you are saying is that the PSA in the E2 is not able to use
> the output_acl_128 bundle :(
> 
> Its a shame, I now have to re-write my egress ACLs to act on ingress 
> elsewhere on the chassis.....
> 
> 
> Dave.
> 
> 
> Oliver Boehmer (oboehmer) wrote:
> 
>>>The following "bad" event counters are observed to increment largely:
>>>
>>>- "Packets punt to RP" (does this mean RP CPU or LC CPU?)
>>
>>Generally the RP, this is stuff going to the router itself.. Usually the
>>RP takes care of these control-plane packets, but sometimes the LC
>>engine does (like answering "pings").
>>
>>
>>>- some "HW engine reject" (not much)
>>
>>Ok, so this doesn't point to a problem.
>>
>>
>>>Where can I find a list of which features appear as PSA bundles for
>>>this card in different IOS releases?
>>>(trying to avoid parsing *all* the release notes)
>>
>>Hmm, I don't think there is a document describing this in necessary
>>details.. Those bundles don't usually change that much, it could be that
>>we introduce new bundles as new features come along..
>>
>>
>>>For instance, I note that "output_acl_128" is not enabled,
>>>do I need this to maintain performance with egress ACLs?
>>
>>Ah.. Do you have Engine 3 or 4+ LC in your chassis? Then it could be
>>that the LC-CPU is busy processing the egress ACL. The PSA is not
>>available in the egress path. Take a look at
>>http://www.cisco.com/warp/public/63/acl_12000.html for details on how
>>ACLs are processed on the GSR..
>>
>>Otherwise it would be better to contact TAC since we might need more
>>detailed info to troubleshoot what's going on exactly..
>>
>>	oli
>>
>>
>>>Oliver Boehmer (oboehmer) wrote:
>>>
>>>>David,
>>>>
>>>>E2 generally forwards in hardware, so LC-CPU usage is no straight
>>>>indication for LC utilization. LC-CPU is invoked when the hardware
>>>>punts traffic to the LC-CPU, so you want to find out what is
>>>>happening here. Does your loaded PSA bundle support all the features
>>>>you have configured on your LC (the PSA ASIC doing the forwarding is
>>>>loaded with a microcode bundle depending on the feature you have
>>>>enabled. If the loaded bundle does not support all the features,
>>>>pkts are punted to the LC-CPU for switching).. 
>>>>
>>>>"exec slot 5 sh contr psa bund"
>>>>"exec slot 5 sh contr events" (several times to see which counter
>>>>increases) 
>>>>
>>>>might provide more info..
>>>>
>>>>	oli
>>>>
>>>>
>>>>David Freedman <> wrote on Wednesday, March 16, 2005 12:32 PM:
>>>>
>>>>
>>>>>Hiya,
>>>>>
>>>>>AFAIK, the cisco "marketing" packet rate figure for the GSR120XX E2
>>>>>linecards is 4MPPS. 
>>>>>
>>>>>I'm running a 3GE E2 linecard in a 12012 with the following
>>>>>features: 
>>>>>
>>>>>
>>>>>Global -
>>>>>
>>>>>access-list compiled
>>>>>
>>>>>
>>>>>Port 1 -
>>>>>
>>>>>- Ingress ACL (7 lines)
>>>>>- Egress ACL (7 lines)
>>>>>- netflow sampling (interval = 1/1000)
>>>>>- sparse pim
>>>>>- 120MB/Sec ingress
>>>>>- 190MB/Sec egress
>>>>>- 39Kpps ingress
>>>>>- 34Kpps egress
>>>>>
>>>>>
>>>>>Port 2 -
>>>>>
>>>>>- Ingress ACL (7 lines)
>>>>>- Egress ACL (7 lines)
>>>>>- netflow sampling (interval = 1/1000)
>>>>>- sparse pim
>>>>>- 320MB/Sec ingress
>>>>>- 320MB/Sec egress
>>>>>- 49Kpps ingress
>>>>>- 77Kpps egress
>>>>>
>>>>>Port 3 -
>>>>>
>>>>>- Ingress ACL (7 lines)
>>>>>- Egress ACL (7 lines)
>>>>>- netflow sampling (interval = 1/1000)
>>>>>- sparse pim
>>>>>- MPLS (tag-switching) with NO TE
>>>>>- 532MB/Sec ingress
>>>>>- 513MB/Sec egress
>>>>>- 120Kpps ingress
>>>>>- 97Kpps egress
>>>>>
>>>>>
>>>>>The same ACL is used both ingress and egress on all ports.
>>>>>
>>>>>
>>>>>Now look at this:
>>>>>
>>>>>#execute-on slot 5 sh proc cpu | exc 0.00
>>>>>========= Line Card (Slot 5) =========
>>>>>
>>>>>CPU utilization for five seconds: 94%/73%; one minute: 85%; five
>>>>>  minutes: 84% PID Runtime(ms)   Invoked      uSecs   5Sec   1Min  
>>>>>    5Min TTY Process 4   172497132   1297743     132922  1.53% 
>>>>>   0.51%  0.55% 0 BFLC PSA ACL 10    59944212  62202985        963 
>>>>>0.07%  0.14% 
>>>>>0.13%   0 CEF LC
>>>>>IPC Backg
>>>>>   21   114470916  13418571       8530  0.23%  0.23%  0.22%   0
>>>>>   Per-Second Jobs 46    43622128  66352970        657  0.23% 
>>>>>   0.29%  0.29%   0 Queue Mgr 59  2177980408   6086199     357858
>>>>>18.19%  7.89%  7.70%   0 TAG Stats Backgr
>>>>>
>>>>>
>>>>>Why is the card working so hard?
>>>>>Is the marketing data that far off?
>>>>>or am I doing something known to degrade the performance of the
>>>>>card? 
>>>>>
>>>>>Removing the netflow and PIM doesn't really affect the CPU
>>>>>interrupt percentage much..... 
>>>>>
>>>>>Here is the sh diag for the slot:
>>>>>
>>>>>
>>>>>
>>>>>SLOT 5  (RP/LC 5 ): 3 Port Gigabit Ethernet
>>>>>   MAIN: type 68,  800-6376-05 rev B0
>>>>>         Deviation: 0
>>>>>         HW config: 0x00    SW key: 00-00-00
>>>>>   PCA:  73-4775-07 rev C0 ver 2
>>>>>         Design Release 2.0  S/N XXXXXXXXXX
>>>>>   MBUS: Embedded Agent
>>>>>         Test hist: 0x00    RMA#: 00-00-00    RMA hist: 0x00
>>>>>   DIAG: Test count: 0x00000003    Test results: 0x00000000
>>>>>   FRU:  Linecard/Module: 3GE-GBIC-SC=
>>>>>         Route Memory: MEM-GRP/LC-256=
>>>>>         Packet Memory: MEM-LC1-PKT-256=
>>>>>   L3 Engine: 2 - Backbone OC48 (2.5 Gbps)
>>>>>   MBUS Agent Software version 1.86 (RAM) (ROM version is 2.32)   
>>>>>   ROM Monitor version 16.12 Fabric Downloader version used 9.3
>>>>>   (ROM version is 9.3)    Primary clock is CSC 0 Board is analyzed
>>>>>   Board State is Line Card Enabled (IOS  RUN )
>>>>>   Insertion time: 00:09:58 (21w6d ago)
>>>>>   DRAM size: 268435456 bytes
>>>>>   FrFab SDRAM size: 134217728 bytes, SDRAM pagesize: 8192 bytes
>>>>>   ToFab SDRAM size: 134217728 bytes, SDRAM pagesize: 8192 bytes   
>>>>>0 crashes since restart 
>>>>>
>>>>>
>>>>>
>>>>>System is running  12.0(25)S4 and all linecards had firmware
>>>>>upgraded via the "upgrade all" command.
>>>>>
>>>>>Processor is a fully expanded GRP-B.
>>>>>
>>>>>Any help (or even confirmation that I am running the card hot)
>>>>>would be appreciated. 
>>>>>
>>>>>
>>>>>Thanks,
>>>>>
>>>>>Dave.
>>>>>
>>>>>
>>>>>_______________________________________________
>>>>>cisco-nsp mailing list  cisco-nsp at puck.nether.net
>>>>>https://puck.nether.net/mailman/listinfo/cisco-nsp
>>>>>archive at http://puck.nether.net/pipermail/cisco-nsp/
>>>>
>>>>_______________________________________________
>>>>cisco-nsp mailing list  cisco-nsp at puck.nether.net
>>>>https://puck.nether.net/mailman/listinfo/cisco-nsp
>>>>archive at http://puck.nether.net/pipermail/cisco-nsp/
>>>>
>>>
>>>_______________________________________________
>>>cisco-nsp mailing list  cisco-nsp at puck.nether.net
>>>https://puck.nether.net/mailman/listinfo/cisco-nsp
>>>archive at http://puck.nether.net/pipermail/cisco-nsp/
>>
>>_______________________________________________
>>cisco-nsp mailing list  cisco-nsp at puck.nether.net
>>https://puck.nether.net/mailman/listinfo/cisco-nsp
>>archive at http://puck.nether.net/pipermail/cisco-nsp/
>>
> 
> 
> _______________________________________________
> cisco-nsp mailing list  cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
> 
> 


More information about the cisco-nsp mailing list