[c-nsp] 12.4(11)T QoS problems (again)

Tue Jul 31 08:30:45 EDT 2007

Thanks Tony.

Make note of the overal pps going through the box pre/post upgrade.

sh int | incl packets
sh proc cpu sort | excl 0.00

Rodney

On Tue, Jul 31, 2007 at 01:36:12AM -0700, Tony Varriale wrote:
> We have one G1 that was just implemented with 15T1.  We also have 2 running 
> 11T and 1 running 9T.  The 2 running 11T have, IMO, very high CPU (60+% 
> average durning middle of day).  The 9T barely runs 20%.
> 
> I am planning on upgrading one of the 11Ts to see if the CPU issue was 
> resolved.  I will post up when I get some data.
> 
> tv
> ----- Original Message ----- 
> From: "Rodney Dunn" <rodunn at cisco.com>
> To: "Rafi Sadowsky" <rafi at noc.ilan.net.il>
> Cc: <cisco-nsp at puck.nether.net>; "Hank Nussbacher" <hank at efes.iucc.ac.il>
> Sent: Monday, July 30, 2007 8:15 AM
> Subject: Re: [c-nsp] 12.4(11)T QoS problems (again)
> 
> 
> > On Sat, Jul 28, 2007 at 07:18:24AM +0300, Rafi Sadowsky wrote:
> >> Hi Rodney,
> >>
> >>  FYI: The router is a 7206VXR NPE-G1 with most of traffic between 2 of 
> >> the
> >> NPE GbE interfaces
> >
> > Ok. Good because the CPU change if there is one significant would
> > show up between the two versions.
> >
> >>
> >> Any particular reason you mentioned a NPE-G2 ?
> >
> > Search the archives for the NPE-G2 measurement discussions
> > we've had.
> >
> > Rodney
> >
> >>
> >>
> >> -- 
> >> Regards
> >> Rafi
> >>
> >> [sh ver]
> >> Cisco IOS Software, 7200 Software (C7200-ADVIPSERVICESK9-M), Version
> >> 12.4(4)T1, RELEASE SOFTWARE (fc4)
> >> Technical Support: http://www.cisco.com/techsupport
> >> Copyright (c) 1986-2005 by Cisco Systems, Inc.
> >> Compiled Wed 21-Dec-05 22:58 by ccai
> >>
> >> ROM: System Bootstrap, Version 12.3(4r)T3, RELEASE SOFTWARE (fc1)
> >> BOOTLDR: 7200 Software (C7200-KBOOT-M), Version 12.3(15), RELEASE 
> >> SOFTWARE
> >> (fc3)
> >>
> >> Tel-Aviv uptime is 1 week, 5 days, 1 minute
> >> System returned to ROM by reload at 07:07:17 IDT Mon Jul 16 2007
> >> System restarted at 07:09:50 IDT Mon Jul 16 2007
> >> System image file is "disk2:c7200-advipservicesk9-mz.124-4.T1.bin"
> >> Last reload reason: Reload Command
> >>
> >> Cisco 7206VXR (NPE-G1) processor (revision B) with 491520K/32768K bytes 
> >> of
> >> memory.
> >> Processor board ID 26812591
> >> SB-1 CPU at 700MHz, Implementation 1025, Rev 0.2, 512KB L2 Cache
> >> 6 slot VXR midplane, Version 2.6
> >>
> >> Last reset from power-on
> >>
> >> [snipped]
> >>
> >> PCI bus mb1 (Slots 1, 3 and 5) has a capacity of 600 bandwidth points.
> >> [snipped]
> >> PCI bus mb2 (Slots 2, 4 and 6) has a capacity of 600 bandwidth points.
> >> [snipped]
> >>
> >>
> >> 4 Ethernet interfaces
> >> 2 FastEthernet interfaces
> >> 3 Gigabit Ethernet interfaces
> >> 8 Serial interfaces
> >> 1 Packet over SONET interface
> >> 509K bytes of NVRAM.
> >>
> >> 125952K bytes of ATA PCMCIA card at slot 2 (Sector size 512 bytes).
> >> 16384K bytes of Flash internal SIMM (Sector size 256K).
> >> Configuration register is 0x2102
> >>
> >> -- 
> >> Regards,
> >> Rafi
> >>
> >> On Fri, 27 Jul 2007, Rodney Dunn wrote:
> >>
> >> >Date: Fri, 27 Jul 2007 08:52:26 -0400
> >> >From: Rodney Dunn <rodunn at cisco.com>
> >> >To: Hank Nussbacher <hank at efes.iucc.ac.il>
> >> >Cc: cisco-nsp at puck.nether.net, nocplus at noc.ilan.net.il
> >> >Subject: Re: [c-nsp] 12.4(11)T QoS problems (again)
> >> >
> >> >I'd be interested to know this as well. I helped draft the FN on this
> >> >one to make sure they were clear.
> >> >
> >> >In the fixes that went in 15T1 there was really only one, rather large,
> >> >change to improve the dataplane performance. Others were mostly bug
> >> >functionality problems.
> >> >
> >> >The NPE-G2 performance comparisons are harder given our previous
> >> >discussions around how it does CPU measurements.
> >> >
> >> >Rodney
> >> >
> >> >
> >> >On Fri, Jul 27, 2007 at 07:10:06AM +0300, Hank Nussbacher wrote:
> >> >>Cisco Field Notice: Field Notice: FN - 62854 - Cisco IOS - Performance
> >> >>Degradation and Reloads Possible When Using Quality of Service Features 
> >> >>in
> >> >>Version 12.4(11)T - July 26, 2007
> >> >>
> >> >>http://www.cisco.com/en/US/customer/products/ps6441/products_field_notice09186a008088cc2f.shtml
> >> >>
> >> >>"Workaround/Solution
> >> >>There is no workaround that avoids all possible symptoms. A complete
> >> >>solution is expected to be available in Cisco IOS version 12.4(15)T1. 
> >> >>To
> >> >>avoid the symptoms, users may downgrade to a pre-12.4(11)T release or
> >> >>upgrade to 12.4(15)T1 when it becomes available. Cisco strongly
> >> >>discourages deploying an IOS 12.4(11)T release with QoS features 
> >> >>enabled
> >> >>due to the potential severity of symptoms."
> >> >>
> >> >>We went to 12.4(15)T around the beginning of July (it is now a Deferred
> >> >>Release) and it did *not* help so we went back to 12.4(4)T1 which does 
> >> >>fix
> >> >>the performance problems.  One can see the exact change dates via this
> >> >>MRTG CPU graph: http://noc.ilan.net.il/stats/ILAN-CPU/tau-7206-cpu.html
> >> >>
> >> >>I see 12.4(15)T1 came out July 20.  Has anyone tested it (non-Cisco) 
> >> >>and
> >> >>found that it does solve all performance problems?
> >> >>
> >> >>-Hank
> >> >>
> >> >>_______________________________________________
> >> >>cisco-nsp mailing list  cisco-nsp at puck.nether.net
> >> >>https://puck.nether.net/mailman/listinfo/cisco-nsp
> >> >>archive at http://puck.nether.net/pipermail/cisco-nsp/
> >> >
> > _______________________________________________
> > cisco-nsp mailing list  cisco-nsp at puck.nether.net
> > https://puck.nether.net/mailman/listinfo/cisco-nsp
> > archive at http://puck.nether.net/pipermail/cisco-nsp/ 
> 
> _______________________________________________
> cisco-nsp mailing list  cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/