[c-nsp] When do you upgrade IOS?

[Justin M. Streiner]

On Thu, 16 Feb 2012, Rick Martin wrote:

> We are looking at our internal processes and we do not have a 
> documented process for IOS upgrades. I am interested in learning what 
> drives other enterprise network techs or engineers to upgrade IOS. Our 
> current undocumented process is to only upgrade routers and switches 
> that support mission critical applications when driven to by one of the 
> following;
>
> 1.    Security vulnerability
> 2.    Known or discovered bug that has potential of, or is currently, 
> impacting the device
> 3.    Feature enhancement for required or desirable feature.
>
> Beyond these we typically do not upgrade code on network devices that 
> support mission critical applications that we host. When we do run into 
> one of the triggers above we typically evaluate the risk of leaving the 
> current IOS vrs the impact of an outage while the code is upgraded and 
> we base a "go/no go" decision on that.

I would also add "vendor-mandated obsolescence" to that list.  Cisco 
sunsets software in much the same way they sunset hardware.  They will 
make an end-of-life announcement for software version XYZ, stating the 
dates that engineering releases will stop, and the date that TAC support 
will cease.  In some environments, running supportable code is less of an 
issue, but our environment pretty much requires that we be in a position 
to ask for vendor support if we determine we need it, which puts some 
burden on us to remain 'supportable'.

We pretty much do the same thing as you do, and part of what gets baked 
into that process is:

1. Making a determination of the issue we're trying to solve can be done 
by staying in the same major/minor rev and release train of code, or if we 
have to jump to something new.

2. Assessing any changes that need to be made to the configuration as a 
result of the upgrade.  Syntax can change, and features can be 
added/changed/deprecated over time.

3. Determining if the security vulnerability can be mitigated by a 
workaround, short of a code upgrade.

4. Reviewing the release notes of the potential new software version, and 
trying it out in our lab and development environments.

5. Running the devices through a battery of QA/acceptance tests.

6. Using a "slow-start" rollout schedule.  In other words, upgrade a 
lightly-used router first, see how it behaves, and then ramp up the number 
of upgrades that are done over a series of regular maintenance windows 
until all devices are upgraded.

> So my questions are;
>
> 1.    What drives you to upgrade code on critical routers & switches?

When we need to :)

> 2.    Do you upgrade code on a fixed schedule if not driven to by some 
> other immediate requirement?

Other than as noted above, normally not.

There is always the possibility of something requiring quick action in 
some cases, i.e. a really nasty bug is uncovered that has no workaround, 
is remotely exploitable, can compromise the device (force a reload, or 
allow access to the device or its configuration) and has been seen in the 
wild.  We would follow the same process as above, but the end-to-end time 
would be greatly compressed.  I don't like ramming upgrades into place 
without testing, but sometimes vendors don't leave us much choice :(

jms