[c-nsp] Acceptance Test Procedure for New Cisco Devices

[Phil Mayers]

Ziv Leyes wrote:
> Ok, let me be more specific When we buy devices for our own use, we
> just open it, plug it, and start using them, if there are any
> problems, we call the provider and they fix the problem (RMA or
> whatever) In this case, we're going to sell the equipment as a kind
> of turn-key project, and the customer asked us to provide them with
> "our" ATP, which we don't really use for ourselves, so I'd like to
> implement one sort of testing procedure from now on for this type of
> cases. We're going to attach this to a legal statement so we can't
> just type some BS there and that's it, we want to actually implement
> it, and if we write we do a,b,c,d then we'll going to do a,b,c,d
> procedure for real. I was thinking some of you guys may already use
> this kind of test routines and can help me creating one. I don't need
> some really serious stuff, I can imagine I'll check the delivery
> status of the package, open it, check all the contents that need to
> be there are there, to plug the device and see it works, perhaps load
> some configuration, plug the hardware that is planned to hold if any
> (HWICS and so), perform some soft and hard reboots, see the device
> responds, there are links on all interfaces, and pack it back exactly
> as it was. The problem is I don't know how exactly write it down on a
> kind of form that there's a checkbox for each test. Does anybody have
> some ready to go stuff?

Well, it's going to depend very much on the kind of equipment.

For example, a mandatory step when we get anything for our 6500s is a 
complete run passing all GOLD tests (including the disruptive tests). We 
maintain a spare chassis specifically for this.

I don't know if ASA5510 and 1861 have diagnostics, but I don't think so. 
In that case, you're probably going to want something like:

  * Build a standard config involving (at least) your ASA & 18xx router, 
which all or a large subset of the features are enabled

  * For each pair of devices you distribute, load the standard config on 
and run some test traffic

  * Leave it powered up for long enough to count as "burn in" i.e. 7 days?

So you'd write something like:

"""Party X will undertake to:

  * Unpack all equipment and check inventory
  * Check that equipment will power up
  * Load on a standard config, which tests:
    * OSPF routing
    * BGP routing
    * Packet forwarding
    * IPSec
    * Coffee making
  * Run test traffic for 48 hours, to ensure the devices compare to a 
known-good platform
  * Leave the config running for 7 days, to eliminate early-life failure

...before shipping to Customer Y"""