[c-nsp] Cisco TAC issues - can someone from Cisco enlighten me onthis?

[asaudale at gmail.com]

THE BEST way to work with TAC and possibly anticipate failure in your case is (assuming you don't have backup 6500 to load the intended image in lab) :


1. Open up the TAC case 3 - 4 hrs before upgrading through the web (open w/ P2)

2. Provide all necessary information through web (these information will go through system before reaching the correct guy) 

3. If an engineer has been assigned, call him up and tell him about your upgrade plan. (In this time, you can ensure yourself there's not any communication issue)  If you're not happy with the assigned engineer, call duty manager to get native speaker. 

4. Only after you've found TAC engineer you're comfortable (tech & language) with, get him to understand your plan (details), and get him remote access to console your 6500. Don't forget to get his desk number as well. If everything is settled, ask him to wait when the maintenance window comes  and leave the case  as P2. 

5. You upgrade 6500 during the window, when problems come. Call up the engineer, tell your problem and if necessary ask him to console in to your core switch.(Or ask him right away)
 
I'm sure you'll get appropriate TAC help within minutes.


Asa







Powered by Telkomsel BlackBerry®

-----Original Message-----
From: "Steve Fischer" <sfischer1967 at gmail.com>
Date: Sun, 20 Sep 2009 17:41:08 
To: <cisco-nsp at puck.nether.net>
Subject: [c-nsp] Cisco TAC issues - can someone from Cisco enlighten me on
	this?

Last Thursday evening, at around midnight, in the course of my organizations
network maintenance, we had not one but two of our core 6500 switches go
into ROMMON (after being rebooted with new code, and being operational for
approximately 45 minutes)at the same time and for no apparent reason.
Attempts to reboot the devices were in vain, and attempts to roll-back also
appeared to be in vain, so I called the Cisco TAC and opened a P1 case.
Immediately, the call was routed over to India.  I was in a loud data
center, and the engineers accent was very thick, to the point I could not
hear him over the background noise, much less understand him.  Other than
asking for a webex session - made impossible by the fact that the network
core is down, he offers nothing in the way of assistance.

 

I asked to have the case transferred to a native-English speaking engineer.
Call transferred to Indian engineer #2, and the communications issues
persist.  I have two core switches down, and am becoming more than a little
concerned.  Same result - engineer really offers nothing in the way of
assistance, and I again, request the call to be transferred to a
native-English speaking engineer.  Enter Indian engineer #3.  Now let me
state here for the record that I am in no way questioning the competence of
the three gentlemen I spoke to, nor do I have any xenophobic tendencies, but
I would like to make a few points here:

 

1.       If I cannot understand the support engineer, it will be difficult
for him to assist me, regardless of his skill level.

2.       Having a native-English speaking engineer available would have been
at this time very disarming, and calming in the midst of for what was for me
a crisis.  In the medical field, they call it bed-side manner, which would
have been of immense value given the crisis I was facing.

3.       My organization spends well over $100K annually in Cisco
maintenance.

 

Case transferred to Indian engineer #4.  Now, while this was occurring, I
called Cisco's TAC and asked the case be re-queued to an engineer in North
America.  I was told that there were no support engineers on duty in North
America.  Now, I'm getting upset, and more than just a little. Also, in the
meantime, it was suggested that I remove one of the CompactFlash cards from
one of the 6500's that was still working (we have 4 total), and try to boot
from the IOS image on it.  Upon ejecting the Flash card, that 6500 too, went
immediately into ROMMON.  So, now, we have 3 of 4 core switches down.  The
entire data center is down, and are one step away from the phone system
going down as well - which indeed did happen.  As we now have all four cores
down, the options of rebooting them with the old code. One by one, through
all four cores, they are rolled back, and finally the network comes up.  Let
me say the fourth engineer suggested this, by prior to that, I had concluded
this was going to be the best course of action.

 

Now, back up two weeks.  I had a Cisco Works issue at around 3:00PM EST, and
open a case for it.  The call is transferred to.wait for it.India.  So, it
doesn't appear that the time of an issue completely influences to what Cisco
support center a call is routed.  As a matter of fact, the support engineer
for that particular call informed me it was 2:00AM where he was.  

 

This leads me to several questions that perhaps someone from Cisco
monitoring this forum could answer.

 

1.       Given the stature of the 6500 platform within Cisco's product line,
and given the severity of the issue I was experiencing, is there no Cisco
Support Center in North America staffed 24X7X365 to deal with such issues?

2.       Was the only option available to me a webex session?  It seemed
strange that the engineer would even ask for that, given that this was
reported as a network down emergency.

3.       Upon asking the call be transferred to a native English speaking
engineer, why was the call routed to 3 more Indian engineers? Rather than
informing me that no native English-speaking engineers were available, why
did I have to request this three more times?  I find it INCREDIBLE that an
organization of Cisco's size could not find a single native English-speaking
engineer to assist me in this crisis concerning their flagship product.

4.       Does Cisco as a company understand the value of being disarming to
a stressed-out client in the midst of a crisis, and how much providing a
support engineer who can clearly and effectively communicate with the
customer?

 

If I were to grade Cisco on this support call, and to my knowledge, this was
the first network down call I've opened since I've been with this
organization, I'd give Cisco an F.no, an F-.  This was an epic-fail for
Cisco, to the point that my organization is considering dumping their
support all together, and going with a third party.  There is just no way I
should have been bounced to four Indian engineers when after the first one,
and the second one, I calmly explained the rationale behind requesting a
native English-speaking engineer.  

 

What can Cisco do to make this right?

 

1.       A written letter of apology.  I don't blame the engineers for this
- they are likely highly competent support engineers.  Cisco failed to
provide an engineer who could communicate with me clearly and concisely. 

2.       Provide an additional one year of maintenance coverage on all four
of our 6500's free of charge, as that pales by comparison to the amount of
money lost by my company for the two hours our network was down, and I was
unable to reach an engineer who could effectively communicate with me.

3.       Staff a support center in North America 24X7X365 geared to deal
with exactly the issue I encountered.  A P1 case should be treated
differently than a P3 case, and this case wasn't, at least not from my
perspective.

4.       When it became clear that the communications issues that existed
between the support engineer and myself precluded progress being made to
resolve the issue, and I requested the case be escalated, escalate the case.
I DO blame the engineer for that - not escalating the case when requested to
do so.    

5.       Stop making the first course of action being requesting a webex.
There will be scenarios where that won't be possible, as was the case
Thursday evening.  Engineer 1 seemed to give up when webex wasn't an option.

 

I am interested in any and all feedback from the community on this.  If
there is someone within Cisco (other than my salesperson, who's heard this
before from me.on more than one occasion) who I can send this to, and can
respond to it, it would also be appreciated.

_______________________________________________
cisco-nsp mailing list  cisco-nsp at puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/