[c-nsp] Cisco TAC issues - can someone from Cisco enlighten me onthis?

Mon Sep 21 00:26:43 EDT 2009

Richard - 

This would be more acceptable (at least to me), were this an issue with a
3560 switch, or a 2800 series router, but this was 2 core switches of their
flagship product, the 6500.  Enterprise data centers throughout the US. Like
the one at my organization, rely heavily on this product, and it should be
supported as such.  I understand the problem, but given the criticality of
these devices as they relate to the core infrastructure of so many
organizations, transferring the call to India is not an acceptable way of
dealing with it.  

This is an embarrassment for an organization of this size to provide a
substandard level of support, when the support is being paid for, and at the
level it is being paid for.  

If I had waited a few hours...I would have been fired, and rightfully so.

What I also feel really bad about is this must have been frustrating for the
four Indian engineers as well.  Cisco put them in a position where they
really couldn't be successful in helping me, due to something that isn't
their fault.  It wouldn't be fair to transfer their P1 network emergency
calls to me, either.  There is a communication barrier that makes conversing
difficult.

I view the maintenance contracts like I view car insurance - I'm paying for
something I hope I never need.  But, a situation arose Thursday evening
where I did need it, and the service wasn't available in a manner that could
assist me.  At a minimum, a refund or credit should be forthcoming.

To be fair to Cisco as a whole, my local account team is picking this case
up and running with it.  But, that doesn't absolve the TAC of malfeasance. 

-----Original Message-----
From: rgolodner at infratection.com [mailto:rgolodner at infratection.com] 
Sent: Sunday, September 20, 2009 6:17 PM
To: Steve Fischer
Subject: Re: [c-nsp] Cisco TAC issues - can someone from Cisco enlighten me
onthis?

I have had the same problem. Basically TAC follows the sun. If you had
waited a few hours you would have had an Irish tech. It is a problem and not
just with Cisco
       Richard
Sent via BlackBerry from T-Mobile

-----Original Message-----
From: "Steve Fischer" <sfischer1967 at gmail.com>

Date: Sun, 20 Sep 2009 17:41:08 
To: <cisco-nsp at puck.nether.net>
Subject: [c-nsp] Cisco TAC issues - can someone from Cisco enlighten me on
	this?

Last Thursday evening, at around midnight, in the course of my organizations
network maintenance, we had not one but two of our core 6500 switches go
into ROMMON (after being rebooted with new code, and being operational for
approximately 45 minutes)at the same time and for no apparent reason.
Attempts to reboot the devices were in vain, and attempts to roll-back also
appeared to be in vain, so I called the Cisco TAC and opened a P1 case.
Immediately, the call was routed over to India.  I was in a loud data
center, and the engineers accent was very thick, to the point I could not
hear him over the background noise, much less understand him.  Other than
asking for a webex session - made impossible by the fact that the network
core is down, he offers nothing in the way of assistance.

I asked to have the case transferred to a native-English speaking engineer.
Call transferred to Indian engineer #2, and the communications issues
persist.  I have two core switches down, and am becoming more than a little
concerned.  Same result - engineer really offers nothing in the way of
assistance, and I again, request the call to be transferred to a
native-English speaking engineer.  Enter Indian engineer #3.  Now let me
state here for the record that I am in no way questioning the competence of
the three gentlemen I spoke to, nor do I have any xenophobic tendencies, but
I would like to make a few points here:

1.       If I cannot understand the support engineer, it will be difficult
for him to assist me, regardless of his skill level.

2.       Having a native-English speaking engineer available would have been
at this time very disarming, and calming in the midst of for what was for me
a crisis.  In the medical field, they call it bed-side manner, which would
have been of immense value given the crisis I was facing.

3.       My organization spends well over $100K annually in Cisco
maintenance.

Case transferred to Indian engineer #4.  Now, while this was occurring, I
called Cisco's TAC and asked the case be re-queued to an engineer in North
America.  I was told that there were no support engineers on duty in North
America.  Now, I'm getting upset, and more than just a little. Also, in the
meantime, it was suggested that I remove one of the CompactFlash cards from
one of the 6500's that was still working (we have 4 total), and try to boot
from the IOS image on it.  Upon ejecting the Flash card, that 6500 too, went
immediately into ROMMON.  So, now, we have 3 of 4 core switches down.  The
entire data center is down, and are one step away from the phone system
going down as well - which indeed did happen.  As we now have all four cores
down, the options of rebooting them with the old code. One by one, through
all four cores, they are rolled back, and finally the network comes up.  Let
me say the fourth engineer suggested this, by prior to that, I had concluded
this was going to be the best course of action.

Now, back up two weeks.  I had a Cisco Works issue at around 3:00PM EST, and
open a case for it.  The call is transferred to.wait for it.India.  So, it
doesn't appear that the time of an issue completely influences to what Cisco
support center a call is routed.  As a matter of fact, the support engineer
for that particular call informed me it was 2:00AM where he was.  

This leads me to several questions that perhaps someone from Cisco
monitoring this forum could answer.

1.       Given the stature of the 6500 platform within Cisco's product line,
and given the severity of the issue I was experiencing, is there no Cisco
Support Center in North America staffed 24X7X365 to deal with such issues?

2.       Was the only option available to me a webex session?  It seemed
strange that the engineer would even ask for that, given that this was
reported as a network down emergency.

3.       Upon asking the call be transferred to a native English speaking
engineer, why was the call routed to 3 more Indian engineers? Rather than
informing me that no native English-speaking engineers were available, why
did I have to request this three more times?  I find it INCREDIBLE that an
organization of Cisco's size could not find a single native English-speaking
engineer to assist me in this crisis concerning their flagship product.

4.       Does Cisco as a company understand the value of being disarming to
a stressed-out client in the midst of a crisis, and how much providing a
support engineer who can clearly and effectively communicate with the
customer?

If I were to grade Cisco on this support call, and to my knowledge, this was
the first network down call I've opened since I've been with this
organization, I'd give Cisco an F.no, an F-.  This was an epic-fail for
Cisco, to the point that my organization is considering dumping their
support all together, and going with a third party.  There is just no way I
should have been bounced to four Indian engineers when after the first one,
and the second one, I calmly explained the rationale behind requesting a
native English-speaking engineer.  

What can Cisco do to make this right?

1.       A written letter of apology.  I don't blame the engineers for this
- they are likely highly competent support engineers.  Cisco failed to
provide an engineer who could communicate with me clearly and concisely. 

2.       Provide an additional one year of maintenance coverage on all four
of our 6500's free of charge, as that pales by comparison to the amount of
money lost by my company for the two hours our network was down, and I was
unable to reach an engineer who could effectively communicate with me.

3.       Staff a support center in North America 24X7X365 geared to deal
with exactly the issue I encountered.  A P1 case should be treated
differently than a P3 case, and this case wasn't, at least not from my
perspective.

4.       When it became clear that the communications issues that existed
between the support engineer and myself precluded progress being made to
resolve the issue, and I requested the case be escalated, escalate the case.
I DO blame the engineer for that - not escalating the case when requested to
do so.    

5.       Stop making the first course of action being requesting a webex.
There will be scenarios where that won't be possible, as was the case
Thursday evening.  Engineer 1 seemed to give up when webex wasn't an option.

I am interested in any and all feedback from the community on this.  If
there is someone within Cisco (other than my salesperson, who's heard this
before from me.on more than one occasion) who I can send this to, and can
respond to it, it would also be appreciated.

_______________________________________________
cisco-nsp mailing list  cisco-nsp at puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/