[c-nsp] subscriber termination issues

[Justin Shore]

Mike wrote:
> Howdy,
>    Lastly, we actually provide great customer support and have built in 
> tools (and have trained folks) to do packet captures in order to 
> identify customer misconfiguration, locked up / insane home routers 
> spewing garbage or disobeying protocol, and other common customer side 
> troubles that keep our phones warm. My linux pppoe server lets my techs 
> look people up by account name and also initiate a dump on that customer 
> (internally, tcpdump on their mac address). I am wondering what or if 
> there are compatible or superior cisco tools for doing same or similar 
> operations in order to more fully support the customer base. This is 
> important - I am not willing to spend 4 hours on the phone diagnosing 
> the fact that the customer has a wrong dns server statically programmed 
> into some device somewhere, I want to be able to do a dump and catch him 
> in the act and then move on to the next call.

Mike,

I'm afraid I can't help you with the first 2 problems though I am 
definitely interested in setting up a walled garden myself for much the 
same reasons you already have.  On the 3rd topic I do have some input. 
I currently do something similar myself with a set of span ports on our 
core routers.  This allows me to capture customer traffic and diagnose 
issues much more effectively without taking the customer's word for what 
is happening (frankly the customer is always wrong when it comes to 
technical details).  I usually leave my monitor sessions set up spanning 
the interfaces connected to our border routers but I swing them around 
as needed to face other devices.  Depending on your network design you 
could do something similar.  That gets your guys back to tcpdump which 
they're already familiar with.  The downside is that they will have to 
pick out the target host by IP (implying that they have to determine the 
IP in advance).  The dump may also not catch the right interfaces (like 
the interface heading to where your DNS server resides) which requires 
the tech to realize this and for someone to have to change the monitor 
session.  The last one can be mitigated with careful network design (ie, 
not using SVIs in your core for server farms and instead routing 
downstream to another server farm router/switch).  Other option is a 
multi-port mirroring appliance.  With it you mirror all links from edge 
devices connecting to your core to a sniffing server.  That's a bit of a 
kludge as well I'm afraid.  Perhaps it will prompt a better idea though.

Justin