[j-nsp] Juniper support vs. Cisco TAC - experiences?

[Richard A Steenbergen]

On Wed, Jul 09, 2008 at 06:25:07PM -0600, David Ball wrote:
>   Another thing which I (and surely most of you) try to do is inundate
> them with data when opening a case right at the get-go, in hopes of
> avoiding the mundane questions Richard alluded to.  Not just a 'req
> sup info', but attach any core files (/var/crash I think), many of the
> log files found in /var/log/ (not just 'messages'), the 'traceoptions'
> log for the problem/protocol/interface you're working on, a network
> diagram, and a rediculously detailed explanation of precisely what
> you're seeing, what else you've tried during troubleshooting, and what
> those results were.  It will take much longer to open the ticket, but
> in my experience it shaves days off the resolution time.  I've even
> had a JTAC person reply with 'wow, thanks for all the great
> information when opening the ticket...I'm escalating to ATAC now' when
> they realized the issue was clearly out of their comfort zone.

I have an op script which helps with some of this monotony by uploading 
log files to the FTP site (*), but there are a few nagging software issues 
which keep it from being a "good" solution. For example, there is no 
scriptable command to MKDIR on an ftp site, nor is there a way to run a 
CLI command and save the results to a temp file (in non-XML format) so you 
can upload it. Why do you need this? For example, "show config | display 
omit | display inheritence | display commit", since 99% of regular JTAC is 
completely incapable of understanding commit scripts, group inheritence, 
or apply-flags omit. At best you get to run a few commands manually, save 
them to temp files in known locations, and then use the script to automate 
a few upload commands.

This particular issue is actually my primary example of why having such 
wonderful op script functionality can sometimes be a bad thing. I've been 
trying to submit a feature request to get a built in method to securely 
upload support information for literally years now, and with the addition 
of op scripts the universal cop-out is "we don't need to do this, you can 
do it yourself with op scripts". Sounds great in theory, except that you 
can't actually do most of the work required due to a few missing commands. 
Requiring an op script also puts the functionality beyond the reach of the 
vast majority of users, who don't happen to be reading this list or 
browsing juniper.cluepon.net. But even more importantly, I wonder how many 
people actually stop to think about the security of the data they are 
transmitting over the wire. Not only are you uploadig the entire contents 
of your router's configuration in plain-text, but often times you are 
trasmitting the entire contents of your memory as part of core dumps. If 
someone wanted to do some industrial espionage its hard to think of a 
better place to start than by intercepting some packets going to 
ftp.juniper.net. A proper JUNOS implementation, in addition to being able 
to upload all kinds of things that you can't do with op scripts, would 
distribute the public key cert file for Juniper's ftp server as part of 
JUNOS, then sftp the data securely.

Op scripts are great things, but sometimes they aren't a substitute for 
the real thing, and this would be so simple to implement and beneficial 
for users that it absolutely boggles the mind why it keeps getting 
rejected.

At any rate, here is the not-really-working-well version of the script if 
anyone wants to play with it. FYI the point of the temp file is so that 
you can grab files off of any RE, including the "other RE" which might not 
have connectivity to anythig except the active RE via TNP. Another good 
addition would be the uploading of an arbitrary file, or the uploading of 
the results of arbitrary pfe commands, incase you need to gather more data 
than the default log files. I thought I had uploaded this to 
juniper.cluepon.net but I must have been waiting until I could get it 
fixed. :)

version 1.0;
ns junos = "http://xml.juniper.net/junos/*/junos";
ns xnm = "http://xml.juniper.net/xnm/1.1/xnm";
ns jcs = "http://xml.juniper.net/junos/commit-scripts/1.0";
import "../import/junos.xsl";

template upload($file, $dest) {
    var $filename = jcs:regex("[^\/]+$", $file)[1];
    var $destfile =  $hostname _ "." _ $filename;
    var $tmpfile = "/tmp/" _ $filename _ "XXXFIXMEWITHREALTMPNAME";

    /* Copy specified file to a temporary local location */
    var $copy_temp = {
        <file-copy> {
            <source> $file;
            <destination> $tmpfile;
        }
    }

    var $results_temp = jcs:invoke($copy_temp); 
    <output> $results_temp;
     

    /* Upload file from temp location to the remote server */
    var $copy_upload = {
        <file-copy> {
            <source> $tmpfile;
            <destination> $dest _ "/" _ $destfile;
        }
    }   

    <output> "Uploading " _ $filename _ " to " _ $dest;
    var $results_upload = jcs:invoke($copy_upload);
    <output> $results_upload;


    /* Clean up the mess left behind */
    var $delete = {
        <file-delete> {
            <path> $tmpfile;
        }   
    }       
            
    var $results_delete = jcs:invoke($delete);
}    


/* Requires JUNOS 8.4+, and still doesn't work right */
template save($data, $file) {
    var $put = {
        <file-put> {
            <filename> $file;
            <encoding> "ascii";
            <file-contents> $data;
        }
    }

    var $results_put = jcs:invoke($put);
    <output> "Debug: " _ $results_put;
}

var $arguments = {
    <argument> {
        <name> "destination";
        <description> "Destination Path";
    }
    <argument> {
        <name> "case";
        <description> "Juniper Support Case Number";
    }
    <argument> {
        <name> "routing-engine";
        <description> "Routing Engine";
    }
}

param $case;
param $destination;
param $routing-engine;

/* Set defaults for some arguments */
var $dest = jcs:first-of($destination, "ftp://ftp.juniper.net/pub/incoming/");
/* XXX Fix this with "which RE am I" code */
var $re = jcs:first-of($routing-engine, "re0");

match / {
    <op-script-results> {
        if ($case) {
            <output> "Automated version currently unsupported, run these first:";
            <output> "request support information | save /tmp/support";
            <output> "show config | display omit | display commit | save /tmp/config";

            /*
            var $support = jcs:invoke('get-support-information');
            var $config  = jcs:invoke('get-configuration');
            call save($support, "/tmp/support");
            call save($config, "/tmp/config");
            */

            call upload($file = $re _ ":/tmp/support", $dest = $dest _ $case);
            call upload($file = $re _ ":/tmp/config", $dest = $dest _ $case);
            call upload($file = $re _ ":/var/log/messages", $dest = $dest _ $case);
            call upload($file = $re _ ":/var/log/chassisd", $dest = $dest _ $case);
        } else {
            <xnm:error> {
                <message> "Juniper Support Case Number argument required.";
            }
        }
    }
}


(*) If you can even get them to look at the FTP site. I still run into 
JTAC people who don't know it exists, don't know how to check it, or 
insist that you copy and paste everything into an email attachment or 
webpage upload form anyways. Also beware of large file handlig, I know for 
certain that whatever method they use to access the FTP site contents 
doesn't handle un-tgz'ing a kernel coredump from an RE with 4GB of ram 
(well that and regular JTAC doesn't know what a .tgz is, but I'll stop 
complaining about them for today :P).

-- 
Richard A Steenbergen <ras at e-gerbil.net>       http://www.e-gerbil.net/ras
GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)