[nsp] Syslog best practices.

[Furnish, Trever G]

I've just recently begun work on an infrastructure to consolidate our syslog
messages so that processing can happen on one system.  Switching to
syslog-ng wasn't an option for me (I wish it were), but splitting lines out
based on the fully-qualified domain name of the source of the syslog
messages is working well.

I'll describe my setup just for the sake of offering additional food for
thought.  The code's still too company-specific to release at the moment.

The system is just a few shell scripts.  It works like so:
Each logging device logs to a syslog server that's local - either a unix
server running syslogd or a windows server running kiwi's syslog daemon.
The syslog traffic doesn't cross the WAN.

These files are then gathered via secury copy each night.  They're massaged
just enough to strip the syslog-generated timestamps and replace them with
the cisco-generated timestamps (unless the cisco timestamp isn't
synchronized, in which case it'll be preceded with an asterisk).

Once the logs have been massaged they're split into one file per monitored
device and the hostname is removed from each line.  Any lines that don't
match a device are reported separately as a warning (since they won't be
processed further).

>From there, each device-specific logfile gets summarized with a script that
mostly has lines telling it what can be ignored.  Anything it hasn't been
told to ignore is what gets reported on, summarized if possible.  That way
if something new pops up, you still don't miss it.

The log summaries are currently sent via email, but only to the admins
listed for each device - ie a remote admin only sees reports for his
switches.  They also get dropped into a spot on a web server for the remote
admin.

One problem I've run into is difficulty figuring out when a router or switch
is supposed to log a message.  For example, I have one switch that reported
errors over a thousand times in one day and yet showed almost zero errors on
its interface stats.  Another reported over 200,000 interface up/downs as
link-flaps for one port in one day - but the user reported no problems.  I
found plenty of info on Cisco's site about setting up syslog, but none to
explain that behavior.  Any advice would be appreciated.

Best of luck on your endeavor.

> -----Original Message-----
> From: James Kilton [mailto:kilton9 at yahoo.com]
> Sent: Saturday, January 25, 2003 5:09 PM
> To: cisco-nsp at puck.nether.net
> Subject: Re: [nsp] Syslog best practices.
> 
> 
> Thanks for the helpful responses on this, folks.  Very
> much appreciated.
> 
> On Fri, Jan 24, 2003 at 07:22:36AM -0800, James
> Kilton wrote:
> > > I'm preparing to deploy a few Syslog servers to
> > > receive logs from our Cisco devices, and I'm
> > wondering
> > > how people typically handle having only 8 Syslog
> > > facilities to use per server when there are more
> > than
> > > 8 Cisco devices on the network.  Do you just have
> > all
> > > Cisco devices write to the same file?  Do you
> > split it
> > > up randomly?  Or maybe have 1 file per criticality
> > > level?
> > 
> 
> 
> __________________________________________________
> Do you Yahoo!?
> Yahoo! Mail Plus - Powerful. Affordable. Sign up now.
> http://mailplus.yahoo.com
> _______________________________________________
> cisco-nsp mailing list  cisco-nsp at puck.nether.net
> http://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
>