[c-nsp] Re: Netflow traffic study

[Kevin Vuong]

Although I don't have a copy of the scripts anymore (they were  
written at a previous company and their property), here's what we did.

Basically we started out by using Flowscan (http://net.doit.wisc.edu/ 
~plonka/FlowScan/) to gather the netflow data from our routers in 5  
minute intervals.  That will give you some general graphs of the  
usage patterns on your network.

 From there, we used flow-tools (http://www.splintered.net/sw/flow- 
tools/) to process the netflow data to look for what we wanted.  For  
spam/virus type traffic, I would look at traffic pattern types along  
with connection thresholds.  When someone would violate the policies  
specified (say 10,000 emails in a 5 minute period from residential IP  
space), I would drop them into a database.  A separate script would  
update ACLs by aging old entries and inserting new violators.  If a  
customer had not fixed their computers yet when their entry aged,  
then it would get popped back on there next run.  Support would have  
a web tool that could query this database to find if a customer  
complaint was due to abuse or an actual trouble.  While this method  
isn't signature based, we avoided false-positives with the thresholds  
that were specified.

I also wrote a web page which was a front end for flow-tools.  It  
basically generated a config file on the fly to query the flowscan  
data based on whatever criteria you wanted.  For example, if a  
customer called complaining about latency or something, then a tech  
could fill out a web form for something like .. from src x.x.x.x/x   
and pick a timeperiod (from flowscan files).  From there they would  
be able to see all traffic from that IP space to see if maybe they  
had some rogue computers doing a bunch of port scans or spamming.   
For the form, I believe i had used src/dst ip, src/dst port, type  
(tcp,udp,icmp, etc), and timeperiod.  That way people could get  
relatively detailed with their reports.

-Kevin



On Dec 29, 2005, at 12:19 AM, Gerry Boudreaux wrote:

> In a previous job, we were analyzing netflow data to look for unusual
> patterns, like resi customers making too many smtp connections in a 5
> minute window, or portscans, etc....
>
> most were custom written perl scripts, but we used the results of our
> analysis to push filters to specific customer interfaces, and then
> expired them after two weeks, assuming that either the customer would
> have fixed the problem, or the scanner would re-capture them.
>
> We also pushed the results to a searchable web-page for support to
> use to troubleshoot issues like "Why cannot I sent e-mail?"  um well,
> you tried to send 50000 messages in a 5 minute window, and you are a
> resi customer, you might have a virus...
>
> By limiting unwanted traffic, you might save money by not needing
> additional upstream bandwidth.
>
> Just one possibility on how you can manipulate netflow data.  Use
> your imagination.
>
> G
>
>
> On Dec 28, 2005, at 9:57 PM, Kanagaraj Krishna wrote:
>
>> Hi,
>>    I'm working for a medium size ISP providing transit services to
>> our customers. We are currently in the process of expanding our
>> network
>> which includes addition to our upstream providers. In the process
>> of identifying where to put our money into, we want to study our
>> customers
>> traffic pattern (destination, type of traffic, region etc) before
>> deciding on the most suitable  upstream provider that fits the bill
>> (coverage and quality). One of the option that we are looking into
>> in doing these is the Netflow function on cisco routers. I have a few
>> questions regarding this issue:
>>
>> - Any good (free/open source) software that can analyze (stats,
>> graph etc) Netflow data?
>> - Any comments on the use of Netflow for this purpose?
>> - Any other suggestions in reaching our objectives (other than
>> Netflow)?
>>
>> Hope to get input from you guys out there. Thanks.
>>
>> Regards,
>> Kanagaraj Krishna
>> _______________________________________________
>> cisco-nsp mailing list  cisco-nsp at puck.nether.net
>> https://puck.nether.net/mailman/listinfo/cisco-nsp
>> archive at http://puck.nether.net/pipermail/cisco-nsp/
>>
>
> _______________________________________________
> cisco-nsp mailing list  cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
>