[VoiceOps] Crazy bouncy month

[anorexicpoodle]

Well i recently got bitten by a small issue where the Acme BCP setting
in the media manager for "Max Signaling Bandwidth" wasn't high enough to
support our burst traffic, so it would drop signaling traffic during
spikes. Of course this is a global setting so the traffic that it
dropped was not even isolated to a single realm, but across the entire
SD, and would often include keep-alive traffic. All the traffic policing
in the media manager is implemented in the network processor so short of
being in full debug mode, it generates absolutely no messages at all
that it is doing this. Traffic just goes into a black hole and peers
seem to fall off the earth for a while. Of course those thinking
creatively will quickly realize this will generate retransmissions,
which will drive up the signaling traffic and make the problem worse.

I'm not saying this is your issue, but i chased BGP updates, upstream
routing issues, switching issues, solar flares, etc forever trying to
figure out what would cause this. Once i figured out this was the
problem i just cranked the setting to max, and let my trust mode
settings handle DOS protection, since its realm specific and far better
suited to the task. 

On Fri, 2010-08-20 at 08:04 -0700, Beth Johnson wrote:

> Acme SBCs: moderately loaded, very healthy, very stable.
> 
> All traffic graphs show no interesting increase or decrease in total
> traffic; there is no traffic-volume based attack detectable by our
> equipment.
> 
> A couple of RFOs have discovered planned, unannounced changes that
> occurred.  Maybe everyone has decided to sneak upgrades and other
> changes in before school starts and they lose that summer intern on
> which everything can be conveniently blamed?  Interns and dogs are
> most handy in that regard...
> 
> Hopefully, however, its just the NSA dropping in more Einstein taps.
> 
> 
> -B
> 
> 
> 
> On Fri, Aug 20, 2010 at 7:27 AM, anorexicpoodle
> <anorexicpoodle at gmail.com> wrote:
> 
>         What SBC and what kinds of traffic volume? I saw a similarly
>         bizarre issue on mine.
>         
>         
>         
>         
>         On Fri, 2010-08-20 at 06:50 -0700, Beth Johnson wrote:
>         
>         > 
>         > Hi Groupers,
>         > This has been a wild month over here.  Our peer SBC has been
>         > lit up with alarms with bounces from this peer or that.
>         > We'll lose one gateway from a peer for a minute or so, then
>         > it'll come right back up.  Sometimes the SIP bounce is
>         > accompanied with a BGP route flap, sometimes not.  It almost
>         > always happens late at night, but a few sneak in during the
>         > day.
>         > 
>         > We've got plenty of redundancy to cover the events, so ASR
>         > remains high.
>         > 
>         > Part of our root-cause analysis is seeking an uderstanding
>         > of the experiences of similar networks, so I need to ask...
>         > Is anyone else seeing a general increase in internet-based
>         > peer bounces?
>         > 
>         > -B
>         > 
>         > 
>         > 
>         > _______________________________________________
>         > VoiceOps mailing list
>         > VoiceOps at voiceops.org
>         > https://puck.nether.net/mailman/listinfo/voiceops
>         
>         
>         
> 
> 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/voiceops/attachments/20100820/20a1e8b9/attachment.html>