<div dir="ltr"><div dir="ltr"><div dir="ltr"><br><div>The outage blog highlights something I've put into nearly religious practice,</div><div>ever since I learned it from Sean Doran so many years ago.</div><div><br></div><div>When doing traffic engineering, *always* use values that are</div><div>*below* (less preferred than) the "default" setting for that knob,</div><div>so that you are "pushing" traffic away from nodes, rather than</div><div>"pulling" traffic towards; that way, in case a match clause is </div><div>mistakenly deactivated like this, at worst, you fall back to the </div><div>default value, and traffic spreads across all nodes at the default</div><div>level, rather than become a violent attractor like this.</div><div><br></div><div>I suspect that's exactly what this bullet point in the</div><div>post-mortem is about:</div><div><br></div><div><ul style="box-sizing:border-box;max-width:100%;margin:0px;font-size:20px;padding-left:1.3em;padding-right:1.5em;list-style-position:initial;width:760px;line-height:1.6;color:rgb(54,57,58);font-family:-apple-system,BlinkMacSystemFont,"avenir next",avenir,"helvetica neue",helvetica,ubuntu,roboto,noto,"segoe ui",arial,sans-serif"><li style="box-sizing:border-box;padding-bottom:1em;max-width:34em;margin:0.5em 0px;padding-left:0.3em;line-height:1.6em">Change the BGP local-preference for local server routes. This change will prevent a single location from attracting other locations’ traffic in a similar manner. This change has been deployed following the incident.</li></ul><div><br></div></div><div><br></div><div>Kudos to the Cloudflare team for recognizing the </div><div>danger in "strong attractors" and moving away from</div><div>them. I'd council anyone else doing traffic engineering</div><div>with similarly strong knobs to consider doing likewise;</div><div>shift your policies so that your traffic engineering consists</div><div>of de-preferencing sites and pathways away from default</div><div>value, rather than raising *above* the default value.</div><div><br></div><div>Good work on being so open and transparent,</div><div>Cloudflare team--we should all have management</div><div>that is so honest. :)</div><div><br></div><div>Matt</div><div><br></div><div>(not crossposting to NANOG, though I am debating with</div><div>myself if it might not be worth sharing my "lesson learned"</div><div>with that list as well, for any networks that might have a</div><div>similar ticking time bomb waiting to go off...)</div><div><br></div><div><br></div><div><br></div><div><br></div><div> </div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jul 17, 2020 at 10:24 PM DaZZa <<a href="mailto:dazzagibbs@gmail.com">dazzagibbs@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="auto">RFO here, if anyone is interested<div dir="auto"><br></div><div dir="auto"><p dir="ltr"><a href="https://blog.cloudflare.com/cloudflare-outage-on-july-17-2020/" rel="noreferrer" target="_blank">https://blog.cloudflare.com/cloudflare-outage-on-july-17-2020/</a></p><p dir="ltr">D</p></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, 18 Jul 2020, 9:20 am Pete Templin, <<a href="mailto:petelists@templin.org" rel="noreferrer" target="_blank">petelists@templin.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">It caused some potions? That's some magical stuff!<br>
<br>
On 7/17/20 3:11 PM, Daniel Marks via Outages wrote:<br>
> From Cloudflare:<br>
><br>
> "This afternoon we saw an outage across some parts of our network. It was not as a result of an attack. It appears a router on our global backbone announced bad routes and caused some potions of the network to not be available. We believe we have addressed the root cause and monitoring systems for stability now."<br>
><br>
> Daniel Marks<br>
> Systems Administrator<br>
> May Mobility</blockquote></div><br>
</blockquote></div></div>