<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">We saw the problem and we have redundant connections to Level 3 in Atlanta..........<div><br></div><div><br><div><div>On Oct 20, 2009, at 1:31 PM, Bill Wichers wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div lang="EN-US" link="blue" vlink="purple"><o:smarttagtype namespaceuri="urn:schemas-microsoft-com:office:smarttags" name="country-region"><o:smarttagtype namespaceuri="urn:schemas-microsoft-com:office:smarttags" name="State"><o:smarttagtype namespaceuri="urn:schemas-microsoft-com:office:smarttags" name="City"><o:smarttagtype namespaceuri="urn:schemas-microsoft-com:office:smarttags" name="place"><div class="Section1"><div style="margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Courier New"><span style="font-size: 10pt; font-family: 'Courier New'; ">It's possible the faulty device was an aggregation router and only those customers connected to it saw the outage. Redundancy within Level(3)'s network would not have helped customers with only a single connection off the affected device. At some point it becomes the customer's responsibility to redundantly connect to their upstreams (as in buy two connections on different aggregation routers :-).<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Courier New"><span style="font-size: 10pt; font-family: 'Courier New'; "><o:p> </o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Courier New"><span style="font-size: 10pt; font-family: 'Courier New'; ">I didn't see any L3 issues up here in<span class="Apple-converted-space"> </span><st1:city w:st="on"><st1:place w:st="on">Detroit</st1:place></st1:city>, so whatever outage L3 had was apparently contained within that one POP or region. No cascade failures I can see...<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Courier New"><span style="font-size: 10pt; font-family: 'Courier New'; "><o:p> </o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Courier New"><span style="font-size: 10pt; font-family: 'Courier New'; "> -Bill<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Courier New"><span style="font-size: 10pt; font-family: 'Courier New'; "><o:p> </o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Courier New"><span style="font-size: 10pt; font-family: 'Courier New'; ">> Level 3 has a single router or switch handling packets at a major POP?<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Courier New"><span style="font-size: 10pt; font-family: 'Courier New'; ">> I doubt this, but the outage is confirmation something bad happened.<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Courier New"><span style="font-size: 10pt; font-family: 'Courier New'; ">><o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Courier New"><span style="font-size: 10pt; font-family: 'Courier New'; ">> That said: where's the redundancy, and why didn't it kick in?<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Courier New"><span style="font-size: 10pt; font-family: 'Courier New'; ">><o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Courier New"><span style="font-size: 10pt; font-family: 'Courier New'; ">> --<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Courier New"><span style="font-size: 10pt; font-family: 'Courier New'; ">> | Jeremy Chadwick <span class="Apple-converted-space"> </span><a href="mailto:jdc@parodius.com" style="color: blue; text-decoration: underline; ">jdc@parodius.com</a><span class="Apple-converted-space"> </span>|<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Courier New"><span style="font-size: 10pt; font-family: 'Courier New'; ">> | Parodius Networking <span class="Apple-converted-space"> </span><a href="http://www.parodius.com/" style="color: blue; text-decoration: underline; ">http://www.parodius.com/</a><span class="Apple-converted-space"> </span>|<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Courier New"><span style="font-size: 10pt; font-family: 'Courier New'; ">> | UNIX Systems Administrator <span class="Apple-converted-space"> </span><st1:place w:st="on"><st1:city w:st="on">Mountain View</st1:city>,<span class="Apple-converted-space"> </span><st1:state w:st="on">CA</st1:state>,<span class="Apple-converted-space"> </span><st1:country-region w:st="on">USA</st1:country-region></st1:place><span class="Apple-converted-space"> </span>|<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman'; "><font size="2" face="Courier New"><span style="font-size: 10pt; font-family: 'Courier New'; ">> | Making life hard for others since 1977. PGP: 4BD6C0CB |<o:p></o:p></span></font></div></div>_______________________________________________<br>outages mailing list<br><a href="mailto:outages@outages.org" style="color: blue; text-decoration: underline; ">outages@outages.org</a><br><a href="https://puck.nether.net/mailman/listinfo/outages" style="color: blue; text-decoration: underline; ">https://puck.nether.net/mailman/listinfo/outages</a><br></o:smarttagtype></o:smarttagtype></o:smarttagtype></o:smarttagtype></div></span></blockquote></div><br></div></body></html>