<div dir="ltr">This is a stand-alone switch in a cabinet so no L2 loop there. Pretty simple setup -- single BGP session with an upstream provider with the default route pointing right to them. CPU utilization currently sitting at 1%.<div>
<br></div><div>Initially when I noticed the packet loss I thought I was getting DoS attacked, but I have sFlow monitoring activated on all ports and don't see anything out of the ordinary.</div><div><br></div><div>I'll check the boot time diagnostics soon -- thanks for your input.</div>
<div><br></div><div>- Elliot</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, May 7, 2014 at 4:28 PM, Jeroen Wunnink | Hibernia Networks <span dir="ltr"><<a href="mailto:jeroen.wunnink@atrato.com" target="_blank">jeroen.wunnink@atrato.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>Could be a L2 loop or ddos against the
mgmt IP. is the CPU load also high?<div><div class="h5"><br>
<br>
On 07/05/14 20:46, <a href="mailto:ebradsha@gmail.com" target="_blank">ebradsha@gmail.com</a> wrote:<br>
</div></div></div>
<blockquote type="cite"><div><div class="h5">
<div dir="ltr"><span style="font-family:arial,sans-serif;font-size:13px">Hi all,</span>
<div style="font-family:arial,sans-serif;font-size:13px"><br>
</div>
<div style="font-family:arial,sans-serif;font-size:13px">I
believe I have a failing switch on my hands and I'm wondering
if you might be able to provide an assessment based on the
symptoms I've seeing.</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br>
</div>
<div style="font-family:arial,sans-serif;font-size:13px">I'm
currently running a Foundry FESX648-PREM with the following
version info:</div>
<div style="font-family:arial,sans-serif;font-size:13px">
<br>
</div>
<div>
<div><font face="arial, sans-serif">SSH@FESX648
Router>show version</font></div>
<div><font face="arial, sans-serif"> SW: Version
07.4.00eT3e3 Copyright (c) 1996-2012 Brocade
Communications Systems, Inc. All rights reserved.</font></div>
<div><font face="arial, sans-serif"> Compiled on
Dec 11 2013 at 19:00:43 labeled as SXR07400e</font></div>
<div><font face="arial, sans-serif"> (4593059
bytes) Primary sxr07400e.bin</font></div>
<div>
<font face="arial, sans-serif"> BootROM: Version
07.4.01T3e5 (FEv2)</font></div>
<div><font face="arial, sans-serif"> HW: Stackable
FESX648-PREM6 (PROM-TYPE FESX648-L3U-IPV6)</font></div>
<div><font face="arial, sans-serif">==========================================================================</font></div>
<div><font face="arial, sans-serif"> Serial #:
FL18090011</font></div>
<div><font face="arial, sans-serif"> License:
SX_V6_HW_ROUTER_IPv6_SOFT_PACKAGE (LID: XXXXXXXXXXX)</font></div>
<div>
<font face="arial, sans-serif"> P-ASIC 0: type 0111,
rev 00 subrev 01</font></div>
<div><font face="arial, sans-serif"> P-ASIC 1:
type 0111, rev 00 subrev 01</font></div>
<div><font face="arial, sans-serif"> P-ASIC 2:
type 0111, rev 00 subrev 01</font></div>
<div><font face="arial, sans-serif"> P-ASIC 3:
type 0111, rev 00 subrev 01</font></div>
<div><font face="arial, sans-serif">==========================================================================</font></div>
<div><font face="arial, sans-serif"> 300 MHz Power
PC processor 8245 (version 0081/1014) 66 MHz bus</font></div>
<div><font face="arial, sans-serif"> 512 KB boot
flash memory</font></div>
<div><font face="arial, sans-serif"> 8192 KB code
flash memory</font></div>
<div><font face="arial, sans-serif"> 256 MB DRAM</font></div>
<div><font face="arial, sans-serif">The system uptime
is 26 minutes 49 seconds</font></div>
<div><font face="arial, sans-serif">The system :
started=warm start reloaded=by "reload"</font></div>
<div style="font-family:arial,sans-serif;font-size:13px">
<br>
</div>
</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br>
</div>
<div style="font-family:arial,sans-serif;font-size:13px">Quick
summary of the symptoms:</div>
<div style="font-family:arial,sans-serif;font-size:13px">
<br>
</div>
<div style="font-family:arial,sans-serif;font-size:13px">1.
These problems started only after ~15 servers were connected
to the switch. Although many servers were connected,
utilization remains low, only ~40Mbit on a 1Gbit uplink.</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br>
</div>
<div style="font-family:arial,sans-serif;font-size:13px">2. I
just rebooted my switch 20 minutes ago, but I'm already seeing
a ton of FCS errors across many ports: <a href="http://pbrd.co/SABLtk" target="_blank">http://pbrd.co/SABLtk</a><br>
</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br>
</div>
<div style="font-family:arial,sans-serif;font-size:13px">3.
Inexplicably high and erratic ping times (80ms, instead of the
usual 20ms over the same route and variation of +- 20ms on
every ping). Ping times were low and stable before many
servers were connected.</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br>
</div>
<div style="font-family:arial,sans-serif;font-size:13px">4. High
packet loss. Before a lot of servers were connected, there was
no packet loss. Yesterday, the packet loss was hovering around
10%. It seems to be worsening now. Today the average packet
loss is 20%.</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br>
</div>
<div style="font-family:arial,sans-serif;font-size:13px">Screen
capture: <a href="http://pbrd.co/SABZ3D" target="_blank">http://pbrd.co/SADKO7</a><br>
</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br>
</div>
<div style="font-family:arial,sans-serif;font-size:13px">5.
Yesterday I was also able to temporarily eliminate packet loss
and the high ping times by disabling specific ports. Today,
disabling ports 7 and 11 has no effect.</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br>
</div>
<div style="font-family:arial,sans-serif;font-size:13px">6. The
cross-connect cables were suspect, but all cables have since
been tested with a MicroTest PentaScanner and all passed. We
even replaced the CAT5 cross-connect with a machined and
molded CAT6 cable -- the same packet loss and erratic ping
times persisted.</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br>
</div>
<div style="font-family:arial,sans-serif;font-size:13px">7.
Other strange things have happened. Yesterday I attempted to
connect up two new servers to the switch on port 37 and 38.
Ports 5-48 belong to the same default VLAN. The servers could
connect to the switch, and ping the gateway IP, but they could
not ping to the outside world. I then moved the CAT5 cables to
ports 22 and 23 -- same VLAN -- and everything worked
perfectly.</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br>
</div>
<div style="font-family:arial,sans-serif;font-size:13px">Does
this seem like a failing switch? Are there any further
diagnostic tests I could run to verify this?</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br>
</div>
<div style="font-family:arial,sans-serif;font-size:13px">Thanks,</div>
<div style="font-family:arial,sans-serif;font-size:13px">Elliot</div>
<div><br>
</div>
</div>
<br>
<fieldset></fieldset>
<br>
</div></div><pre>_______________________________________________
foundry-nsp mailing list
<a href="mailto:foundry-nsp@puck.nether.net" target="_blank">foundry-nsp@puck.nether.net</a>
<a href="http://puck.nether.net/mailman/listinfo/foundry-nsp" target="_blank">http://puck.nether.net/mailman/listinfo/foundry-nsp</a></pre><span class="HOEnZb"><font color="#888888">
</font></span></blockquote><span class="HOEnZb"><font color="#888888">
<br>
<br>
<pre cols="72">--
Jeroen Wunnink
IP NOC Manager - Hibernia Networks
<a href="mailto:jeroen.wunnink@hibernianetworks.com" target="_blank">jeroen.wunnink@hibernianetworks.com</a>
Phone: <a href="tel:%2B1%20908%20516%204200" value="+19085164200" target="_blank">+1 908 516 4200</a> (Ext: 1011)
24/7 NOC Phone: <a href="tel:%2B31%2020%2082%2000%20623" value="+31208200623" target="_blank">+31 20 82 00 623</a></pre>
</font></span></div>
</blockquote></div><br></div>