<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">Try physically pulling the SFM's one by
one rather then just powercycling them. Yes there is a difference
there :-)<br>
<br>
Also, 5400d is fairly buggy, there were some major issues with CRC
checksums in hSFM's and on 100G cards. We have fairly good results
with 5500e<br>
<br>
<br>
<br>
On 18/02/15 16:50, Brad Fleming wrote:<br>
</div>
<blockquote
cite="mid:40A30463-6928-417C-B635-B17FD04F94F0@gmail.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
TAC replaced hSFMs and line cards the first couple times but we’ve
seen this issue at least once on every node in the network. The
ones where we replaced every module (SFM, mgmt, port cards, even
PSUs) have still had at least one event. So I’m not even sure what
hardware we’d replace at this point. That lead us to thinking a
config problem since each box uses the same template but after a
lengthy audit with TAC nobody could find anything. It happens
infrequently enough that we grew to just live with it.
<div class=""><br class="">
</div>
<div class="">
<div class=""><br class="">
</div>
<div class=""><br class="">
<div>
<blockquote type="cite" class="">
<div class="">On Feb 18, 2015, at 12:45 AM, Frank Bulk
<<a moz-do-not-send="true"
href="mailto:frnkblk@iname.com" class="">frnkblk@iname.com</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="WordSection1" style="page: WordSection1;
font-family: Helvetica; font-size: 12px; font-style:
normal; font-variant: normal; font-weight: normal;
letter-spacing: normal; line-height: normal; orphans:
auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows:
auto; word-spacing: 0px; -webkit-text-stroke-width:
0px;">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt;
font-family: 'Times New Roman', serif;" class=""><span
style="font-size: 11pt; font-family: Calibri,
sans-serif; color: rgb(31, 73, 125);" class="">So
don’t errors like this suggest replacing the
hardware?<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt;
font-family: 'Times New Roman', serif;" class=""><span
style="font-size: 11pt; font-family: Calibri,
sans-serif; color: rgb(31, 73, 125);" class=""> </span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt;
font-family: 'Times New Roman', serif;" class=""><span
style="font-size: 11pt; font-family: Calibri,
sans-serif; color: rgb(31, 73, 125);" class="">Frank<o:p
class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt;
font-family: 'Times New Roman', serif;" class=""><span
style="font-size: 11pt; font-family: Calibri,
sans-serif; color: rgb(31, 73, 125);" class=""> </span></div>
<div class="">
<div style="border-style: solid none none;
border-top-color: rgb(225, 225, 225);
border-top-width: 1pt; padding: 3pt 0in 0in;"
class="">
<div style="margin: 0in 0in 0.0001pt; font-size:
12pt; font-family: 'Times New Roman', serif;"
class=""><b class=""><span style="font-size:
11pt; font-family: Calibri, sans-serif;"
class="">From:</span></b><span
style="font-size: 11pt; font-family: Calibri,
sans-serif;" class=""><span
class="Apple-converted-space"> </span>foundry-nsp
[<a moz-do-not-send="true"
href="mailto:foundry-nsp-bounces@puck.nether.net"
class="">mailto:foundry-nsp-bounces@puck.nether.net</a>]<span
class="Apple-converted-space"> </span><b
class="">On Behalf Of<span
class="Apple-converted-space"> </span></b>Brad
Fleming<br class="">
<b class="">Sent:</b><span
class="Apple-converted-space"> </span>Tuesday,
February 17, 2015 3:10 PM<br class="">
<b class="">To:</b><span
class="Apple-converted-space"> </span>Josh
Galvez<br class="">
<b class="">Cc:</b><span
class="Apple-converted-space"> </span><a
moz-do-not-send="true"
href="mailto:foundry-nsp@puck.nether.net"
class="">foundry-nsp@puck.nether.net</a><br
class="">
<b class="">Subject:</b><span
class="Apple-converted-space"> </span>Re:
[f-nsp] MLX throughput issues<o:p class=""></o:p></span></div>
</div>
</div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt;
font-family: 'Times New Roman', serif;" class=""><o:p
class=""> </o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt;
font-family: 'Times New Roman', serif;" class="">The
common symptoms for us are alarms of TM errors /
resets. We’ve been told on multiple TAC cases that
logs indicating transmit TM errors are likely caused
by problems in one of the SFM links / lanes. We’ve
been told that resetting the SFMs one at a time will
clear the issue.<o:p class=""></o:p></div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size:
12pt; font-family: 'Times New Roman', serif;"
class=""><o:p class=""> </o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size:
12pt; font-family: 'Times New Roman', serif;"
class="">Symptoms during the issue is that 1/3rd
of the traffic moving from one TM to another TM
will simply get dropped. So we see TCP globally
start to throttle like crazy and if enough errors
count up the TM will simply reset. After the TM
reset is seems a 50/50 chance the box will remain
stable or go back to dropping packets within
~20mins. So when we see a TM reset we simply do
the SFM Dance no matter what.<o:p class=""></o:p></div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size:
12pt; font-family: 'Times New Roman', serif;"
class=""><o:p class=""> </o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size:
12pt; font-family: 'Times New Roman', serif;"
class=""><o:p class=""> </o:p></div>
<div class="">
<blockquote style="margin-top: 5pt;
margin-bottom: 5pt;" class="" type="cite">
<div class="">
<div style="margin: 0in 0in 0.0001pt;
font-size: 12pt; font-family: 'Times New
Roman', serif;" class="">On Feb 16, 2015,
at 10:08 PM, Josh Galvez <<a
moz-do-not-send="true"
href="mailto:josh@zevlag.com"
style="color: purple; text-decoration:
underline;" class="">josh@zevlag.com</a>>
wrote:<o:p class=""></o:p></div>
</div>
<div style="margin: 0in 0in 0.0001pt;
font-size: 12pt; font-family: 'Times New
Roman', serif;" class=""><o:p class=""> </o:p></div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt;
font-size: 12pt; font-family: 'Times New
Roman', serif;" class="">Why kind of
wigout? And how do you diagnose the
corruption? I'm intrigued.<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt;
font-size: 12pt; font-family: 'Times New
Roman', serif;" class=""><o:p class=""> </o:p></div>
<div class="">
<div style="margin: 0in 0in 0.0001pt;
font-size: 12pt; font-family: 'Times
New Roman', serif;" class="">On Mon,
Feb 16, 2015 at 8:43 AM, Brad Fleming
<<a moz-do-not-send="true"
href="mailto:bdflemin@gmail.com"
target="_blank" style="color:
purple; text-decoration: underline;"
class="">bdflemin@gmail.com</a>>
wrote:<o:p class=""></o:p></div>
<blockquote style="border-style: none
none none solid; border-left-color:
rgb(204, 204, 204); border-left-width:
1pt; padding: 0in 0in 0in 6pt;
margin-left: 4.8pt; margin-right:
0in;" class="" type="cite">
<div style="margin: 0in 0in 0.0001pt;
font-size: 12pt; font-family: 'Times
New Roman', serif;" class="">We’ve
seen it since installing the
high-capacity switch fabrics into
our XMR4000 chassis roughly 4 years
ago. We saw it through IronWare
5.4.00d. I’m not sure what software
we were using when they were first
installed; probably whatever would
have been stable/popular around
December 2010.<br class="">
<br class="">
Command is simply “power-off snm
[1-3]” then “power-on snm [1-3]”.<br
class="">
<br class="">
Note that the power-on process
causes your management session to
hang for a few seconds. The device
isn’t broken and packets aren’t
getting dropped; it’s just going
through checks and echoing back
status.<br class="">
<br class="">
-brad<o:p class=""></o:p></div>
<div class="">
<div class="">
<div style="margin: 0in 0in
0.0001pt; font-size: 12pt;
font-family: 'Times New Roman',
serif;" class=""><br class="">
<br class="">
> On Feb 16, 2015, at 7:07
AM, Jethro R Binks <<a
moz-do-not-send="true"
href="mailto:jethro.binks@strath.ac.uk"
style="color: purple;
text-decoration: underline;"
class="">jethro.binks@strath.ac.uk</a>>
wrote:<br class="">
><br class="">
> On Fri, 13 Feb 2015, Brad
Fleming wrote:<br class="">
><br class="">
>> Over the years we’ve
seen odd issues where one of the<br
class="">
>> switch-fabric-links
will “wigout” and some of the
data moving between<br class="">
>> cards will get
corrupted. When this happens we
power cycle each switch<br
class="">
>> fab one at a time using
this process:<br class="">
>><br class="">
>> 1) Shutdown SFM #3<br
class="">
>> 2) Wait 1 minute<br
class="">
>> 3) Power SFM #3 on
again<br class="">
>> 4) Verify all SFM links
are up to SFM#3<br class="">
>> 5) Wait 1 minute<br
class="">
>> 6) Perform steps 1-5
for SFM #2<br class="">
>> 7) Perform steps 1-5
for SFM #3<br class="">
>><br class="">
>> Not sure you’re seeing
the same issue that we see but
the “SFM Dance”<br class="">
>> (as we call it) is a
once-every-four-months thing
somewhere across our<br class="">
>> 16 XMR4000 boxes. It
can be done with little to no
impact if you are<br class="">
>> patient verify status
before moving to the next SFM.<br
class="">
><br class="">
> That's all interesting.
What code versions is this?
Also, how do you<br class="">
> shutdown the SFMs? I don't
recall seeing documentation for
that.<br class="">
><br class="">
> Jethro.<br class="">
><br class="">
><br class="">
>><br class="">
>>> On Feb 13, 2015, at
11:41 AM,<span
class="Apple-converted-space"> </span><a
moz-do-not-send="true"
href="mailto:nethub@gmail.com"
style="color: purple;
text-decoration: underline;"
class="">nethub@gmail.com</a><span
class="Apple-converted-space"> </span>wrote:<br
class="">
>>><br class="">
>>> We have three
switch fabrics installed, all
are under 1% utilized.<br
class="">
>>><br class="">
>>><br class="">
>>> From: Jeroen
Wunnink | Hibernia Networks
[mailto:<a
moz-do-not-send="true"
href="mailto:jeroen.wunnink@atrato.com"
style="color: purple;
text-decoration: underline;"
class="">jeroen.wunnink@atrato.com</a><span
class="Apple-converted-space"> </span><mailto:<a
moz-do-not-send="true"
href="mailto:jeroen.wunnink@atrato.com"
style="color: purple;
text-decoration: underline;"
class="">jeroen.wunnink@atrato.com</a>>]<br
class="">
>>> Sent: Friday,
February 13, 2015 12:27 PM<br
class="">
>>> To:<span
class="Apple-converted-space"> </span><a
moz-do-not-send="true"
href="mailto:nethub@gmail.com"
style="color: purple;
text-decoration: underline;"
class="">nethub@gmail.com</a><span
class="Apple-converted-space"> </span><mailto:<a
moz-do-not-send="true"
href="mailto:nethub@gmail.com"
style="color: purple;
text-decoration: underline;"
class="">nethub@gmail.com</a>>;
'Jeroen Wunnink | Hibernia
Networks'<br class="">
>>> Subject: Re:
[f-nsp] MLX throughput issues<br
class="">
>>><br class="">
>>> How many
switchfabrics do you have in
that MLX and how high is the
utilization on them<br class="">
>>><br class="">
>>> On 13/02/15 18:12,<span
class="Apple-converted-space"> </span><a
moz-do-not-send="true"
href="mailto:nethub@gmail.com"
style="color: purple;
text-decoration: underline;"
class="">nethub@gmail.com</a><span
class="Apple-converted-space"> </span><mailto:<a
moz-do-not-send="true"
href="mailto:nethub@gmail.com"
style="color: purple;
text-decoration: underline;"
class="">nethub@gmail.com</a>>
wrote:<br class="">
>>>> We also tested
with a spare Quanta LB4M we have
and are seeing about the same
speeds as we are seeing with the
FLS648 (around 20MB/s or
160Mbps).<br class="">
>>>><br class="">
>>>> I also reduced
the number of routes we are
accepting down to about 189K and
that did not make a difference.<br
class="">
>>>><br class="">
>>>><br class="">
>>>> From:
foundry-nsp [mailto:<a
moz-do-not-send="true"
href="mailto:foundry-nsp-bounces@puck.nether.net"
style="color: purple;
text-decoration: underline;"
class="">foundry-nsp-bounces@puck.nether.net</a><span
class="Apple-converted-space"> </span><mailto:<a
moz-do-not-send="true"
href="mailto:foundry-nsp-bounces@puck.nether.net"
style="color: purple;
text-decoration: underline;"
class="">foundry-nsp-bounces@puck.nether.net</a>>]
On Behalf Of Jeroen Wunnink |
Hibernia Networks<br class="">
>>>> Sent: Friday,
February 13, 2015 3:35 AM<br
class="">
>>>> To:<span
class="Apple-converted-space"> </span><a
moz-do-not-send="true"
href="mailto:foundry-nsp@puck.nether.net"
style="color: purple;
text-decoration: underline;"
class="">foundry-nsp@puck.nether.net</a><span
class="Apple-converted-space"> </span><mailto:<a
moz-do-not-send="true"
href="mailto:foundry-nsp@puck.nether.net"
style="color: purple;
text-decoration: underline;"
class="">foundry-nsp@puck.nether.net</a>><br
class="">
>>>> Subject: Re:
[f-nsp] MLX throughput issues<br
class="">
>>>><br class="">
>>>> The FLS
switches do something weird with
packets. I've noticed they
somehow interfere with changing
the MSS window size dynamically,
resulting in destinations
further away having very poor
speed results compared to
destinations close by.<br
class="">
>>>><br class="">
>>>> We got rid of
those a while ago.<br class="">
>>>><br class="">
>>>><br class="">
>>>> On 12/02/15
17:37,<span
class="Apple-converted-space"> </span><a
moz-do-not-send="true"
href="mailto:nethub@gmail.com"
style="color: purple;
text-decoration: underline;"
class="">nethub@gmail.com</a><span
class="Apple-converted-space"> </span><mailto:<a
moz-do-not-send="true"
href="mailto:nethub@gmail.com"
style="color: purple;
text-decoration: underline;"
class="">nethub@gmail.com</a>>
wrote:<br class="">
>>>>> We are
having a strange issue on our
MLX running code 5.6.00c. We
are encountering some throughput
issues that seem to be randomly
impacting specific networks.<br
class="">
>>>>><br class="">
>>>>> We use the
MLX to handle both external BGP
and internal VLAN routing. Each
FLS648 is used for Layer 2 VLANs
only.<br class="">
>>>>><br class="">
>>>>> From a
server connected by 1 Gbps
uplink to a Foundry FLS648
switch, which is then connected
to the MLX on a 10 Gbps port,
running a speed test to an
external network is getting
20MB/s.<br class="">
>>>>><br class="">
>>>>> Connecting
the same server directly to the
MLX is getting 70MB/s.<br
class="">
>>>>><br class="">
>>>>> Connecting
the same server to one of my
customer's Juniper EX3200 (which
BGP peers with the MLX) also
gets 70MB/s.<br class="">
>>>>><br class="">
>>>>> Testing to
another external network, all
three scenarios get 110MB/s.<br
class="">
>>>>><br class="">
>>>>> The path to
both test network locations goes
through the same IP transit
provider.<br class="">
>>>>><br class="">
>>>>> We are
running NI-MLX-MR with 2GB RAM,
NI-MLX-10Gx4 connect to the
Foundry FLS648 by XFP-10G-LR,
NI-MLX-1Gx20-GC was used for
directly connecting the server.
A separate NI-MLX-10Gx4 connects
to our upstream BGP providers.
Customer’s Juniper EX3200
connects to the same
NI-MLX-10Gx4 as the FLS648. We
take default routes plus full
tables from three providers by
BGP, but filter out most of the
routes.<br class="">
>>>>><br class="">
>>>>> The fiber
and optics on everything look
fine. CPU usage is less than
10% on the MLX and all line
cards and CPU usage at 1% on the
FLS648. ARP table on the MLX is
about 12K, and BGP table is
about 308K routes.<br class="">
>>>>><br class="">
>>>>> Any
assistance would be
appreciated. I suspect there is
a setting that we’re missing on
the MLX that is causing this
issue.<br class="">
>>>>><br class="">
>>>>><br class="">
>>>>><br class="">
>>>>><br class="">
>>>>>
_______________________________________________<br
class="">
>>>>> foundry-nsp
mailing list<br class="">
>>>>><span
class="Apple-converted-space"> </span><a
moz-do-not-send="true"
href="mailto:foundry-nsp@puck.nether.net"
style="color: purple;
text-decoration: underline;"
class="">foundry-nsp@puck.nether.net</a><span
class="Apple-converted-space"> </span><mailto:<a
moz-do-not-send="true"
href="mailto:foundry-nsp@puck.nether.net"
style="color: purple;
text-decoration: underline;"
class="">foundry-nsp@puck.nether.net</a>><br
class="">
>>>>><span
class="Apple-converted-space"> </span><a
moz-do-not-send="true"
href="http://puck.nether.net/mailman/listinfo/foundry-nsp"
target="_blank" style="color:
purple; text-decoration:
underline;" class="">http://puck.nether.net/mailman/listinfo/foundry-nsp</a><span
class="Apple-converted-space"> </span><<a
moz-do-not-send="true"
href="http://puck.nether.net/mailman/listinfo/foundry-nsp"
target="_blank" style="color:
purple; text-decoration:
underline;" class="">http://puck.nether.net/mailman/listinfo/foundry-nsp</a>><br
class="">
>>>><br class="">
>>>><br class="">
>>>><br class="">
>>>> --<br class="">
>>>><br class="">
>>>> Jeroen Wunnink<br
class="">
>>>> IP NOC Manager
- Hibernia Networks<br class="">
>>>> Main numbers
(Ext: 1011): USA<span
class="Apple-converted-space"> </span><a
moz-do-not-send="true"
href="tel:%2B1.908.516.4200"
style="color: purple;
text-decoration: underline;"
class="">+1.908.516.4200</a><span
class="Apple-converted-space"> </span>|
UK<span
class="Apple-converted-space"> </span><a
moz-do-not-send="true"
href="tel:%2B44.1704.322.300"
style="color: purple;
text-decoration: underline;"
class="">+44.1704.322.300</a><br
class="">
>>>> Netherlands<span
class="Apple-converted-space"> </span><a
moz-do-not-send="true"
href="tel:%2B31.208.200.622"
style="color: purple;
text-decoration: underline;"
class="">+31.208.200.622</a><span
class="Apple-converted-space"> </span>|
24/7 IP NOC Phone:<span
class="Apple-converted-space"> </span><a
moz-do-not-send="true"
href="tel:%2B31.20.82.00.623"
style="color: purple;
text-decoration: underline;"
class="">+31.20.82.00.623</a><br
class="">
>>>><span
class="Apple-converted-space"> </span><a
moz-do-not-send="true"
href="mailto:jeroen.wunnink@hibernianetworks.com"
style="color: purple;
text-decoration: underline;"
class="">jeroen.wunnink@hibernianetworks.com</a><span
class="Apple-converted-space"> </span><mailto:<a
moz-do-not-send="true"
href="mailto:jeroen.wunnink@hibernianetworks.com"
style="color: purple;
text-decoration: underline;"
class="">jeroen.wunnink@hibernianetworks.com</a>><br
class="">
>>>><span
class="Apple-converted-space"> </span><a
moz-do-not-send="true"
href="http://www.hibernianetworks.com/"
target="_blank" style="color:
purple; text-decoration:
underline;" class="">www.hibernianetworks.com</a><span
class="Apple-converted-space"> </span><<a
moz-do-not-send="true"
href="http://www.hibernianetworks.com/"
target="_blank" style="color:
purple; text-decoration:
underline;" class="">http://www.hibernianetworks.com/</a>><br
class="">
>>><br class="">
>>><br class="">
>>> --<br class="">
>>><br class="">
>>> Jeroen Wunnink<br
class="">
>>> IP NOC Manager -
Hibernia Networks<br class="">
>>> Main numbers (Ext:
1011): USA<span
class="Apple-converted-space"> </span><a
moz-do-not-send="true"
href="tel:%2B1.908.516.4200"
style="color: purple;
text-decoration: underline;"
class="">+1.908.516.4200</a><span
class="Apple-converted-space"> </span>|
UK<span
class="Apple-converted-space"> </span><a
moz-do-not-send="true"
href="tel:%2B44.1704.322.300"
style="color: purple;
text-decoration: underline;"
class="">+44.1704.322.300</a><br
class="">
>>> Netherlands<span
class="Apple-converted-space"> </span><a
moz-do-not-send="true"
href="tel:%2B31.208.200.622"
style="color: purple;
text-decoration: underline;"
class="">+31.208.200.622</a><span
class="Apple-converted-space"> </span>|
24/7 IP NOC Phone:<span
class="Apple-converted-space"> </span><a
moz-do-not-send="true"
href="tel:%2B31.20.82.00.623"
style="color: purple;
text-decoration: underline;"
class="">+31.20.82.00.623</a><br
class="">
>>><span
class="Apple-converted-space"> </span><a
moz-do-not-send="true"
href="mailto:jeroen.wunnink@hibernianetworks.com"
style="color: purple;
text-decoration: underline;"
class="">jeroen.wunnink@hibernianetworks.com</a><span
class="Apple-converted-space"> </span><mailto:<a
moz-do-not-send="true"
href="mailto:jeroen.wunnink@hibernianetworks.com"
style="color: purple;
text-decoration: underline;"
class="">jeroen.wunnink@hibernianetworks.com</a>><br
class="">
>>><span
class="Apple-converted-space"> </span><a
moz-do-not-send="true"
href="http://www.hibernianetworks.com/"
target="_blank" style="color:
purple; text-decoration:
underline;" class="">www.hibernianetworks.com</a><span
class="Apple-converted-space"> </span><<a
moz-do-not-send="true"
href="http://www.hibernianetworks.com/"
target="_blank" style="color:
purple; text-decoration:
underline;" class="">http://www.hibernianetworks.com/</a>>_______________________________________________<br
class="">
>>> foundry-nsp mailing
list<br class="">
>>><span
class="Apple-converted-space"> </span><a
moz-do-not-send="true"
href="mailto:foundry-nsp@puck.nether.net"
style="color: purple;
text-decoration: underline;"
class="">foundry-nsp@puck.nether.net</a><span
class="Apple-converted-space"> </span><mailto:<a
moz-do-not-send="true"
href="mailto:foundry-nsp@puck.nether.net"
style="color: purple;
text-decoration: underline;"
class="">foundry-nsp@puck.nether.net</a>><br
class="">
>>><span
class="Apple-converted-space"> </span><a
moz-do-not-send="true"
href="http://puck.nether.net/mailman/listinfo/foundry-nsp"
target="_blank" style="color:
purple; text-decoration:
underline;" class="">http://puck.nether.net/mailman/listinfo/foundry-nsp</a><span
class="Apple-converted-space"> </span><<a
moz-do-not-send="true"
href="http://puck.nether.net/mailman/listinfo/foundry-nsp"
target="_blank" style="color:
purple; text-decoration:
underline;" class="">http://puck.nether.net/mailman/listinfo/foundry-nsp</a>><br
class="">
>><br class="">
><br class="">
> . . . . . . . . .
. . . . . . . . . . .
. . . . .<br class="">
> Jethro R Binks, Network
Manager,<br class="">
> Information Services
Directorate, University Of
Strathclyde, Glasgow, UK<br
class="">
><br class="">
> The University of
Strathclyde is a charitable
body, registered in<br class="">
> Scotland, number SC015263.<br
class="">
<br class="">
<br class="">
_______________________________________________<br class="">
foundry-nsp mailing list<br
class="">
<a moz-do-not-send="true"
href="mailto:foundry-nsp@puck.nether.net"
style="color: purple;
text-decoration: underline;"
class="">foundry-nsp@puck.nether.net</a><o:p
class=""></o:p></div>
</div>
</div>
<div style="margin: 0in 0in 0.0001pt;
font-size: 12pt; font-family: 'Times
New Roman', serif;" class=""><a
moz-do-not-send="true"
href="http://puck.nether.net/mailman/listinfo/foundry-nsp"
target="_blank" style="color:
purple; text-decoration:
underline;" class="">http://puck.nether.net/mailman/listinfo/foundry-nsp</a></div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
foundry-nsp mailing list
<a class="moz-txt-link-abbreviated" href="mailto:foundry-nsp@puck.nether.net">foundry-nsp@puck.nether.net</a>
<a class="moz-txt-link-freetext" href="http://puck.nether.net/mailman/listinfo/foundry-nsp">http://puck.nether.net/mailman/listinfo/foundry-nsp</a></pre>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="72">--
Jeroen Wunnink
IP NOC Manager - Hibernia Networks
Main numbers (Ext: 1011): USA +1.908.516.4200 | UK +44.1704.322.300
Netherlands +31.208.200.622 | 24/7 IP NOC Phone: +31.20.82.00.623
<a class="moz-txt-link-abbreviated" href="mailto:jeroen.wunnink@hibernianetworks.com">jeroen.wunnink@hibernianetworks.com</a>
<a class="moz-txt-link-abbreviated" href="http://www.hibernianetworks.com">www.hibernianetworks.com</a>
</pre>
</body>
</html>