[j-nsp] What is this ethernet switching trace telling us?
John Neiberger
jneiberger at gmail.com
Mon Jun 10 10:57:05 EDT 2013
Here's an example of what I'm talking about. The MAC learning log has been
stable since they reloaded the SBC on Saturday:
show ethernet-switching mac-learning-log | match 3c:91
Sat Jul 23 12:05:22 2011 vlan_name sbc-core mac 00:08:25:fa:3c:91 was
learned on xe-0/1/1.0
Sat Apr 13 04:24:15 2013 vlan_name sbc-core mac 00:08:25:fa:3c:91 was
deleted on xe-0/1/1.0
Sat Apr 13 04:25:10 2013 vlan_name sbc-core mac 00:08:25:fa:3c:91 was
learned on ge-0/0/5.0
Sat Apr 13 04:25:11 2013 vlan_name sbc-core mac 00:08:25:fa:3c:91 was
deleted on ge-0/0/5.0
Sat Apr 13 04:25:18 2013 vlan_name sbc-core mac 00:08:25:fa:3c:91 was
learned on xe-0/1/1.0
Sat Jun 8 04:27:12 2013 vlan_name sbc-core mac 00:08:25:fa:3c:91 was
deleted on ge-0/0/5.0
Sat Jun 8 04:28:08 2013 vlan_name sbc-core mac 00:08:25:fa:3c:91 was
learned on ge-0/0/5.0
Sat Jun 8 04:28:09 2013 vlan_name sbc-core mac 00:08:25:fa:3c:91 was
deleted on ge-0/0/5.0
Sat Jun 8 04:28:12 2013 vlan_name sbc-core mac 00:08:25:fa:3c:91 was
learned on ge-0/0/5.0
However, we have a firewall filter on ge-0/0/6 and ge-0/0/8 to watch for
that MAC address and its counters are incrementing. They have incremented
several times this morning already:
Filter: ACME-ge-0/0/6.0-i
Counters:
Name Bytes
Packets
from-ACME-3c91-ge-0/0/6.0-i 1090
5
from-others-ge-0/0/6.0-i 215535790813
981754726
Filter: ACME-ge-0/0/8.0-i
Counters:
Name Bytes
Packets
from-ACME-3c91-ge-0/0/8.0-i 15042
69
from-others-ge-0/0/8.0-i 205829895148
944189465
We also have a trace log running that shows this MAC jumping around. Here
is the log output from just the last few minutes:
Jun 10 14:30:31.546683 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
ifname ge-0/0/7.0, pnac_status 0, 0
Jun 10 14:30:31.546978 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
ifname ge-0/0/5.0, pnac_status 0, 0
Jun 10 14:31:31.826971 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
ifname ge-0/0/5.0, pnac_status 0, 0
Jun 10 14:32:32.535585 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
ifname ge-0/0/8.0, pnac_status 0, 0
Jun 10 14:32:32.536080 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
ifname ge-0/0/5.0, pnac_status 0, 0
Jun 10 14:36:43.660324 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
ifname ge-0/0/8.0, pnac_status 0, 0
Jun 10 14:36:43.660961 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
ifname ge-0/0/5.0, pnac_status 0, 0
Jun 10 14:36:55.239016 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
ifname ge-0/0/8.0, pnac_status 0, 0
Jun 10 14:36:55.239425 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
ifname ge-0/0/5.0, pnac_status 0, 0
Jun 10 14:37:29.244518 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
ifname ge-0/0/5.0, pnac_status 0, 0
Jun 10 14:38:32.854226 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
ifname ge-0/0/8.0, pnac_status 0, 0
Jun 10 14:38:32.854532 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
ifname ge-0/0/5.0, pnac_status 0, 0
Jun 10 14:39:36.204933 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
ifname ge-0/0/8.0, pnac_status 0, 0
Jun 10 14:39:36.205232 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
ifname ge-0/0/5.0, pnac_status 0, 0
Jun 10 14:40:11.707066 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
ifname ge-0/0/5.0, pnac_status 0, 0
Jun 10 14:40:37.537264 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
ifname ge-0/0/8.0, pnac_status 0, 0
Jun 10 14:40:37.537562 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
ifname ge-0/0/5.0, pnac_status 0, 0
Jun 10 14:43:18.421993 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
ifname ge-0/0/8.0, pnac_status 0, 0
Jun 10 14:43:18.422492 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
ifname ge-0/0/5.0, pnac_status 0, 0
Jun 10 14:43:59.154585 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
ifname ge-0/0/5.0, pnac_status 0, 0
Jun 10 14:44:00.000894 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
ifname ge-0/0/8.0, pnac_status 0, 0
Jun 10 14:44:00.001405 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
ifname ge-0/0/5.0, pnac_status 0, 0
Jun 10 14:44:47.068248 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
ifname ge-0/0/8.0, pnac_status 0, 0
Jun 10 14:44:47.068539 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
ifname ge-0/0/5.0, pnac_status 0, 0
Jun 10 14:45:36.634048 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
ifname ge-0/0/5.0, pnac_status 0, 0
Jun 10 14:47:27.367950 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
ifname ge-0/0/8.0, pnac_status 0, 0
Jun 10 14:47:27.368359 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
ifname ge-0/0/5.0, pnac_status 0, 0
Jun 10 14:48:15.151240 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
ifname ge-0/0/8.0, pnac_status 0, 0
Jun 10 14:48:15.151530 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
ifname ge-0/0/5.0, pnac_status 0, 0
Jun 10 14:48:51.587305 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
ifname ge-0/0/5.0, pnac_status 0, 0
Jun 10 14:50:00.933896 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
ifname ge-0/0/8.0, pnac_status 0, 0
Jun 10 14:50:00.934196 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
ifname ge-0/0/5.0, pnac_status 0, 0
Jun 10 14:54:43.553048 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
ifname ge-0/0/8.0, pnac_status 0, 0
Jun 10 14:54:43.553342 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
ifname ge-0/0/5.0, pnac_status 0, 0
Jun 10 14:54:50.862562 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
ifname ge-0/0/8.0, pnac_status 0, 0
Jun 10 14:54:50.862887 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
ifname ge-0/0/5.0, pnac_status 0, 0
If the MAC address really is jumping around, why isn't the MAC learning log
being updated? I get worried when the tools don't agree about what is
occurring.
Thanks,
John
On Mon, Jun 10, 2013 at 8:36 AM, John Neiberger <jneiberger at gmail.com>wrote:
> I was just checking again and I'm really having difficulty reconciling the
> conflicting information reported by the switch. The output of "show
> ethernet mac-learning-log" is stable and the MAC addresses show as learned
> on the expected interfaces. However, we created firewall filters to count
> frames coming in with the wrong MAC address and those counters are
> incrementing. We also have the ethernet trace lot that shows the MAC
> addresses jumping around.
>
> So, which is it? Do I trust the trace logs and firewall filter or do I
> trust the MAC learning log? I really with they all agreed.
>
> Thanks,
> John
>
>
> On Mon, Jun 10, 2013 at 8:23 AM, John Neiberger <jneiberger at gmail.com>wrote:
>
>> I think they do have them configured differently, but they swear they
>> don't and I don't have the access or the knowledge on that platform to
>> disagree with them. The network side is straightforward switching. You are
>> correct that we have multiple groups managing these devices. This group is
>> the only one having these problems, but even then it's only with a couple
>> of locations. The rest of theirs are fine. We've been having major
>> conference calls to discuss this problem and those have included several
>> people from Acme Packet. I can only assume that they've verified the high
>> availability config.
>>
>> I was hoping that they were running linux under the hood since this is
>> apparently the default behavior of linux. However, I just heard that
>> they're running Vxworks. I'll have to do some digging to see if it also
>> behaves similarly by default.
>>
>> Thanks!
>> John
>>
>>
>> On Mon, Jun 10, 2013 at 2:37 AM, Phil Mayers <p.mayers at imperial.ac.uk>wrote:
>>
>>> On 06/09/2013 04:59 PM, John Neiberger wrote:
>>>
>>> We have several of these throughout our network and we're only seeing
>>>> this
>>>> problem in a couple of cases. The rest work just fine. Most of the SBCs
>>>> are
>>>>
>>>
>>> Obvious question: are you sure those two aren't configured different
>>> (wrongly) compared to the others, or running different software?
>>>
>>> Reading between the lines it sounds like a different group runs these
>>> devices, and in my experience, kit like this with odd network interface
>>> capabilities can be misconfigured by staff that don't fully understand
>>> networking - and it can be exacerbated by vendors using odd terminology
>>> (e.g. "PHY" used in a non-standard way) and confusing people.
>>>
>>> Are you sure they haven't just setup these two devices in active-active
>>> mode? Or more likely, something that *sounds* like an active-passive mode
>>> in the UI, to a non-expert, but is really some kind of weird active-active
>>> spatchcock.
>>>
>>> Our storage team did this with a couple of NetApps and the symptoms were
>>> pretty much identical. It only really brought things crashing down when
>>> they started sending 1Gbit/sec of performance testing traffic from them...
>>>
>>> ______________________________**_________________
>>> juniper-nsp mailing list juniper-nsp at puck.nether.net
>>> https://puck.nether.net/**mailman/listinfo/juniper-nsp<https://puck.nether.net/mailman/listinfo/juniper-nsp>
>>>
>>
>>
>
More information about the juniper-nsp
mailing list