[j-nsp] What is this ethernet switching trace telling us?
John Neiberger
jneiberger at gmail.com
Mon Jun 10 11:44:36 EDT 2013
This all has to be a problem on the SBC. I've only been peripherally
involved up until now and I just found out they have another SBC in the
same VLAN on the same pair of switches and it is not having a problem and
is not experiencing this mad MAC address behavior at all. It's perfectly
stable. That pretty much proves to me beyond any reasonable doubt that this
is a problem with the SBC and the switch itself is not contributing to the
problem in any way.
On Mon, Jun 10, 2013 at 8:57 AM, John Neiberger <jneiberger at gmail.com>wrote:
> Here's an example of what I'm talking about. The MAC learning log has been
> stable since they reloaded the SBC on Saturday:
>
> show ethernet-switching mac-learning-log | match 3c:91
> Sat Jul 23 12:05:22 2011 vlan_name sbc-core mac 00:08:25:fa:3c:91 was
> learned on xe-0/1/1.0
> Sat Apr 13 04:24:15 2013 vlan_name sbc-core mac 00:08:25:fa:3c:91 was
> deleted on xe-0/1/1.0
> Sat Apr 13 04:25:10 2013 vlan_name sbc-core mac 00:08:25:fa:3c:91 was
> learned on ge-0/0/5.0
> Sat Apr 13 04:25:11 2013 vlan_name sbc-core mac 00:08:25:fa:3c:91 was
> deleted on ge-0/0/5.0
> Sat Apr 13 04:25:18 2013 vlan_name sbc-core mac 00:08:25:fa:3c:91 was
> learned on xe-0/1/1.0
> Sat Jun 8 04:27:12 2013 vlan_name sbc-core mac 00:08:25:fa:3c:91 was
> deleted on ge-0/0/5.0
> Sat Jun 8 04:28:08 2013 vlan_name sbc-core mac 00:08:25:fa:3c:91 was
> learned on ge-0/0/5.0
> Sat Jun 8 04:28:09 2013 vlan_name sbc-core mac 00:08:25:fa:3c:91 was
> deleted on ge-0/0/5.0
> Sat Jun 8 04:28:12 2013 vlan_name sbc-core mac 00:08:25:fa:3c:91 was
> learned on ge-0/0/5.0
>
>
> However, we have a firewall filter on ge-0/0/6 and ge-0/0/8 to watch for
> that MAC address and its counters are incrementing. They have incremented
> several times this morning already:
>
> Filter: ACME-ge-0/0/6.0-i
> Counters:
> Name Bytes
> Packets
> from-ACME-3c91-ge-0/0/6.0-i 1090
> 5
> from-others-ge-0/0/6.0-i 215535790813
> 981754726
>
> Filter: ACME-ge-0/0/8.0-i
> Counters:
> Name Bytes
> Packets
> from-ACME-3c91-ge-0/0/8.0-i 15042
> 69
> from-others-ge-0/0/8.0-i 205829895148
> 944189465
>
> We also have a trace log running that shows this MAC jumping around. Here
> is the log output from just the last few minutes:
>
> Jun 10 14:30:31.546683 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
> ifname ge-0/0/7.0, pnac_status 0, 0
> Jun 10 14:30:31.546978 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
> ifname ge-0/0/5.0, pnac_status 0, 0
> Jun 10 14:31:31.826971 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
> ifname ge-0/0/5.0, pnac_status 0, 0
> Jun 10 14:32:32.535585 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
> ifname ge-0/0/8.0, pnac_status 0, 0
> Jun 10 14:32:32.536080 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
> ifname ge-0/0/5.0, pnac_status 0, 0
> Jun 10 14:36:43.660324 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
> ifname ge-0/0/8.0, pnac_status 0, 0
> Jun 10 14:36:43.660961 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
> ifname ge-0/0/5.0, pnac_status 0, 0
> Jun 10 14:36:55.239016 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
> ifname ge-0/0/8.0, pnac_status 0, 0
> Jun 10 14:36:55.239425 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
> ifname ge-0/0/5.0, pnac_status 0, 0
> Jun 10 14:37:29.244518 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
> ifname ge-0/0/5.0, pnac_status 0, 0
> Jun 10 14:38:32.854226 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
> ifname ge-0/0/8.0, pnac_status 0, 0
> Jun 10 14:38:32.854532 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
> ifname ge-0/0/5.0, pnac_status 0, 0
> Jun 10 14:39:36.204933 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
> ifname ge-0/0/8.0, pnac_status 0, 0
> Jun 10 14:39:36.205232 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
> ifname ge-0/0/5.0, pnac_status 0, 0
> Jun 10 14:40:11.707066 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
> ifname ge-0/0/5.0, pnac_status 0, 0
> Jun 10 14:40:37.537264 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
> ifname ge-0/0/8.0, pnac_status 0, 0
> Jun 10 14:40:37.537562 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
> ifname ge-0/0/5.0, pnac_status 0, 0
> Jun 10 14:43:18.421993 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
> ifname ge-0/0/8.0, pnac_status 0, 0
> Jun 10 14:43:18.422492 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
> ifname ge-0/0/5.0, pnac_status 0, 0
> Jun 10 14:43:59.154585 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
> ifname ge-0/0/5.0, pnac_status 0, 0
> Jun 10 14:44:00.000894 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
> ifname ge-0/0/8.0, pnac_status 0, 0
> Jun 10 14:44:00.001405 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
> ifname ge-0/0/5.0, pnac_status 0, 0
> Jun 10 14:44:47.068248 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
> ifname ge-0/0/8.0, pnac_status 0, 0
> Jun 10 14:44:47.068539 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
> ifname ge-0/0/5.0, pnac_status 0, 0
> Jun 10 14:45:36.634048 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
> ifname ge-0/0/5.0, pnac_status 0, 0
> Jun 10 14:47:27.367950 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
> ifname ge-0/0/8.0, pnac_status 0, 0
> Jun 10 14:47:27.368359 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
> ifname ge-0/0/5.0, pnac_status 0, 0
> Jun 10 14:48:15.151240 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
> ifname ge-0/0/8.0, pnac_status 0, 0
> Jun 10 14:48:15.151530 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
> ifname ge-0/0/5.0, pnac_status 0, 0
> Jun 10 14:48:51.587305 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
> ifname ge-0/0/5.0, pnac_status 0, 0
> Jun 10 14:50:00.933896 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
> ifname ge-0/0/8.0, pnac_status 0, 0
> Jun 10 14:50:00.934196 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
> ifname ge-0/0/5.0, pnac_status 0, 0
> Jun 10 14:54:43.553048 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
> ifname ge-0/0/8.0, pnac_status 0, 0
> Jun 10 14:54:43.553342 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
> ifname ge-0/0/5.0, pnac_status 0, 0
> Jun 10 14:54:50.862562 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
> ifname ge-0/0/8.0, pnac_status 0, 0
> Jun 10 14:54:50.862887 Attempt to add vlan sbc-core mac 00:08:25:fa:3c:91,
> ifname ge-0/0/5.0, pnac_status 0, 0
>
> If the MAC address really is jumping around, why isn't the MAC learning
> log being updated? I get worried when the tools don't agree about what is
> occurring.
>
> Thanks,
> John
>
>
>
> On Mon, Jun 10, 2013 at 8:36 AM, John Neiberger <jneiberger at gmail.com>wrote:
>
>> I was just checking again and I'm really having difficulty reconciling
>> the conflicting information reported by the switch. The output of "show
>> ethernet mac-learning-log" is stable and the MAC addresses show as learned
>> on the expected interfaces. However, we created firewall filters to count
>> frames coming in with the wrong MAC address and those counters are
>> incrementing. We also have the ethernet trace lot that shows the MAC
>> addresses jumping around.
>>
>> So, which is it? Do I trust the trace logs and firewall filter or do I
>> trust the MAC learning log? I really with they all agreed.
>>
>> Thanks,
>> John
>>
>>
>> On Mon, Jun 10, 2013 at 8:23 AM, John Neiberger <jneiberger at gmail.com>wrote:
>>
>>> I think they do have them configured differently, but they swear they
>>> don't and I don't have the access or the knowledge on that platform to
>>> disagree with them. The network side is straightforward switching. You are
>>> correct that we have multiple groups managing these devices. This group is
>>> the only one having these problems, but even then it's only with a couple
>>> of locations. The rest of theirs are fine. We've been having major
>>> conference calls to discuss this problem and those have included several
>>> people from Acme Packet. I can only assume that they've verified the high
>>> availability config.
>>>
>>> I was hoping that they were running linux under the hood since this is
>>> apparently the default behavior of linux. However, I just heard that
>>> they're running Vxworks. I'll have to do some digging to see if it also
>>> behaves similarly by default.
>>>
>>> Thanks!
>>> John
>>>
>>>
>>> On Mon, Jun 10, 2013 at 2:37 AM, Phil Mayers <p.mayers at imperial.ac.uk>wrote:
>>>
>>>> On 06/09/2013 04:59 PM, John Neiberger wrote:
>>>>
>>>> We have several of these throughout our network and we're only seeing
>>>>> this
>>>>> problem in a couple of cases. The rest work just fine. Most of the
>>>>> SBCs are
>>>>>
>>>>
>>>> Obvious question: are you sure those two aren't configured different
>>>> (wrongly) compared to the others, or running different software?
>>>>
>>>> Reading between the lines it sounds like a different group runs these
>>>> devices, and in my experience, kit like this with odd network interface
>>>> capabilities can be misconfigured by staff that don't fully understand
>>>> networking - and it can be exacerbated by vendors using odd terminology
>>>> (e.g. "PHY" used in a non-standard way) and confusing people.
>>>>
>>>> Are you sure they haven't just setup these two devices in active-active
>>>> mode? Or more likely, something that *sounds* like an active-passive mode
>>>> in the UI, to a non-expert, but is really some kind of weird active-active
>>>> spatchcock.
>>>>
>>>> Our storage team did this with a couple of NetApps and the symptoms
>>>> were pretty much identical. It only really brought things crashing down
>>>> when they started sending 1Gbit/sec of performance testing traffic from
>>>> them...
>>>>
>>>> ______________________________**_________________
>>>> juniper-nsp mailing list juniper-nsp at puck.nether.net
>>>> https://puck.nether.net/**mailman/listinfo/juniper-nsp<https://puck.nether.net/mailman/listinfo/juniper-nsp>
>>>>
>>>
>>>
>>
>
More information about the juniper-nsp
mailing list