[c-nsp] Nexus Architecture question
Drew Weaver
drew.weaver at thenap.com
Tue Jul 6 15:30:26 EDT 2021
Hello,
Just to circle back since there appears to be some controversy regarding whether or not the original issue was being caused by a problem where NXOS doesn't compensate for a flaw in the hardware.
I have confirmed that the same config on the same version works fine on the 93108TC-EX so it does appear that this is a hardware flaw that the OS is not compensating for on the Broadcom based platforms.
Not sure how that would've gotten through QA but I am glad it was finally fixed in newer hardware.
Thanks,
-Drew
From: Tim Stevenson (tstevens) <tstevens at cisco.com>
Sent: Friday, June 11, 2021 11:55 AM
To: Drew Weaver <drew.weaver at thenap.com>; 'Jeffrey G. Fitzwater' <jfitz at princeton.edu>; 'cisco-nsp at puck.nether.net' <cisco-nsp at puck.nether.net>
Subject: RE: Nexus Architecture question
I don't know that we can/should draw that conclusion - as you mentioned, you opened a TAC case but from my understanding it was never driven to a terminal resolution - either "known limitation, live with it" or "bug, we will/won't fix it" or "you're doing it wrong". I tested this on 2nd gen n9k and it works fine, I don't have 1st gen gear in my lab so can't empirically confirm that the broadcom-based line cards do or don't behave properly and why, or whether there's an alternative config that does work. One wrinkle here is that these linecards are now well past end of sale, and in fact are past End of Vulnerability/Security Support as well, so maybe it's a moot point anyway. For 10GBASE-T in the 2nd gen I'd suggest the X9788TC-FX linecard and FM-E2 fabric module, these should behave properly with the config I posted earlier.
Tim
From: Drew Weaver <drew.weaver at thenap.com<mailto:drew.weaver at thenap.com>>
Sent: Friday, June 11, 2021 8:20 AM
To: 'Jeffrey G. Fitzwater' <jfitz at princeton.edu<mailto:jfitz at princeton.edu>>; Tim Stevenson (tstevens) <tstevens at cisco.com<mailto:tstevens at cisco.com>>; 'cisco-nsp at puck.nether.net' <cisco-nsp at puck.nether.net<mailto:cisco-nsp at puck.nether.net>>
Subject: RE: Nexus Architecture question
Yes, the problem is that the OS doesn't compensate for the flaws in the Broadcom based line cards.
Thanks,
-Drew
From: Jeffrey G. Fitzwater <jfitz at princeton.edu<mailto:jfitz at princeton.edu>>
Sent: Friday, June 11, 2021 11:19 AM
To: Drew Weaver <drew.weaver at thenap.com<mailto:drew.weaver at thenap.com>>; 'Tim Stevenson (tstevens)' <tstevens at cisco.com<mailto:tstevens at cisco.com>>; 'cisco-nsp at puck.nether.net' <cisco-nsp at puck.nether.net<mailto:cisco-nsp at puck.nether.net>>
Subject: Re: Nexus Architecture question
I am not sure this question was asked in this thread, but are you using a custom COPP and not the default?
If you have a custom COPP you must apply the new policy with that name prefix i.e. router-core-copp-acl-hsrp
Vs copp-acl-hsrp.
We do this on our 7 and 9ks so that any new code does not override the custom COPP policies.
Jeff Fitzwater
Princeton University
From: cisco-nsp <cisco-nsp-bounces at puck.nether.net<mailto:cisco-nsp-bounces at puck.nether.net>> on behalf of Drew Weaver <drew.weaver at thenap.com<mailto:drew.weaver at thenap.com>>
Date: Friday, June 11, 2021 at 08:43
To: Drew Weaver <drew.weaver at thenap.com<mailto:drew.weaver at thenap.com>>, 'Tim Stevenson (tstevens)' <tstevens at cisco.com<mailto:tstevens at cisco.com>>, 'cisco-nsp at puck.nether.net' <cisco-nsp at puck.nether.net<mailto:cisco-nsp at puck.nether.net>>
Subject: Re: [c-nsp] Nexus Architecture question
Just for the list this ended up being a Nexus/NXOS limitation.
-----Original Message-----
From: cisco-nsp <cisco-nsp-bounces at puck.nether.net<mailto:cisco-nsp-bounces at puck.nether.net>> On Behalf Of Drew Weaver
Sent: Tuesday, June 8, 2021 11:07 AM
To: 'Tim Stevenson (tstevens)' <tstevens at cisco.com<mailto:tstevens at cisco.com>>; 'cisco-nsp at puck.nether.net' <cisco-nsp at puck.nether.net<mailto:cisco-nsp at puck.nether.net>>
Subject: Re: [c-nsp] Nexus Architecture question
Sure,
N9K-SUP-B
version 9.3(6)
Also tried: 7.0(3)I7(8)
Thanks,
-Drew
-----Original Message-----
From: Tim Stevenson (tstevens) <tstevens at cisco.com<mailto:tstevens at cisco.com>>
Sent: Friday, June 4, 2021 4:40 PM
To: Drew Weaver <drew.weaver at thenap.com<mailto:drew.weaver at thenap.com>>; 'cisco-nsp at puck.nether.net' <cisco-nsp at puck.nether.net<mailto:cisco-nsp at puck.nether.net>>
Subject: RE: Nexus Architecture question
Hi Drew,
Can you specify hardware platform and software version here? I am not seeing what you're seeing, the config I sent blocks a BGP port scan in nmap, and prevents BGP peering to anything other than the specified IP. I am testing on Nexus 9500 with 10.1 NXOS; I suspect you are using some other platform eg n5k or n7k? I may be able to try it out here depending on what you're using.
Thanks,
Tim
-----Original Message-----
From: Drew Weaver <drew.weaver at thenap.com<mailto:drew.weaver at thenap.com>>
Sent: Thursday, June 3, 2021 12:37 PM
To: Tim Stevenson (tstevens) <tstevens at cisco.com<mailto:tstevens at cisco.com>>; 'cisco-nsp at puck.nether.net' <cisco-nsp at puck.nether.net<mailto:cisco-nsp at puck.nether.net>>
Subject: RE: Nexus Architecture question
Hi Tim,
I replied off-list with additional details but I wanted to reply on list to answer your specific questions:
I copied the strict policy and then edited it with these classes at the top:
policy-map type control-plane Test6
class custom-copp-system-p-bgp-allow
police cir 19000 pps bc 128 packets conform transmit violate drop
class custom-copp-system-p-bgp-deny
police cir 0 pps bc 1 packets conform transmit violate drop
These are the configured class-maps:
Class-map type control-plane match-any custom-copp-system-p-bgp-allow
match access-group name custom-copp-system-p-acl-bgp-allow
Class-map type control-plane match-any custom-copp-system-p-bgp-deny
match access-group name custom-copp-system-p-acl-bgp-deny
These are the configured ACLs:
IP access list custom-copp-system-p-acl-bgp-allow
3 permit tcp 192.168.1.2/32 gt 1023 any eq bgp
4 permit tcp 192.168.1.2/32 eq bgp any gt 1023
IP access list custom-copp-system-p-acl-bgp-deny
1 permit tcp any any eq bgp
10 permit tcp any gt 1023 any eq bgp
20 permit tcp any eq bgp any gt 1023
control-plane
service-policy input Test6
The primary difference that I notice between the two configurations is that your allow is set to cos 7 and on mine it is not specified and yours is policing on bps and mine is policing on pps. (I am pretty certain that throughout the process I have tried it both ways).
I find it a bit hard to follow that on this platform there is no way to write "just block everything that matches this class" you still have to give it a "burst" [which I obviously don't want] I understand that behind the scenes the control plane is supposed to "do the right thing" but it is just difficult to understand that from reading it.
With this configuration applied:
1) Any host that NMAPs the device sees 179 open.
2) BGP sessions establish (if configured on the Nexus) for hosts that are not 192.168.1.2/32
Is there any way to find out what class traffic from a specific SRC IP or to a specific port is getting caught at in a CoPP policy?
I suspect that the TCP/179 traffic is just bypassing these classes altogether and most likely are getting handled further down in the CoPP policy but I haven't been able to prove that.
Thanks so much.
-Drew
-----Original Message-----
From: Tim Stevenson (tstevens) <tstevens at cisco.com<mailto:tstevens at cisco.com>>
Sent: Wednesday, June 2, 2021 5:31 PM
To: Drew Weaver <drew.weaver at thenap.com<mailto:drew.weaver at thenap.com>>; 'cisco-nsp at puck.nether.net' <cisco-nsp at puck.nether.net<mailto:cisco-nsp at puck.nether.net>>
Subject: RE: Nexus Architecture question
Hi Drew,
In answer to your question about BGP, the BGP process runs only on the supervisor engine, it does not run on the linecards or anywhere else. It's a single process, not a per-interface process or anything like that.
Curious how exactly you are configuring CoPP to filter this? With default CoPP, I get an "open/tcpwrapped" (green) response on TCP 179; I was able to get a "filtered" (red) response by adding a CoPP class that matches on BGP and polices to a CIR of 0. I preceeded that class with a class that matches on a specific neighborship - that BGP neighborship is successfully established while nmap still returns as filtered from my host.
ip access-list allow-bgp
10 permit tcp 10.1.1.1/32 gt 1023 10.1.1.2/32 eq bgp
20 permit tcp 10.1.1.2/32 eq bgp 10.1.1.1/32 gt 1023 ip access-list drop-bgp
10 permit tcp any any eq bgp
20 permit tcp any eq bgp any
!
class-map type control-plane match-any allow-bgp
match access-group name allow-bgp
class-map type control-plane match-any drop-bgp
match access-group name drop-bgp
!
policy-map type control-plane test-copp-policy-strict
class allow-bgp
set cos 7
police cir 36000 kbps bc 1280000 bytes conform transmit violate drop
class drop-bgp
police cir 0 bps bc 32000 bytes conform transmit violate drop
Hope that helps,
Tim
-----Original Message-----
From: cisco-nsp <cisco-nsp-bounces at puck.nether.net<mailto:cisco-nsp-bounces at puck.nether.net>> On Behalf Of Drew Weaver
Sent: Wednesday, June 2, 2021 6:41 AM
To: 'cisco-nsp at puck.nether.net' <cisco-nsp at puck.nether.net<mailto:cisco-nsp at puck.nether.net>>
Subject: [c-nsp] Nexus Architecture question
Has anyone seen a document from Cisco that shows where various processes running on various Nexus switches actually run from?
For example on a 9508 the nxapi runs in a Linux VM and in order to secure it you have to drop into the VM and use iptables.
I am trying to figure out where the BGP process lives (for lack of a better word). Does it run on the line cards? In the control plane? Both? Does it vary depending on which model and which line cards?
The reason I am asking is because I've noticed that no matter what I do I cannot seem to "close" the BGP port by using CoPP.
It always shows up as being open when doing a port scan against the system using NMAP. I know that the switch should not establish a connection with random hosts but I really am getting hung up on it being 'scannable'/visible at all.
_______________________________________________
cisco-nsp mailing list cisco-nsp at puck.nether.net<mailto:cisco-nsp at puck.nether.net> https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman_listinfo_cisco-2Dnsp&d=DwIFAg&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=OPufM5oSy-PFpzfoijO_w76wskMALE1o4LtA3tMGmuw&m=YtllAoz3HDjZVAXXoCsZK6ipRakorNNjjRatpXyS2WI&s=4ERFXSbBNsxA_cctrBhoVJkOGZGvRGL-K6poiFabwpE&e=
archive at https://urldefense.proofpoint.com/v2/url?u=http-3A__puck.nether.net_pipermail_cisco-2Dnsp_&d=DwIFAg&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=OPufM5oSy-PFpzfoijO_w76wskMALE1o4LtA3tMGmuw&m=YtllAoz3HDjZVAXXoCsZK6ipRakorNNjjRatpXyS2WI&s=9jw1ySSVK0D_6O6lG4HjPujMRWV8aM028yprf0soAJo&e=
_______________________________________________
cisco-nsp mailing list cisco-nsp at puck.nether.net<mailto:cisco-nsp at puck.nether.net> https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman_listinfo_cisco-2Dnsp&d=DwICAg&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=OPufM5oSy-PFpzfoijO_w76wskMALE1o4LtA3tMGmuw&m=mYLjY0RLcn6OVAmqcv-X_w4FBT1V0XTIJZSucpPJhds&s=IlFHv5LgNwuMOWVcYDTNTlTk0Zlli1Cj_MYVH6Qa43U&e=
archive at https://urldefense.proofpoint.com/v2/url?u=http-3A__puck.nether.net_pipermail_cisco-2Dnsp_&d=DwICAg&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=OPufM5oSy-PFpzfoijO_w76wskMALE1o4LtA3tMGmuw&m=mYLjY0RLcn6OVAmqcv-X_w4FBT1V0XTIJZSucpPJhds&s=mJntVnKCTkwsmnr6uagyFaS9KSTHJb2bB3qbTgSqqR0&e=
_______________________________________________
cisco-nsp mailing list cisco-nsp at puck.nether.net<mailto:cisco-nsp at puck.nether.net>
https://puck.nether.net/mailman/listinfo/cisco-nsp<https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman_listinfo_cisco-2Dnsp&d=DwMFAg&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=OPufM5oSy-PFpzfoijO_w76wskMALE1o4LtA3tMGmuw&m=Br2TQfuvu52-RIQlz4YEnn2fOWpvNX4QUgNs5adFx3Y&s=zSiUlwkCnh0NCTrlSnyOkd3KTiUMqg1FPSG6hJn6QzM&e=>
archive at http://puck.nether.net/pipermail/cisco-nsp/<https://urldefense.proofpoint.com/v2/url?u=http-3A__puck.nether.net_pipermail_cisco-2Dnsp_&d=DwMFAg&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=OPufM5oSy-PFpzfoijO_w76wskMALE1o4LtA3tMGmuw&m=Br2TQfuvu52-RIQlz4YEnn2fOWpvNX4QUgNs5adFx3Y&s=i9poNi30MoUiKcv9veM7OEtYveaw3tPfaoTKrYYKHv8&e=>
More information about the cisco-nsp
mailing list