[c-nsp] Cisco ACE Context
Justin Shore
justin at justinshore.com
Fri Aug 22 15:16:49 EDT 2008
Which time? :-) I didn't work the FWSM problem but from what I was
told there was some problem with the internal vs overall VLAN structure
having to do with VLANs being added or removed while already (or still)
being assigned to the FWSM, etc. I can try to locate the case notes if
needed. The problem was IOS-related, not FWSM-specific. That was the
FWSM fix.
The IPSec SPA problem was horrible. I beat my head on the wall for
months on that one, went through numerous TAC engineers too. I finally
got that one TAC VPN Specialist and he knocked it out of the park. I'd
turn up a new VLAN and within seconds the CPU would hit 100%. Come to
find out that the VLAN I enabled was permitted as part of a range of
VLANs that the SMEs configured on both sides of the IPSec SPA. Frames
were getting looped through the SPA and in a never-ending circle. The
VLAN I'm referring to had HSRP configured on it. The Sup was bombarded
with HSRP packets which I assume are processed in SW because the CPU was
thoroughly hammered. I did a packet capture on the IPSec SPA ints.
Within a second of running tcpdump I'd ctrl-c it. tcpdump had caught
roughly 250k of packets and missed 500k. That's a lot of packets for a
processor to chew on! I had some VLANs without a FHRP configured.
Mirroring an IPSec SPA GigE int showed all sorts of traffic that
shouldn't have been on the IPSec SPA at all. And in the end I came to
find out that the VLAN ranges on those ports were causing the problem.
I got conflicting answers from TAC on what those 2 ints did. Some
engineers said that the first int was L2 and the second was L3 (made no
sense but I couldn't argue). Some said that all VLANs should be
permitted on both ints and that the SPA would sort it out (that would
have been fun!). Others had it partially right, that one was the
encrypted outside of the VLAN and the other was the unencrypted inside.
However they were on to state that the VLAN list for each int should
match what you have configured with your crypto engine commands. Ie,
all 'crypto engine slot x/y outside' VLANs were to be allowed on the
first 1Q trunk and all the inside VLANs were to be on the second. Some
engineers added to that saying that the inside VLANs should only be on
the 2nd int but that the outside VLANs should be on both. It was a real
cluster and resulted in some of the longest downtime this phone company
and our class-5 switches had ever experienced. That VPN Specialist
bailed us out though. I'm always coming up with really weird, baffling
problems like that.
One last thing. Our 7600s have single Sups. It was a design choice
that I plan on someday correcting. However in the 3 TAC cases where
we've been told to reboot the chassis to correct the problem I always
make a point to ask if we had 2 Sups would failing over the Sups fix the
problem. In none of the 3 cases was the answer ever yes. It took
reboots to fix all 3 major problems (the 3rd had to do with VRFs that
would not disappear after deleting them).
Good luck
Justin
Teller, Robert wrote:
> Hmmm that is really weird rebooting the chassis fixed it. Any idea what
> could have happened?
>
> Is it just me or do the ace modules take FOREVER to boot up?
>
> I also noticed that if I configure the ANM software to administer the
> ace modules when I type show run on the active context it just hangs and
> nothing happens. Anyone else experience this?
>
> -----Original Message-----
> From: Justin Shore [mailto:justin at justinshore.com]
> Sent: Friday, August 22, 2008 11:23 AM
> To: Teller, Robert
> Cc: Tony Varriale; cisco-nsp at puck.nether.net
> Subject: Re: [c-nsp] Cisco ACE Context
>
> I haven't worked with an ACE yet but I have two possibly related stories
>
> to relay.
>
> Our FWSM internal 1Q trunks (firewall-group) got hosed up shortly after
> their deployment in our 7600s (SR code). We'd add a VLAN and it would
> show up in the firewall-group config line and it would appear in the
> FWSM sys context but it would not come up/up in the context. No data
> could be passed by the FWSM on those VLANs. TAC determined that a
> reboot of the FWSM was necessary. We rebooted the FWSM to no avail.
> When that failed TAC instructed us to power cycle the chassis. Doing
> that resolved the VLAN issue. IIRC we were on a SRAn release at the
> time. I later upgraded to SRB. Prior to the mentioning of the 10G
> interface this fit you problem more but I didn't have time to write it
> up at the time.
>
> The second story has to do with the special 10G internal interfaces. We
>
> had a couple SMEs out to install and configure a pair of IPSec SPAs in
> the SSC-400 carriers in our 7600s. The SMEs manually configured the 2
> internal GigE ints on the SPAs with the VLANs that they thought so be on
>
> them. The virtual ints were 1Q trunks. A few months later after
> battling extremely weird problems (traffic from VLAN x appearing on VLAN
>
> y with a significant delay in the middle, dupe frames, packet loss,
> 7600s crashing, etc) I found a TAC engineer who could explain how the
> IPSec SPA ints were supposed to be configured. As it turns out you are
> not supposed to touch the virtual ints when running in VRF Mode, period.
>
> Under no circumstances do you touch the ints when in VRF Mode. The
> inside and outside VLANs are configured automatically as you configure
> VRF in crypto statements. Turns out that the SMEs had configured
> numerous VLANs on both virtual ints and in many cases the VLANs
> overlapped. Ie, you had the same VLANs on both sides of the SPA, both
> the encrypted side and the unencrypted side. The auto config stopped as
>
> soon as they modified the interface config manually. My TAC engineer (a
>
> VPN specialist) couldn't believe it actually worked, even a little. He
> helped me fix the problem though. I had to pull the SPAs, reboot both
> 7600s, reinsert the SPAs, and reconfigure crypto from the ground up
> without touching the 1 GigE internal ints. I mention this story in case
>
> these internal 10G ints aren't supposed to be manually configured but
> are instead supposed to be configured automatically based on the svclc
> group commands. None of this may be related though. Good luck.
>
> FYI
> Justin
>
> Teller, Robert wrote:
>> So it looks like the problem is that the interface associated to the
> ace
>> is configurable. Does anyone know how to remove it without rebuilding
>> the chassis?
>>
>> -----Original Message-----
>> From: cisco-nsp-bounces at puck.nether.net
>> [mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of Teller, Robert
>> Sent: Friday, August 22, 2008 9:08 AM
>> To: Tony Varriale; cisco-nsp at puck.nether.net
>> Subject: Re: [c-nsp] Cisco ACE Context
>>
>> So on Chassis-B interface tengig 7/1 is configured differently then
>> chassis-A. And I can't even get into chassis-a tengig 7/1 to make any
>> changes to it.
>>
>> interface TenGigabitEthernet7/1
>> switchport
>> switchport trunk encapsulation dot1q
>> switchport trunk allowed vlan
>> 100,120,138,150,190,200,210,235,238,555,575
>> switchport trunk allowed vlan add 801-804,999
>> switchport mode trunk
>> switchport nonegotiate
>> mls qos trust cos
>> flowcontrol receive on
>> no cdp enable
>> end
>>
>> -----Original Message-----
>> From: cisco-nsp-bounces at puck.nether.net
>> [mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of Tony Varriale
>> Sent: Thursday, August 21, 2008 5:22 PM
>> To: cisco-nsp at puck.nether.net
>> Subject: Re: [c-nsp] Cisco ACE Context
>>
>> I'm partially confused as you are missing a number of vlans not just
>> 138.
>>
>> Can you remove it and reapply?
>>
>> The only other thing I can think of is sh int trunk and see if the
> vlan
>> is
>> getting pruned back.
>>
>> tv
>> ----- Original Message -----
>> From: "Teller, Robert" <RTeller at deltadentalwa.com>
>> To: "Christian Koch" <christian at broknrobot.com>
>> Cc: "Tony Varriale" <tvarriale at comcast.net>;
> <cisco-nsp at puck.nether.net>
>> Sent: Thursday, August 21, 2008 6:53 PM
>> Subject: RE: [c-nsp] Cisco ACE Context
>>
>>
>> Sea-6509-B#sh svclc vlan-group
>> Display vlan-groups created by both ACE module and FWSM commands
>>
>> Group Created by vlans
>> ----- ---------- -----
>> 9706 FWSM
>> 100,120,138,150,190,200,210,235,238,555,575,801-804,999
>>
>> -----Original Message-----
>> From: Christian Koch [mailto:christian at broknrobot.com]
>> Sent: Thursday, August 21, 2008 4:53 PM
>> To: Teller, Robert
>> Cc: Tony Varriale; cisco-nsp at puck.nether.net
>> Subject: Re: [c-nsp] Cisco ACE Context
>>
>> what do you see when you do a 'sh svclc vlan-group' on the 6500 that
>> ace-b is installed in?
>>
>>
>> On Thu, Aug 21, 2008 at 7:32 PM, Teller, Robert
>> <RTeller at deltadentalwa.com> wrote:
>>> That is correct. But if I do show vlan on the ace module it doesn't
>> show
>>> up even though it is associated to vlan group 9706
>>>
>>> Sea-ACE-A/Admin# show vlans
>>> Vlans configured on SUP for this module
>>> vlan100 vlan120 vlan138 vlan150 vlan190 vlan200 vlan210
>> vlan235
>>> vlan238 vlan555 vlan801-803 vlan999
>>>
>>> Sea-ACE-B/Admin# show vlans
>>> Vlans configured on SUP for this module
>>> vlan100 vlan200 vlan210 vlan235 vlan238 vlan555 vlan801-803
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: Tony Varriale [mailto:tvarriale at comcast.net]
>>> Sent: Thursday, August 21, 2008 4:16 PM
>>> To: Teller, Robert; cisco-nsp at puck.nether.net
>>> Subject: Re: [c-nsp] Cisco ACE Context
>>>
>>> Would you do a sh vlan b on sup-b?
>>>
>>> Is 138 there?
>>>
>>> tv
>>> ----- Original Message -----
>>> From: "Teller, Robert" <RTeller at deltadentalwa.com>
>>> To: <cisco-nsp at puck.nether.net>
>>> Sent: Thursday, August 21, 2008 5:47 PM
>>> Subject: [c-nsp] Cisco ACE Context
>>>
>>>
>>>> I have two cisco 6509 chassis with ace and fwsm modules. I have
>>>> configured the ace blades to use an internal and external conext. On
>>>> ACE-A I am able to bring up both contexts and everything talks just
>>> fine
>>>> but on ACE-B I can't bring up vlan 138. Is there something I'm
>>> missing?
>>>>
>>>>
> ------------------------------------------------------------------------
>>>> -----------------------------------------
>>>>
>>>> svclc autostate
>>>>
>>>> svclc multiple-vlan-interfaces
>>>>
>>>> svclc module 7 vlan-group 9706,
>>>>
>>>> firewall autostate
>>>>
>>>> firewall multiple-vlan-interfaces
>>>>
>>>> firewall module 3 vlan-group 9706,
>>>>
>>>> firewall vlan-group 9706
>>>> 100,120,138,150,190,200,210,235,238,555,575,801-804
>>>>
>>>> firewall vlan-group 9706 999
>>>>
>>>>
> ------------------------------------------------------------------------
>>>> -----------------------------------------
>>>>
>>>>
>>>>
>>>> ADMIN Context
>>>>
>>>>
> ------------------------------------------------------------------------
>>>> -----------------------------------------
>>>>
>>>> ft interface vlan 801
>>>>
>>>> ip address XXX.XXX.XXX.145 255.255.255.252
>>>>
>>>> peer ip address XXX.XXX.XXX.146 255.255.255.252
>>>>
>>>> no shutdown
>>>>
>>>>
>>>>
>>>> ft peer 1
>>>>
>>>> heartbeat interval 300
>>>>
>>>> heartbeat count 20
>>>>
>>>> ft-interface vlan 801
>>>>
>>>> ft group 1
>>>>
>>>> peer 1
>>>>
>>>> priority 200
>>>>
>>>> associate-context Admin
>>>>
>>>> inservice
>>>>
>>>>
>>>>
>>>> context WDS-External
>>>>
>>>> allocate-interface vlan 138
>>>>
>>>> context WDS-Internal
>>>>
>>>> allocate-interface vlan 238
>>>>
>>>>
>>>>
>>>> ft group 2
>>>>
>>>> peer 1
>>>>
>>>> priority 200
>>>>
>>>> associate-context WDS-Internal
>>>>
>>>> inservice
>>>>
>>>> ft group 3
>>>>
>>>> peer 1
>>>>
>>>> priority 200
>>>>
>>>> associate-context WDS-External
>>>>
>>>> inservice
>>>>
>>>>
> ------------------------------------------------------------------------
>>>> -----------------------------------------
>>>>
>>>>
>>>>
>>>> context WDS-External
>>>>
>>>>
> ------------------------------------------------------------------------
>>>> -----------------------------------------
>>>>
>>>> interface vlan 138
>>>>
>>>> ip address XXX.XXX.XXX.150 255.255.255.192
>>>>
>>>> alias XXX.XXX.XXX.188 255.255.255.192
>>>>
>>>> peer ip address XXX.XXX.XXX.189 255.255.255.192
>>>>
>>>> access-group input any
>>>>
>>>> service-policy input REMOTE_MGMT_ALLOW_POLICY
>>>>
>>>> no shutdown
>>>>
>>>>
>>>>
>>>> vlan138 is down, VLAN not assigned from the supervisor
>>>>
>>>> Hardware type is VLAN
>>>>
>>>> MAC address is 00:1f:6c:89:0c:33
>>>>
>>>> Mode : routed
>>>>
>>>> IP address is XXX.XXX.XXX.150 netmask is 255.255.255.192
>>>>
>>>> FT status is standby
>>>>
>>>> Description:not set
>>>>
>>>> MTU: 1500 bytes
>>>>
>>>> Last cleared: never
>>>>
>>>> Alias IP address is XXX.XXX.XXX.188 netmask is 255.255.255.192
>>>>
>>>> Peer IP address is XXX.XXX.XXX.189 Peer IP netmask is
>> 255.255.255.192
>>>> Not assigned from the Supervisor, down on Supervisor
>>>>
>>>> Service-policy download failures : 3
>>>>
>>>> 0 unicast packets input, 0 bytes
>>>>
>>>> 0 multicast, 0 broadcast
>>>>
>>>> 0 input errors, 0 unknown, 0 ignored, 0 unicast RPF drops
>>>>
>>>> 0 unicast packets output, 0 bytes
>>>>
>>>> 0 multicast, 0 broadcast
>>>>
>>>> 0 output errors, 0 ignored
>>>>
>>>>
> ------------------------------------------------------------------------
>>>> -----------------------------------------
>>>>
>>>>
>>>>
>>>> Robert Teller
>>>> Washington Dental Service
>>>> Network Administrator
>>>> (206) 528-2371
>>>> RTeller at DeltaDentalWa.com <mailto:RTeller at DeltaDentalWa.com>
>>>>
>>>>
>>>>
>>>>
>>>> #########################################################
>>>> The information contained in this e-mail and subsequent attachments
>>> may be
>>>> privileged,
>>>> confidential and protected from disclosure. This transmission is
>>> intended
>>>> for the sole
>>>> use of the individual and entity to whom it is addressed. If you
> are
>>> not
>>>> the intended
>>>> recipient, any dissemination, distribution or copying is strictly
>>>> prohibited. If you
>>>> think that you have received this message in error, please e-mail
> the
>>>> sender at the above
>>>> e-mail address.
>>>> #########################################################
>>>> _______________________________________________
>>>> cisco-nsp mailing list cisco-nsp at puck.nether.net
>>>> https://puck.nether.net/mailman/listinfo/cisco-nsp
>>>> archive at http://puck.nether.net/pipermail/cisco-nsp/
>>>
>>> _______________________________________________
>>> cisco-nsp mailing list cisco-nsp at puck.nether.net
>>> https://puck.nether.net/mailman/listinfo/cisco-nsp
>>> archive at http://puck.nether.net/pipermail/cisco-nsp/
>>>
>>
>> _______________________________________________
>> cisco-nsp mailing list cisco-nsp at puck.nether.net
>> https://puck.nether.net/mailman/listinfo/cisco-nsp
>> archive at http://puck.nether.net/pipermail/cisco-nsp/
>>
>>
>> _______________________________________________
>> cisco-nsp mailing list cisco-nsp at puck.nether.net
>> https://puck.nether.net/mailman/listinfo/cisco-nsp
>> archive at http://puck.nether.net/pipermail/cisco-nsp/
>>
>>
>> _______________________________________________
>> cisco-nsp mailing list cisco-nsp at puck.nether.net
>> https://puck.nether.net/mailman/listinfo/cisco-nsp
>> archive at http://puck.nether.net/pipermail/cisco-nsp/
>
>
More information about the cisco-nsp
mailing list