[c-nsp] Unable to pass traffic for new vpn connection

Mon Aug 12 12:18:47 EDT 2013

Scenario:
I have three sites all connected ( full mesh) with IPsec/GRE tunnels and these work fine. I attempted to add a satellite office to one our sites. The sat device is a 3rd party device and is behind a rotuer/fw device. The IPSec tunnel (non-gre) appears to come up but no traffic passes.
When I ping 192.168.3.1 from the sat device (monitored using tcpdump), it cause the tunnel to come up but I don't see the Cisco side replying back. 
The 192.168.180.0/24 network is at the Sat office and the 192.168.3.0/24 network is at the main office.
If I initiate a ping from the Cisco side, it doesn't prompt the tunnel to come up. ???? Any ideas?
Cisco config
crypto isakmp policy 10
encr 3des
hash md5
authentication pre-share
group 2
crypto isakmp key secret address x.x.x.x
crypto isakmp key secret address x.x.x.x
crypto isakmp key secret address 7.7.7.7
crypto isakmp keepalive 10 5 periodic
!
crypto ipsec security-association lifetime seconds 86400
crypto ipsec security-association replay window-size 1024
!
crypto ipsec transform-set vpn_set esp-3des esp-md5-hmac 
crypto ipsec transform-set f5_set esp-3des esp-sha-hmac 
!
crypto map vpnmap 31 ipsec-isakmp 
set peer x.x.x.x
set transform-set vpn_set 
match address 131
crypto map vpnmap 32 ipsec-isakmp 
set peer x.x.x.x
set transform-set vpn_set 
match address 132
crypto map vpnmap 33 ipsec-isakmp 
set peer 7.7.7.7
set transform-set f5_set 
match address 133
!
interface Tunnel31
bandwidth 1200000
ip address 172.16.31.34 255.255.255.252
ip mtu 1400
ip tcp adjust-mss 1360
tunnel source 5.5.5.5
tunnel destination x.x.x.x
!
interface Tunnel32
bandwidth 1200000
ip address 172.16.31.57 255.255.255.252
ip mtu 1400
ip tcp adjust-mss 1360
tunnel source 5.5.5.5
tunnel destination x.x.x.x
!
interface FastEthernet0/1
bandwidth 51200
ip address 50.50.50.1
ip access-group 101 in
ip flow ingress
ip flow egress
ip nat outside
ip inspect ISP2-cbac out
ip virtual-reassembly
duplex auto
speed auto
crypto map vpnmap
!
ip nat inside source route-map nonat interface FastEthernet0/1 overload
partial acl
access-list 101 permit udp host 7.7.7.7 any eq isakmp
access-list 101 permit udp host 7.7.7.7 eq isakmp any
access-list 101 permit esp host 7.7.7.7 any
!
route-map nonat permit 41
match ip address 175
access-list 133 permit ip 192.168.3.0 0.0.0.255 192.168.180.0 0.0.0.255
access-list 175 deny ip 192.168.3.0 0.0.0.255 192.168.60.0 0.0.0.255
access-list 175 deny ip 192.168.3.0 0.0.0.255 192.168.1.0 0.0.0.255
access-list 175 deny ip 192.168.3.0 0.0.0.255 192.168.2.0 0.0.0.255
access-list 175 deny ip 192.168.3.0 0.0.0.255 192.168.180.0 0.0.0.255
access-list 175 permit ip 192.168.3.0 0.0.0.255 any
!
ip route 0.0.0.0 0.0.0.0 50.50.50.x
ip route 10.1.0.0 255.255.0.0 Tunnel32
ip route 172.18.1.0 255.255.255.0 192.168.3.254
ip route 172.18.2.0 255.255.255.0 192.168.3.254
ip route 172.18.3.2 255.255.255.255 Service-Engine0/0
ip route 192.168.1.0 255.255.255.0 Tunnel31
ip route 192.168.2.0 255.255.255.0 Tunnel32
ip route 192.168.10.0 255.255.255.0 192.168.3.254
---------------------------------------------------------------------------------------
sh cry isa sa
IPv4 Crypto ISAKMP SA
dst src state conn-id status
50.50.50.1 7.7.7.7 QM_IDLE 1003 ACTIVE
sh crypto isa sa
protected vrf: (none)
local ident (addr/mask/prot/port): (192.168.3.0/255.255.255.0/0/0)
remote ident (addr/mask/prot/port): (192.168.180.0/255.255.255.0/0/0)
current_peer 7.7.7.7 port 35381
PERMIT, flags={origin_is_acl,}
#pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
#pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 0, #pkts compr. failed: 0
#pkts not decompressed: 0, #pkts decompress failed: 0
#send errors 0, #recv errors 0
local crypto endpt.: 50.50.50.1, remote crypto endpt.: 7.7.7.7
path mtu 1500, ip mtu 1500, ip mtu idb FastEthernet0/1
current outbound spi: 0xFF024E3E(4278341182)
PFS (Y/N): Y, DH group: group2
inbound esp sas:
spi: 0x8E538667(2387838567)
transform: esp-3des esp-sha-hmac ,
in use settings ={Tunnel, }
conn id: 2007, flow_id: FPGA:7, sibling_flags 80000046, crypto map: vpnmap
sa timing: remaining key lifetime (k/sec): (4493323/82118)
IV size: 8 bytes
replay detection support: Y replay window size: 1024
Status: ACTIVE
inbound ah sas:
inbound pcp sas:
outbound esp sas:
spi: 0xFF024E3E(4278341182)
transform: esp-3des esp-sha-hmac ,
in use settings ={Tunnel, }
conn id: 2008, flow_id: FPGA:8, sibling_flags 80000046, crypto map: vpnmap
sa timing: remaining key lifetime (k/sec): (4493323/82118)
IV size: 8 bytes
replay detection support: Y replay window size: 1024
Status: ACTIVE
outbound ah sas:
outbound pcp sas:
DEBUG 
#show debug

Cryptographic Subsystem:
Crypto ISAKMP debugging is on
Crypto ISAKMP Error debugging is on
Crypto IPSEC debugging is on
Crypto IPSEC Error debugging is on
#sh log | inc 7.7.7.7
000202: *Aug 12 02:20:16.006: ISAKMP:(1003): sending packet to 7.7.7.7 my_port 500 peer_port 35381 (R) QM_IDLE
000207: *Aug 12 02:20:16.046: ISAKMP (1003): received packet from 7.7.7.7 dport 500 sport 35381 Global (R) QM_IDLE
000211: *Aug 12 02:20:16.046: ISAKMP:(1003): DPD/R_U_THERE_ACK received from peer 7.7.7.7 sequence 0x1C6F72FD
000287: *Aug 12 02:20:25.962: ISAKMP:(1003): sending packet to 7.7.7.7 my_port 500 peer_port 35381 (R) QM_IDLE
000292: *Aug 12 02:20:25.998: ISAKMP (1003): received packet from 7.7.7.7 dport 500 sport 35381 Global (R) QM_IDLE
000296: *Aug 12 02:20:25.998: ISAKMP:(1003): DPD/R_U_THERE_ACK received from peer 7.7.7.7 sequence 0x1C6F72FE
000389: *Aug 12 02:20:35.542: ISAKMP:(1003): sending packet to 7.7.7.7 my_port 500 peer_port 35381 (R) QM_IDLE
000394: *Aug 12 02:20:35.578: ISAKMP (1003): received packet from 7.7.7.7 dport 500 sport 35381 Global (R) QM_IDLE
000398: *Aug 12 02:20:35.582: ISAKMP:(1003): DPD/R_U_THERE_ACK received from peer 7.7.7.7 sequence 0x1C6F72FF
000402: *Aug 12 02:20:36.582: ISAKMP (1003): received packet from 7.7.7.7 dport 500 sport 35381 Global (R) QM_IDLE
000409: *Aug 12 02:20:36.586: ISAKMP:(1003):DPD/R_U_THERE received from peer 7.7.7.7, sequence 0x5FF
000413: *Aug 12 02:20:36.586: ISAKMP:(1003): sending packet to 7.7.7.7 my_port 500 peer_port 35381 (R) QM_IDLE
3rd party device:
# racoonctl -l show-sa isakmp
Destination Cookies ST S V E Created Phase2
50.50.50.1.500 e1866e9ee2830764:575a7489971701ad 9 I 10 M 2013-08-11 20:04:57 1
[root at ltm1:Active:Disconnected] log # racoonctl -l show-sa isakmp
Destination Cookies ST S V E Created Phase2
50.50.50.1.500 e1866e9ee2830764:575a7489971701ad 9 I 10 M 2013-08-11 20:04:57 1

# racoonctl -l show-sa ipsec
192.168.180.5 50.50.50.1
esp mode=tunnel spi=2387838567(0x8e538667) reqid=62829(0x0000f56d)
E: 3des-cbc 74583bf5 4fe29310 07603be7 d52516d6 7269c35f 51b24a52
A: hmac-sha1 c0d2254c ea2ec11a 6a22bf41 dad35582 00d91a30
seq=0x00000000 replay=64 flags=0x00000000 state=mature
created: Aug 11 20:04:59 2013 current: Aug 11 21:18:57 2013
diff: 4438(s) hard: 5184000(s) soft: 4147200(s)
last: Aug 11 21:18:56 2013 hard: 0(s) soft: 0(s)
current: 421660(bytes) hard: 0(bytes) soft: 0(bytes)
allocated: 3635 hard: 0 soft: 0
sadb_seq=1 pid=8526 refcnt=0
50.50.50.1 192.168.180.5
esp mode=tunnel spi=4278341182(0xff024e3e) reqid=62828(0x0000f56c)
E: 3des-cbc 3bc26d98 0a230000 54c64896 e1a68815 6c696a15 f6779541
A: hmac-sha1 96de21a0 b5f52539 0616acfa b5a09994 03306e92
seq=0x00000000 replay=64 flags=0x00000000 state=mature
created: Aug 11 20:04:59 2013 current: Aug 11 21:18:57 2013
diff: 4438(s) hard: 5184000(s) soft: 4147200(s)
last: hard: 0(s) soft: 0(s)
current: 0(bytes) hard: 0(bytes) soft: 0(bytes)
allocated: 0 hard: 0 soft: 0
sadb_seq=0 pid=8526 refcnt=0