[c-nsp] Cisco ASA5516-x DATAPATH-0-1648 and DATAPATH-0-1648 CPU hog
Lee Starnes
lee.t.starnes at gmail.com
Fri Jun 7 16:52:25 EDT 2024
So, this was tracked down to be an issue with the ASA doing debug logging.
As soon as we changed the logging back down to alert level logging, the
issue resolved.
Only caught the issue when watching the Process CPU-Usage and saw that the
logger process pop up at 52%, then 64% and then 84% CPU usage before
falling off in a 10 second period.
vpn-gw# sh proc cpu-usage non-zero
Hardware: ASA5516
Cisco Adaptive Security Appliance Software Version 9.16(4)57
ASLR enabled, text region 56474105f000-564744cef285
PC Thread 5Sec 1Min 5Min Process
0x0000564743c1c050 0x00007f2cef4bbe80 1.0% 1.5% 1.0%
Unicorn Proxy Thread
0x0000564743a1b57b 0x00007f2cef4bb000 0.0% 0.2% 0.3%
0x00005647439821d6 0x00007f2cef4bbae0 0.0% 0.2% 0.1%
snmp_master_callback_thread
0x0000564743982226 0x00007f2cef4bb740 0.0% 0.4% 0.4%
snmp_client_callback_thread
0x00005647437cf58c 0x00007f2cef4be2c0 0.0% 0.1% 0.1%
radius_snd
0x00005647421c5dd4 0x00007f2cef4dc4e0 52.1% 11.5% 11.1% Logger
0x00005647427c3cb6 0x00007f2cef4c5e00 0.0% 0.1% 0.1% ARP
Thread
0x0000564743c1c050 0x00007f2cef4ebf00 0.0% 0.1% 0.1%
aaa_shim_thread
0x0000564741cdf31c 0x00007f2cef4ec640 0.0% 0.1% 0.1% aaa
- - 0.8% 2.1% 2.2% DATAPATH-0-1665
- - 2.6% 2.3% 2.4% DATAPATH-1-1666
vpn-gw# sh proc cpu-usage non-zero
Hardware: ASA5516
Cisco Adaptive Security Appliance Software Version 9.16(4)57
ASLR enabled, text region 56474105f000-564744cef285
PC Thread 5Sec 1Min 5Min Process
0x0000564743bf433f 0x00007f2cef4bbe80 0.0% 1.2% 1.0%
Unicorn Proxy Thread
0x0000564743a1b57b 0x00007f2cef4bb000 0.0% 0.1% 0.2%
0x00005647439821d6 0x00007f2cef4bbae0 0.0% 0.1% 0.1%
snmp_master_callback_thread
0x0000564743982226 0x00007f2cef4bb740 0.0% 0.2% 0.4%
snmp_client_callback_thread
0x00005647421c5dd4 0x00007f2cef4dc4e0 64.1% 12.7% 11.3% Logger
0x00005647427c3cb6 0x00007f2cef4c5e00 0.0% 0.1% 0.1% ARP
Thread
- - 1.3% 2.1% 2.2% DATAPATH-0-1665
- - 4.7% 2.5% 2.4% DATAPATH-1-1666
vpn-gw# sh proc cpu-usage non-zero
Hardware: ASA5516
Cisco Adaptive Security Appliance Software Version 9.16(4)57
ASLR enabled, text region 56474105f000-564744cef285
PC Thread 5Sec 1Min 5Min Process
0x0000564743c1c050 0x00007f2cef4bbe80 0.0% 0.9% 0.9%
Unicorn Proxy Thread
0x0000564743a1b57b 0x00007f2cef4bb000 0.0% 0.1% 0.2%
0x00005647439821d6 0x00007f2cef4bbae0 0.0% 0.1% 0.0%
snmp_master_callback_thread
0x0000564743982226 0x00007f2cef4bb740 0.0% 0.1% 0.3%
snmp_client_callback_thread
0x00005647421c5dd4 0x00007f2cef4dc4e0 84.0% 15.1% 11.8% Logger
- - 3.0% 2.3% 2.3% DATAPATH-0-1665
- - 0.4% 2.3% 2.4% DATAPATH-1-1666
vpn-gw# sh proc cpu-usage non-zero
Hardware: ASA5516
Cisco Adaptive Security Appliance Software Version 9.16(4)57
ASLR enabled, text region 56474105f000-564744cef285
PC Thread 5Sec 1Min 5Min Process
0x0000564743c1c050 0x00007f2cef4bbe80 1.3% 1.0% 0.9%
Unicorn Proxy Thread
0x0000564743a1b57b 0x00007f2cef4bb000 0.0% 0.1% 0.2%
0x0000564743982226 0x00007f2cef4bb740 0.0% 0.1% 0.3%
snmp_client_callback_thread
0x00005647421c5dd4 0x00007f2cef4dc4e0 0.0% 12.8% 11.4% Logger
0x000056474221df82 0x00007f2cef4bc5c0 0.1% 0.1% 0.1%
emweb/https
0x00005647427c3cb6 0x00007f2cef4c5e00 0.1% 0.0% 0.0% ARP
Thread
- - 2.2% 2.3% 2.3% DATAPATH-0-1665
- - 2.4% 2.3% 2.4% DATAPATH-1-1666
vpn-gw# sh proc cpu-usage non-zero
Best,
-Lee
On Wed, Jun 5, 2024 at 2:22 PM Lee Starnes <lee.t.starnes at gmail.com> wrote:
> Thank you for the link and info. Unfortunately can['t open a TAC case as
> this model (5516-X) is not under support. We have a 5508-X under contract
> which is how we are able to get the firmware.
>
> I will check out the links. Thank you for your help.
>
> Best,
>
> -Lee
>
> On Wed, Jun 5, 2024 at 6:15 AM harbor235 <harbor235 at gmail.com> wrote:
>
>> Here is an overall performance troubleshooting oc:
>>
>>
>> https://www.cisco.com/c/en/us/support/docs/security/asa-5500-x-series-next-generation-firewalls/113185-asaperformance.html
>>
>> Mike
>>
>> On Wed, Jun 5, 2024 at 9:12 AM harbor235 <harbor235 at gmail.com> wrote:
>>
>>> If you cannot open a TAc case I would look through your syslog messages
>>> looking for errors/critcals/warnings. Also look at all interfaces to ensure
>>> there are no input or output errors as well. After that I would verify
>>> traffic is hitting your box and is not an upstream problem. While looking
>>> at the interfaces for errors here is a good doc:
>>>
>>>
>>> https://www.cisco.com/c/en/us/support/docs/security/asa-5500-x-series-next-generation-firewalls/115985-asa-overrun-product-tech-note-00.pdf
>>>
>>> Do you have a router fronting the FW? Make sure you look at all
>>> interfaces for high traffic or traffic directed to the FW itself.
>>>
>>> Hard to tell what it could be without more info.
>>>
>>> Mike
>>>
>>> On Tue, Jun 4, 2024 at 7:41 PM harbor235 <harbor235 at gmail.com> wrote:
>>>
>>>> Hi Lee,
>>>>
>>>> So logger looks like it is consuming a bit of CPU, it is worth noting
>>>> though the data and management plane separation. Logger should not impact
>>>> packet forwarding in the data plane.
>>>>
>>>> Hard to tell what is going on, you must have a support contract you
>>>> upgraded the code, did you try to open a TAC case?
>>>>
>>>>
>>>> Mike
>>>>
>>>> On Tue, Jun 4, 2024 at 7:24 PM Lee Starnes <lee.t.starnes at gmail.com>
>>>> wrote:
>>>>
>>>>> The unit just runs basic NAT-PAT and AnyConnect VPN. No IPS or
>>>>> anything setup on this. And only about 9Mbps peak.
>>>>>
>>>>> -Lee
>>>>>
>>>>> On Tue, Jun 4, 2024 at 4:03 PM harbor235 <harbor235 at gmail.com> wrote:
>>>>>
>>>>>> What features do you have enabled? NGFW and/or NGIPS? These features
>>>>>> can limit the box to 450Mbps.
>>>>>>
>>>>>> Mike
>>>>>>
>>>>>> On Tue, Jun 4, 2024 at 6:53 PM Lee Starnes via cisco-nsp <
>>>>>> cisco-nsp at puck.nether.net> wrote:
>>>>>>
>>>>>>> Hello Everyone,
>>>>>>>
>>>>>>> I have an odd issue trying to track down. We are seeing issue whereby
>>>>>>> traffic just "pauses" through the ASA for about 2-4 seconds before
>>>>>>> resuming.
>>>>>>>
>>>>>>> We started seeing this when the device was low on memory (about 600M
>>>>>>> available). we rebooted it and did an firmware update the current
>>>>>>> version.
>>>>>>>
>>>>>>> Still seeing this behavior.
>>>>>>>
>>>>>>> After another reboot, still seeing this.
>>>>>>>
>>>>>>> Process: DATAPATH-0-1665, PROC_PC_TOTAL: 407, MAXHOG: 10,
>>>>>>> LASTHOG: 5
>>>>>>> MAXHOG At: 15:31:54 PDT Jun 4 2024
>>>>>>> LASTHOG At: 15:37:48 PDT Jun 4 2024
>>>>>>> PC: 0x0000000000000000 (suspend)
>>>>>>>
>>>>>>> Process: DATAPATH-0-1665, NUMHOG: 385, MAXHOG: 10, LASTHOG: 5
>>>>>>> MAXHOG At: 15:31:54 PDT Jun 4 2024
>>>>>>> LASTHOG At: 15:37:48 PDT Jun 4 2024
>>>>>>> PC: 0x0000000000000000 (suspend)
>>>>>>> Call stack: 0x0000564741c98c49 0x0000564742188996
>>>>>>> 0x00005647436c2d28
>>>>>>> 0x00005647436d2abc 0x00005647436e2ae0
>>>>>>> 0x00007f2d2067bff5
>>>>>>> 0x00007f2d1f88416f
>>>>>>>
>>>>>>>
>>>>>>> Process: DATAPATH-1-1666, PROC_PC_TOTAL: 402, MAXHOG: 12,
>>>>>>> LASTHOG: 5
>>>>>>> MAXHOG At: 15:31:48 PDT Jun 4 2024
>>>>>>> LASTHOG At: 15:37:41 PDT Jun 4 2024
>>>>>>> PC: 0x0000000000000000 (suspend)
>>>>>>>
>>>>>>> Process: DATAPATH-1-1666, NUMHOG: 376, MAXHOG: 12, LASTHOG: 5
>>>>>>> MAXHOG At: 15:31:48 PDT Jun 4 2024
>>>>>>> LASTHOG At: 15:37:41 PDT Jun 4 2024
>>>>>>> PC: 0x0000000000000000 (suspend)
>>>>>>> Call stack: 0x0000564741c98c49 0x0000564742188996
>>>>>>> 0x00005647436c2d28
>>>>>>> 0x00005647436d2abc 0x00005647436e2ae0
>>>>>>> 0x00007f2d2067bff5
>>>>>>> 0x00007f2d1f88416f
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I did disable logging flash-bufferwrap to stop it from writing to
>>>>>>> flash.
>>>>>>> The logging process stopped using 29% CPU, but still the issue
>>>>>>> persists.
>>>>>>>
>>>>>>> Anyone got any Ideas on what the cause is and how to resolve it?
>>>>>>>
>>>>>>> Best,
>>>>>>>
>>>>>>> -Lee
>>>>>>> _______________________________________________
>>>>>>> cisco-nsp mailing list cisco-nsp at puck.nether.net
>>>>>>> https://puck.nether.net/mailman/listinfo/cisco-nsp
>>>>>>> archive at http://puck.nether.net/pipermail/cisco-nsp/
>>>>>>>
>>>>>>
More information about the cisco-nsp
mailing list