Saucepan

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Troubleshooting Firewall Performance ======
===== CPU Spikes =====
Could be caused by [[https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA14u000000HAcfCAG|genindex.sh]].

Could be caused by [[https://knowledgebase.paloaltonetworks.com/kcSArticleDetail?id=kA10g000000CmQv|IPsec tunnels]].

===== Performance Issues =====
Possible but in PAN-OS 10.0.6 but we have seen a case where disabling "Forward segments exceeding TCP content inspection queue" cause massive throughput hit.
===== Show Session Limits =====
<code>show session info</code>
<code>show session info | match active</code>
===== Show Resources =====
Show record of each second for the last 60 seconds.
<code>show running resource-monitor second</code>
Show record of each minutefor the last 60 minutes.
<code>show running resource-monitor minute</code>
Show record of each hourfor the last 24 hours.
<code>show running resource-monitor hour</code>
Show record of each dayfor the last 7 days.
<code>show running resource-monitor day</code>
Show record of each weekfor the last 13 weeks.
<code>show running resource-monitor week</code>
Show all records - seconds, minutes, hours, days, weeks.
<code>show running resource-monitor</code>

This is the most important command in getting dataplane CPU usages over different time intervals. Usually, if the CPU stays high (>90), traffic would feel sluggish, latency would also rise. The best strategy is to determine a regular 24-hour usage ("baseline") and then compare it to the times when spikes are experienced.
Resource utilization gives the % usage of sessions and buffers. If the output of packet descriptors or buffers stays > 80%, it indicates the device is over loaded and may lead to packet loss and abnormal behavior of the device.
The same output can be obtained from the dp-monitor log.
<code>less mp-log dp-monitor.log</code>
  * **VM-100** - Has Core 0 - 3 listed but only have data for Core 1.
  * **VM-300**, **PA-220** - Have Core 0 - 3 listed but only have data for Core 1 - 2.
  * **PA-850** - Has Core 0 - 7 listed but only has data for Core 1 - 5.
  * **PA-5220** - Has Core 0 - 7 listed but only has data for Core 1 - 7.
===== Buffer Usage =====
<code>debug dataplane pool statistics</code>
<code>debug dataplane pool statistics | match Packet</code>
This command shows the packet buffers, resource pools and memory cache usages by different processes. If the pools deplete, traffic performance will be affected corresponding to that particular resource pool. Regarding pools, the number of the left shows the remaining while the number on the right shows the total capacity. The total capacity can vary based on platforms, models and OS versions. Likewise, if a certain process uses too much memory, that can also cause issues related to that process.

For Packet Buffer Usage, run the following filter in the System Logs view
<code>( description contains 'Packet buffer congestion is' )</code>

To show the top five session using more than 2% of packet buffers: (this only works on physical firewalls)
<code>show running resource-monitor ingress-backlocs </code>
shows top five session using more than 2% of packet bufer.
Look for ''unknown'' or ''undecided'' App-ID and kill off with 
<code>request session-discard id <session_id></code>

===== Show Counters =====
<code>show counter global</code>
<code>show counter global filter severity drop</code>
<code>show counter global | match deny</code>
<code>show counter global | match error </code>
This command lists all the counters available on the firewall for the given OS version. For every packet that arrives, traverses or even gets dropped, we should see one or more counters go up. These are extremely powerful in troubleshooting traffic related issues when combined with packet-filter.

===== Show Network Data =====
<code>show arp all</code>
If you are using the firewall in Layer 2 mode, the following command is the Layer 2 equivalent of the arp command above.
<code>show mac all</code>

===== Show Logging Data =====
<code>debug log-receiver statistics</code>
<code>debug log-receiver statistics | match discard</code>
Check that "Logs discarded (queue full)" is zero; if it is nonzero, you are losing logs and are probably exceeding the platform limits for logging.

===== Show Decryption Data ====
<code>show system setting ssl-decrypt memory</code>
This command is used to monitor the ssl decryption memory usage; the first sz malloc size is the value to track. This value should increment/decrement, it is a concern if this value only increments. The max value is around 16 mb.

The following command shows all decrypted sessions currently in the session table
<code>show session all filter ssl-decrypt yes</code>
<code>show session all filter ssl-decrypt yes count yes</code>
===== Show GlobalProtect Data ====
<code>show resource limit ssl-vpn</code>
This command shows the number of current number of connected GlobalProtect sessions (both the SSL sessions and the IPsec VPN sessions).

You can show the GlobalProtect cookie cache on the firewall.
<code>show system setting ssl-decrypt gp-cookie-cache</code>