User Tools

Site Tools


paloaltonetworks:troubleshooting:testing_panos

This is an old revision of the document!


Testing PAN-OS

This page lists various methods for testing configuration on a Palo Alto Networks firewall

Set VSYS

If you are working on a multi-vsys appliance, use the following command to switch to the appropriate vsys.

set system setting target-vsys <vsys-name>

Test Managment Connectivity

The following test command can be used to see if the managment interface can establish connectivty to a remote host.

test http-server port 443 address updates.paloaltonetworks.com protocol HTTPS

Test Panorama Firewall Connections

To list the connections from firewalls to Panorama's managment interface on Panorama, us the following command.

show netstat numeric-hosts yes numeric-ports yes | match 3978

Authentication Profiles

LDAP

test authentication authentication-profile LDAP-Profile username User4-LDAP password

Kerberos

test authentication authentication-profile Kerberos-Profile username User5-Kerberos password

RADIUS

test authentication authentication-profile RADIUS-Profile username User2-RADIUS password

TACACS+

test authentication authentication-profile TACACS-Profile username User3-TACACS password

Security Profiles

DNS Sinkholing

To test DNS sinkholing, access one of the workstations that will be protected by the DNS sinkhole and do an nslookup on a malware domain.

To find a malware domain, log into the Palo Alto Networks support portal. Under the home tab, click “Dynamic Updates” to see the list of the latest dynamic updates.

Click on the release notes for the latest Antivirus definition.

Look for a line that has the format

generic:suspicouslettering | 1 variants: com

Then run <script>nslookup suspicouslettering.com</script> on the workstation.

You should get the sinkhole address back and see an entry in the threat log.

DNS Security

If protection is not active, the following domains resolve to 72.5.65.115.

If protection is active, the following domains resolve to whatever your sinkhole is set to (e.g. 72.5.65.111) or nothing if block is set as the action instead of sinkhole.

Don't block Ad Tracking Domains. This will block anything that uses CNAME. This affects most major sites.

  • Ad Tracking
    • CNAME Cloaking
  • Command and Control
    • Tunnelling
    • Infiltration
    • NXNS
    • Rebinding
    • DGA
  • Dynamic DNS
  • Grayware
    • FastFlux
    • Malicious NRD
    • Dangling Domain
    • Wildcard Abuse
    • Strategically Aged
  • Parked
  • Phishing
  • Proxy Avoidance
  • Newly Registered Domains
Default Log Severity Threat Category Test Domain Test Command
informational dns-adtracking test-adtracking.testpanw.com dig +short @192.168.99.9 A test-adtracking.testpanw.com
informational dns-adtracking test-cname-cloaking.testpanw.com dig +short @192.168.99.9 A test-cname-cloaking.testpanw.com
high dns-c2 test-c2.testpanw.com dig +short @192.168.99.9 A test-c2.testpanw.com
high dns-c2 test-dnstun.testpanw.com dig +short @192.168.99.9 A test-dnstun.testpanw.com
high dns-c2 test-dns-infiltration.testpanw.com dig +short @192.168.99.9 A test-dns-infiltration.testpanw.com
high dns-c2 test-nxns.testpanw.com dig +short @192.168.99.9 A test-nxns.testpanw.com
high dns-c2 test-dns-rebinding.testpanw.com dig +short @192.168.99.9 A test-dns-rebinding.testpanw.com
high dns-c2 test-dga.testpanw.com dig +short @192.168.99.9 A test-dga.testpanw.com
informational dns-ddns test-ddns.testpanw.com dig +short @192.168.99.9 A test-ddns.testpanw.com
low dns-grayware test-fastflux.testpanw.com dig +short @192.168.99.9 A test-fastflux.testpanw.com
low dns-grayware test-malicious-nrd.testpanw.com dig +short @192.168.99.9 A test-malicious-nrd.testpanw.com
low dns-grayware test-dangling-domain.testpanw.com dig +short @192.168.99.9 A test-dangling-domain.testpanw.com
low dns-grayware test-wildcard-abuse.testpanw.com dig +short @192.168.99.9 A test-wildcard-abuse.testpanw.com
low dns-grayware test-strategically-aged.testpanw.com dig +short @192.168.99.9 A test-strategically-aged.testpanw.com
medium dns-malware test-malware.testpanw.com dig +short @192.168.99.9 A test-malware.testpanw.com
medium dns-malware test-compromised-dns.testpanw.com dig +short @8.8.8.8 A test-compromised-dns.testpanw.com
informational dns-parked test-parked.testpanw.com dig +short @192.168.99.9 A test-parked.testpanw.com
low dns-phishing test-phishing.testpanw.com dig +short @192.168.99.9 A test-phishing.testpanw.com
low dns-proxy test-proxy.testpanw.com dig +short @192.168.99.9 A test-proxy.testpanw.com
low dns-new-domain test-nrd.testpanw.com dig +short @192.168.99.9 A test-nrd.testpanw.com
show dns-proxy dns-signature info
test dns-proxy dns-signature fqdn
dig +short @8.8.8.8 A test-malware.testpanw.com
dig +short @8.8.8.8 A test-c2.testpanw.com
dig +short @8.8.8.8 A test-phishing.testpanw.com
dig +short @8.8.8.8 A test-grayware.testpanw.com
dig +short @8.8.8.8 A test-proxy.testpanw.com
dig +short @8.8.8.8 A test-parked.testpanw.com
dig +short @8.8.8.8 A test-adtracking.testpanw.com

dig +short @8.8.8.8 A test-dga.testpanw.com
dig +short @8.8.8.8 A test-fastflux.testpanw.com

dig +short @8.8.8.8 A test-ddns.testpanw.com
dig +short @8.8.8.8 A test-nrd.testpanw.com
dig +short @8.8.8.8 A test-malicious-nrd.testpanw.com

dig +short @8.8.8.8 A test-dnstun.testpanw.com
dig +short @8.8.8.8 A test-dns-infiltration.testpanw.com
dig +short @8.8.8.8 A test-nxns.testpanw.com
dig +short @8.8.8.8 A test-dangling-domain.testpanw.com
dig +short @8.8.8.8 A test-dns-rebinding.testpanw.com
dig +short @8.8.8.8 A test-malicious-nrd.testpanw.com
dig +short @8.8.8.8 A test-wildcard-abuse.testpanw.com
dig +short @8.8.8.8 A test-strategically-aged.testpanw.com
dig +short @8.8.8.8 A test-compromised-dns.testpanw.com

dig +short @8.8.8.8 A test-cname-cloaking.testpanw.com

Vulnerability Protection

As listed in this article.

  1. Go to any http site with a search bar.
  2. Enter the following in the search bar and then click the search button:
    <script>test</script>
  3. A vulnerability log should be generated under the threat log. This shows that the vulnerability profile is working properly and generating log entries.

Antivirus

Try and download the following harmless test file from EICAR (European Institute for Computer Anti-Virus Research) that all Antivirus vendors should flag as 'malware'.

http://www.eicar.org/download/eicar.com
http://www.eicar.org/download/eicar.com.txt
http://www.eicar.org/download/eicar_com.zip
http://www.eicar.org/download/eicarcom2.zip
https://secure.eicar.org/eicar.com
https://secure.eicar.org/eicar.com.txt
https://secure.eicar.org/eicar_com.zip
https://secure.eicar.org/eicarcom2.zip

WildFire Protection

Download this file and see if WildFire protects against it. Make sure that there is a WildFire policy checking for PE files. Palo's test web site is here.

You should get a result within 5 - 10 minutes. I found I normally get results for the test file within 7 minutes.

I found on one deployment (lab) that I had to disable the decryption excption for *.wildfire.paloaltonetworks.com to get this working. When I tested and then re-enabled the exception, things continued to work fine. (Device→Certificate Managment→SSL Decryption Exclusion→Seach for wildfire, select it and disable it.)

http://wildfire.paloaltonetworks.com/publicapi/test/pe

To test WildFire uploades, log into the CLI and run the following and look for

upload success
debug wildfire upload-log show

If you have SSL decryption enabled…

  • PE
    https://wildfire.paloaltonetworks.com/publicapi/test/pe
  • APK
    https://wildfire.paloaltonetworks.com/publicapi/test/apk
  • MacOSX
    https://wildfire.paloaltonetworks.com/publicapi/test/macos
  • ELF
    https://wildfire.paloaltonetworks.com/publicapi/test/elf

If you do NOT have SSL decryption enabled…

  • PE
    http://wildfire.paloaltonetworks.com/publicapi/test/pe
  • APK
    http://wildfire.paloaltonetworks.com/publicapi/test/apk
  • MacOSX
    http://wildfire.paloaltonetworks.com/publicapi/test/macos
  • ELF
    http://wildfire.paloaltonetworks.com/publicapi/test/elf

DoS Protection

You can test DoS Protection (TCP/flood). In the following command, we run a slowloris attack (dos attack to see if the web server can cope with move than 10 concurrent sessions.

nmap --script http-slowloris --max-parallelism 10 <target_ip>
  • –script http-slowloris “Test web server target for slowloris culnerbaility”
  • –max-parallelism <num> “Open <num> connections in parallel”

REMEMBER Press the up arrow as the scan runs to get updates on progress.

You just want the 'Connect Scan Timing' bit.

This will take a few seconds to run and then it will start NSE Timing which will take ages and we don't need. Use CTRL+C to cancel the attack test when we reach this bit.

Zone Protection

As listed in this article.

Run nmap against an address hosted within the target zone with this command

nmap -p 1-65535 -T4 -A -v <target_ip>

TCP Scan Attack

nmap -v1 -Pn -T4 --max-retries 1 <target_ip>
  1. -v1 “Set verbosity level to 1”
  2. -Pn “Skip host discovery by treating all hosts as online”
  3. -T4 “Use template 4 (provides faster command execution)”
  4. –max-retries 1 “Retry only once if port is unreachable”

If you have Zone Protection enabled, you will notice that there will be very few Threat logs created when Zone protection drops traffic.

To see if Zone protection is doing anything, you will need to run the following command

show counter global filter packet-filter yes delta yes | match Zone

Test Spoof IP

The following command will spoof the source IP address. This can be defended against using the “Spoofed IP Address” option on the Zone Protection Profile. We can cause a UDP flood with the following command.

hping3 --flood --rand-source --udp -p 80 192.168.1.1
show counter global filter packet-filter yes delta yes | match discard-ip-spoof
  • Zone Protect Description : Spoofed IP Address
  • Name : flow_dos_pf_ipspoof
  • Description: Packets dropped: Zone protection option 'discard-ip-spoof'
  • Severity : drop
  • Category : flow
  • Aspect : dos

You can also use NMap

nmap -e eth0 -p443 -S <source_ip> <target_ip>
  • -e <interface> “Interface to use to connect to the target host”
  • -p<n> “Port number to which to connect on the target host”
  • -S <IP> “Source IP (spoofed IP to use to connect to the target host>”

Test Ping Broadcast Flood

ping -b -f 192.168.39.255 -v

Test UDP Flood

The following command will cause a UDP flood

hping3 -c 20000 -S -P -U -V --flood --rand-source 192.168.1.1
  • -c 20000 means we send 20000 packets
  • -S means set SYN flag
  • -P means set PUSH flag
  • -U means set URG flag
  • -V means verbose mode
  • –flood means we send the packets as fast as possible
  • –rand-source means we genreate spoofed IP addresses.
  • 192.168.1.1 the victim IP address

Logs will show up in the Threat Log.

( subtype eq flood ) and ( name-of-threatid eq 'UDP Flood' ) and ( severity eq critical )
show counter global filter packet-filter yes delta yes | match flow_dos_red_udp
  • Zone Protect Description : Flood Protection → UDP
  • Name : flow_dos_red_udp
  • Description: Packets dropped: Zone protection protocol 'udp' RED
  • Severity : drop
  • Category : flow
  • Aspect : dos

Test TCP Flood

The following command will cause a TCP SYN flood to TCP port 80.

hping3 -c 20000 -d 120 -S -w 64 -p 80 --flood --rand-source 192.168.1.1
  • -c 20000 means we send 20000 packets
  • -d 120 means the packets are 120 bytes in size
  • -w 64 means that the TCP window size is 64
  • -p 80 means we are targeting TCP port 80
  • –flood means we send the packets as fast as possible
  • –rand-source means we generate spoofed IP addresses.
  • 192.168.1.1 the victim IP address

You can also use NMap

nping --tcp-connect -p 80 --rate 100000 -c 10 -q <target_ip>
  • –tcp-connect “Scan using TCP mode”
  • -p “Port number to which we connect”
  • –rate “Number of packets per second to send in each round”
  • -c “Number of connections to complete”
  • -q “Reduce the output verbosity by one level which makes the output easier to read”.

Logs will show up in the Threat Log.

( subtype eq flood ) and ( name-of-threatid eq 'TCP Flood' ) and ( severity eq critical )
show counter global filter packet-filter yes delta yes | match flow_dos_red_tcp
  • Zone Protect Description : Flood Protection → UDP
  • Name : flow_dos_red_tcp
  • Description: Packets dropped: Zone protection protocol 'tcp-syn' RED
  • Severity : drop
  • Category : flow
  • Aspect : dos

Test ICMP Flood

The following command will cause a TCP SYN flood to TCP port 80.

hping3 -1 -c 20000 --flood --rand-source 192.168.1.1

hping3 -c 20000 -d 120 -S -w 64 -p 80 –flood –rand-source 192.168.1.1</code>

  • -1 means we are flooding ICMP rather than TCP or UDP.
  • -c 20000 means we send 20000 packets
  • –flood means we send the packets as fast as possible
  • –rand-source means we generate spoofed IP addresses.
  • 192.168.1.1 the victim IP address

Logs will show up in the Threat Log.

( subtype eq flood ) and ( name-of-threatid eq 'TCP Flood' ) and ( severity eq critical )
show counter global filter packet-filter yes delta yes | match flow_dos_red_tcp
  • Zone Protect Description : Flood Protection → UDP
  • Name : flow_dos_red_tcp
  • Description: Packets dropped: Zone protection protocol 'tcp-syn' RED
  • Severity : drop
  • Category : flow
  • Aspect : dos

Test ICMP Protection

If you run a traceroute from Inside to Outside with zone protection profile applied to the outside zone.

  • Zone Protect Description : ICMP Drop → Suppress ICMP TTL Expired Error
  • Name : flow_dos_pf_noreplyttl
  • Description: Packets dropped: Zone protection option 'suppress-icmp-timeexceeded'
  • Severity : drop
  • Category : flow
  • Aspect : dos

If you run a traceroute from Inside to Outside with zone protection profile applied to the inside zone.

  • Zone Protect Description : ICMP Drop → Discard ICMP embedded with error message
  • Name : fflow_dos_pf_icmperr
  • Description: Packets dropped: Zone protection option 'discard-icmp-error'
  • Severity : drop
  • Category : flow
  • Aspect : dos

Test ICMP Large Packet Protection

You can run the following command on Windows.

ping -t -l 65500 192.168.1.1 

Or, if you are on Linux

ping -t -s 65500 192.168.1.1
show counter global filter packet-filter yes delta yes | match discard-icmp-large-packet
  • Zone Protect Description : Zone Protect Description : ICMP Drop → ICMP Large Packet(>1024)
  • Name : flow_dos_pf_icmplpkt
  • Description: Packets dropped: Zone protection option 'discard-icmp-large-packet'
  • Severity : drop
  • Category : flow
  • Aspect : dos

It will also trigger the following

  • Zone Protect Description : Zone Protect Description : ICMP Drop → ICMP Fragment
  • Name : flow_dos_pf_icmplpkt
  • Description: Packets dropped: Zone protection option 'discard-icmp-frag'
  • Severity : drop
  • Category : flow
  • Aspect : dos

Test IP Drops

Strict IP Address Check will probably trigger alerts if you run a UDP Flood as shown above.

  • Zone Protect Description : Strict IP Address Check
  • Name : flow_dos_pf_strictip
  • Description: Packets dropped: Zone protection option 'strict-ip-check'
  • Severity : drop
  • Category : flow
  • Aspect : dos

Test TCP Drops

I found the following command seems to include data in the TCP SYN packet before it triggers a flood attack (see above).

hping3 -c 20000 -d 120 -S -w 64 -p 80 --flood --rand-source 192.168.1.1
  • Zone Protect Description : Zone Protect Description : TCP Drop → TCP SYN with Data
  • Name : flow_dos_pf_tcpsyndata
  • Description: Packets dropped: Zone protection option 'discard-tcp-syn-with-data'
  • Severity : drop
  • Category : flow
  • Aspect : dos
  • Zone Protect Description : Zone Protect Description : TCP Drop → TCP SYN ACK with Data
  • Name : flow_dos_pf_tcpsynackdata
  • Description: Packets dropped: Zone protection option 'discard-tcp-synack-with-data'
  • Severity : drop
  • Category : flow
  • Aspect : dos
  • Zone Protect Description : Zone Protect Description : TCP Drop → TCP Timestamp
  • Name : flow_dos_pm_tcptimestamp
  • Description: Packets modified: Zone protection option 'remove-tcp-timestamp'
  • Severity : info
  • Category : flow
  • Aspect : dos
  • Zone Protect Description : Zone Protect Description : TCP Drop → TCP Fast Open
  • Name : flow_dos_pm_tcptfodata
  • Description: Packets modified: Zone protection option 'strip-tcp-fast-open-and-data'
  • Severity : info
  • Category : flow
  • Aspect : dos

VPN

Initiate IKE phase 1 by either pinging a host across the tunnel or using the following CLI command:

test vpn ike-sa gateway gateway_name

Then enter the following command to test if IKE phase 1 is set up:

show vpn ike-sa gateway gateway_name

In the output, check if the Security Association displays. If it does not, review the system log messages to interpret the reason for failure. Initiate IKE phase 2 by either pinging a host from across the tunnel or using the following CLI command:

test vpn ipsec-sa tunnel tunnel_name

Then enter the following command to test if IKE phase 1 is set up:

show vpn ipsec-sa tunnel tunnel_name

In the output, check if the Security Association displays. If it does not, review the system log messages to interpret the reason for failure. To view the VPN traffic flow information, use the following command:

show vpn-flow 

You can also start all tunnels with

test vpn ipsec-sa

Test WMI

In some cases the WMI probe will fail because the workstation may be running a local firewall or it may not be a member of the domain. If this happens, the mapping can be deleted once the cache timeout is exceeded, even though the workstation is up and passing traffic. To test, run the following command from the User-ID Agent device (which must be connected to the Domain).

wmic /node:workstationIPaddress computersystem get username

E.G.

> wmic /node:10.1.1.1 computersystem get username
username
example\testuser

User-ID Group Mapping

To list the number of group mappings, run the following command.

show user group list

To list the members of a particular group show in the results of the previous command, run the following command.

show user group name "cn=some groupname with whitespace,ou=AnOUname,ou=AnotherOUname,dc=example,dc=com"

To force the Palo to refresh the members of groups from a particular group mapping, run the following.

debug user-id reset group-mapping NameOfGroupMapping

Good Tests to Run After Deploying a New Palo Alto Networks Firewall

Gratuitous ARP

If devices upstream of the Palo firewall cannot be updated to refresh their ARP cache, try runninging the following command to force an ARP cache update.

test arp gratuitous ip <ip/netmask> interface <interface name>
DNS

Ensure the managment plane can successfully perform DNS lookups with a ping command to a FQDN (required PANOS 8.0 or later).

request resolve address eu.wildfire.paloaltonetworks.com
NTP

Ensure that the NTP servers configured are actually usable.

show ntp

You can test NTP manually using Windows using

w32tm /stripchart /computer:192.168.1.1
Ensure that PANDB is Reachable

If the firewall is licenced for PANDB URL filtering, make sure that the firewall has successufully connected to the PANDB cloud.

show url-cloud status
Ensure WildFire Connectivity

If the firewall is licenced for WildFire, make sure it can communicate with the WildFire cloud.

test wildfire registration channel public
Check Duplex Settings

For every physical interface, run the following command an ensure that the speed is 1000/full (or 100/full if it is connected to a 100Mb appliance).

show interface ethernet1/1 | match duplex 
Arp Table

Make sure the Arp table is not full. Specifically, don't configure a /16 subnet on a small firewall (e.g. PA-220) as this may well cause your ARP table to flood which will lead to session failure.

show arp all
System Files
show system files
show system disk-space
System Environmentals

Ideal operating temperature is 1C to 27C; Operating temperature can span from 0C to 50C.

show system environmentals
Logs
debug log-receiver statistics
Resourse History
show running resource-monitor
show system software status
Example Results

Here are example results of the tests above.

username@hostname> request resolve address bbc.co.uk

151.101.64.81
151.101.0.81
151.101.128.81
151.101.192.81
2a04:4e42::81
2a04:4e42:600::81
2a04:4e42:400::81
2a04:4e42:200::81
username@customerfirewall> show ntp
NTP state:
    NTP synched to 192.168.1.100
    NTP server: 192.168.1.100
        status: synched
        reachable: yes
        authentication-type: none
    NTP server: 192.168.1.101
        status: available
        reachable: yes
        authentication-type: none
username@customerfirewall> show url-cloud status

PAN-DB URL Filtering
License :                          valid
Current cloud server :             s0300.urlcloud.paloaltonetworks.com
Cloud connection :                 connected
Cloud mode :                       public
URL database version - device :    20180814.40125
URL database version - cloud :     20180814.40125  ( last update time 2018/08/15 11:31:45 )
URL database status :              good
URL protocol version - device :    pan/0.0.2
URL protocol version - cloud :     pan/0.0.2
Protocol compatibility status :    compatible
username@customerfirewall> test wildfire registration channel public
This test may take a few minutes to finish. Do you want to continue? (y or n)

Test wildfire Public Cloud

        Testing cloud server eu.wildfire.paloaltonetworks.com ...
        wildfire registration:         successful
        download server list:          successful
        select the best server:        eu-panos.wildfire.paloaltonetworks.com

<preusername@customerfirewall> show interface ethernet1/1 | match duplex

Runtime link speed/duplex/state: 1000/full/up
Configured link speed/duplex/state: auto/auto/auto</code>
username@customerfirewall> show arp ethernet1/1

maximum of entries supported :      1500
default timeout:                    1800 seconds
total ARP entries in table :        1
total ARP entries shown :           1
status: s - static, c - complete, e - expiring, i - incomplete

interface         ip address      hw address        port              status   ttl
--------------------------------------------------------------------------------
ethernet1/1       192.168.10.10   ab:cd:ef:12:34:56 ethernet1/1         c      1290

This is an example of bad NTP

username@customerfirewall> show ntp

NTP state:
    NTP not synched, using local clock
    NTP server: 192.168.1.1
        status: rejected
        reachable: no
        authentication-type: none
    NTP server: 192.168.1.2
        status: rejected
        reachable: no
        authentication-type: none

Search Log Files

If you want to search a whole log file…

grep pattern "pattern_to_search" mp-log *
paloaltonetworks/troubleshooting/testing_panos.1674314223.txt.gz · Last modified: by bstafford