This is an old revision of the document!
Table of Contents
SSL Decryption
Overview
You will find that SSL decryption on Palo Alto devices has its limitations. If SSL decryption is enabled and yet some sites do not work in HTTPS mode, you may find that the site does not support any of the ciphers listed below.
Also, remember that Palo Alto cannot decrypt these applications. Generally, these applications can't be decrypted because they deviate from SSL encryption standards. They may use proprietary encryption, require a specific type of certificate or be unable to add new certificate authorities. Decryption would break them so that they no longer work. This list can't be manipulated to force decryption.
I have found that when 'any/any/any' decryption is in place, you won't get logs for sessions that a blocked because decryption is breaking them. Try disabling decryption temporarily, use the program that isn't working. This will verify that a) the decryption is what is causing the issue and b) the URLs that are being accessed by the application. Add those domains to the exclude list for decryption and see if it fixes the problem (not forgetting to enable decryption again).
Test
By default, if a handshake error occurs when the firewall is trying to do the decryption it will add the IP-port to the ssl-decrypt exclude-cache. You can view it with:
show system setting ssl-decrypt exclude-cache
To get statistics, run,
debug dataplane show ssl decrypt ssl-stats
To disabled decryption across the whole firewall (WITHOUT COMMITING), run
set system setting ssl-decrypt skip-ssl-decrypt yes
to re-enabled it, run
set system setting ssl-decrypt skip-ssl-decrypt no
You will find the following
- session reason end - decrypt-cert-validation - Possibly caused by mutual authentication. You can block or allow this but you can't decrypt
- session reason end - tcp-rst-from-client - Caused by certificate pinning. You can't decrypt this. Android 7+ does not trust manually installed root certificates.
- session reason end - decrypt-unsupport-param - The session used an unsupported protocol version, cipher, or SSH algorithm. This session end reason is also displayed when the session produced a fatal error alert of type unsupported_extension, unexpected_message, or handshake_failure.
Block DoH and DoT
To block DoH, you need to implement decryption and then block doh as an App-ID.
To block DoT, you need to block TCP-853 and then also block dot as an App-ID separately. The App-ID only blocks well known DoT servers.
Suggested Rules
- Destination URL Exceptions
- Source IP Exceptions
- Destination IP Exceptions
- User Exceptions
- Troubleshooting Exceptions
- Decrypt All
Ignore: financial-services, health and medicin, legal and government.
Consider ignoring: “low risk shopping” and “low risk streaming”. (PANOS 9.0+ as this requires customer URL category to stich “low risk” and “shopping/streaming” together.
ECDSA
Remember to also have ECDSA Trust and Untrust certificates. The firewall will use those as appropriate. However, using a ECDSA decryption certificate can cause more CPU usage on the firewall than a RSA decryption certificate.
Decryption Profile
* Consider appending certificate CN value to SAN extension. * Consider blocking RSA and DHE * Consider allowing GCM only and SHA256 and higher only. * Strip ALPN for HTTP Header Insertion and Clientless VPN.
Remember, if you want the firewall to actually check the status of certificates, you have to enable that at
Device→Setup→Session→Decryption Settings→Certificate Revocation Checking
Skype for Business will break if you tick block session with unsupported versions.
Get Firefox to Use System CA Store
In Firefox, go to about:config and create a new boolean value called “security.enterprise_roots.enabled” and set the value to true. This will tell Firefox to use the system certificate store.
This setting can be automated using Firefox's pref.js file (http://kb.mozillazine.org/Prefs.js_file) and/or via user.js file (http://kb.mozillazine.org/User.js_file). Particular settings can also be locked down via files mozilla.cfg and local-settings.js(http://kb.mozillazine.org/Locking_preferences).
GlobalProtect Issues
I found that, when decrypting everything, you should make sure that you import the root and intermediate certificates used to sign the certificates used by the GlobalProtect Portals and Gateways used by the GlobalProtect system you are connecting. You will also need to make the root certificate as a trusted root certificate.
Also remember, if the decryption profile being used has
Block sessions with untrusted issuers
ticked. This will block the GlobalProtect session if the appropriate certificates are not imported into the firewall.
Decryption Exceptions
The HTTPS URLs which we found to be very heavy on firewalls due to a very high number of short connections (handshake and then just a couple of packets and disconnect; and in a short while the process repeats again and again).
Decryption can cause high data plane use if it tries to decrypt a high number of connections that are short (i.e 'bursty' applications that establish the handshake and then disconnect the session after exchanging only a handful of packets). Two specific ones you might want to exclude from decryption are
watson.telemetry.microsoft.com watson.microsoft.com
For Cortex XDR Traffic:
*.traps.paloaltonetworks.com *.xdr.<region>.paloaltonetworks.com app-proxy.<region>.paloaltonetworks.com panw-xdr-evr-prod-<region>.storage.googleapis.com panw-xdr-installers-prod-us.storage.googleapis.com panw-xdr-payloads-prod-us.storage.googleapis.com global-content-profiles-policy.storage.googleapis.com lrc-<region>.paloaltonetworks.com
You should also exclude some Skype for Business domains based on this page.
*.online.lync.com *.infra.lync.com sipdir.online.lync.com sipfed1a.online.lync.com
Or you could just exclude
lync.com *.lync.com
WHen calling another SKype for Business user who has an on premise solution, you may need to add their specific DNS SIP address to the exclusion list. Google provide a list of domains to add to the exception list to get ChromeOS working.
accounts.google.com accounts.google.[country] accounts.gstatic.com accounts.youtube.com alt*.gstatic.com2 (PAN-OS can't add * after alt without a . so you should see the list of alt sites below) chromeos-ca.gstatic.com clients1.google.com clients2.google.com clients3.google.com clients4.google.com commondatastorage.googleapis.com cros-omahaproxy.appspot.com dl.google.com dl-ssl.google.com *gvt1.com gweb-gettingstartedguide.appspot.com m.google.com omahaproxy.appspot.com pack.google.com policies.google.com safebrowsing-cache.google.com safebrowsing.google.com ssl.gstatic.com storage.googleapis.com tools.google.com www.googleapis.com www.gstatic.com
This list of Alt sites is
alt1.gstatic.com alt2.gstatic.com alt3.gstatic.com alt4.gstatic.com alt5.gstatic.com alt6.gstatic.com alt7.gstatic.com alt8.gstatic.com alt9.gstatic.com alt10.gstatic.com alt11.gstatic.com alt12.gstatic.com alt13.gstatic.com alt14.gstatic.com alt15.gstatic.com alt16.gstatic.com alt17.gstatic.com alt18.gstatic.com alt19.gstatic.com alt20.gstatic.com alt21.gstatic.com alt22.gstatic.com alt23.gstatic.com alt24.gstatic.com alt25.gstatic.com
I have applied a 'decrypt all traffic' rule to my firewall. Other than the default exceptions built into PANOS 8.0+, these are the only exceptions I have had to make.
Exception required in order for Google Hangouts to work on Android.
www.googleapis.com
Exception required in order for Google Hangouts to download photos on Android and in Web Browser.
*.googleusercontent.com
Exception required in order for Google Maps to work on Android. It works fine in the web browser without this exception.
clients4.google.com www.google.com
The following exceptions were required for Google Drive in the web browser to work fully.
accounts.google.com
The following exceptions were required for Google Play Store to work on Android.
play.googleapis.com android.clients.google.com *.ggpht.com *.gvt1.com
To get Netflix app on Android working, I had to add the following decryption exceptions. It works fine in the web browser without this exception.
api-global.netflix.com android-appboot.netflix.com android.prod.cloud.netflix.com *.nflxso.net *.nflxvideo.net
I had to add the following to the SSL Decryption Exclusion in order to get the Sync.com client working. I think a decryption exception rule might also have worked.
*.sync.com
I had to add the following exceptions to get NowTV app working on Android.
images.metadata.sky.com *.imageservice.sky.com bskyb.demdex.net bskyb.sc.omtrdc.net p.sky.com ie.api.atom.nowtv.com init.sky.com
I had to add the following exceptions to get YouTube app working on Android. It works fine in the web browser without this exception.
youtubei.googleapis.com i.ytimg.com *.googlevideo.com
I had to add the following exceptions to get Facebook messenger app on Android to send and receive photos. The first allows you to send (upload) images. The second allows you to receive images.
rupload.facebook.com *.xx.fbcdn.net
I had to add the following exceptions to allow the Audible app on Android to download audio books.
cde-ta-g7g.amazon.com
I had to add the following exceptions to allow the BBC iPlayer app on Android to play TV shows (the Radio version seemed to work fine without exceptions).
r.bbci.co.uk open.live.bbc.co.uk ibl.api.bbci.co.uk vod-dash-uk-live.akamaized.net
I had to add the following exceptions to allow the Gumtree app on Android to work.
android-api-de.gumtree.com i.ebayimg.com
I had to add the following exceptions to allow the Channel4 app on Android to work.
api.channel4.com c4.eme.lp.aws.redbeemedia.com
I had to add the following exceptions to allow Microsoft Skype for Business to work.
*.infra.lync.com
I had to add the following exception to get GoToMeeting application on Windows 10 working.
egwglobal.gotomeeting.com
There are several sites will not work through decryption
www.troyhunt.com scotthelme.co.uk www.ssllabs.com ssllabs.com haveibeenpwned.com securityheaders.io (Broken in Chrome. Worked in Firefox.)
Supported Ciphers
The cipher suites supported for Forward Proxy (internal→web) and Inbound Inspection (web→internal)
PAN-OS 8.0
- TLS-RSA-WITH-3DES-EDE-CBC-SHA
- TLS-RSA-WITH-RC4-128-MD5
- TLS-RSA-WITH-RC4-128-SHA
- TLS-RSA-WITH-AES-256-CBC-SHA
- TLS-RSA-WITH-AES-128-CBC-SHA
- TLS-RSA-WITH-AES-128-CBC-SHA256
- TLS-RSA-WITH-AES-256-CBC-SHA256
- TLS-RSA-WITH-AES-128-GCM-SHA256
- TLS-RSA-WITH-AES-256-GCM-SHA384
- TLS-DHE-RSA-WITH-AES-128-CBC-SHA
- TLS-DHE-RSA-WITH-AES-256-CBC-SHA
- TLS-DHE-RSA-WITH-AES-128-GCM-SHA256
- TLS-DHE-RSA-WITH-AES-256-GCM-SHA256
- TLS-ECDHE-RSA-WITH-AES-128-CBC-SHA
- TLS-ECDHE-RSA-WITH-AES-256-CBC-SHA
- TLS-ECDHE-RSA-WITH-AES-128-GCM-SHA256
- TLS-ECDHE-RSA-WITH-AES-256-GCM-SHA256
- TLS-ECDHE-ECDSA-RSA-WITH-AES-128-GCM-SHA256
- TLS-ECDHE-ECDSA-RSA-WITH-AES-256-GCM-SHA256
PAN-OS 7.1
- TLS-RSA-WITH-3DES-EDE-CBC-SHA
- TLS-RSA-WITH-RC4-128-MD5
- TLS-RSA-WITH-RC4-128-SHA
- TLS-RSA-WITH-AES-256-CBC-SHA
- TLS-RSA-WITH-AES-128-CBC-SHA
- TLS-RSA-WITH-AES-128-CBC-SHA256
- TLS-RSA-WITH-AES-256-CBC-SHA256
- TLS-RSA-WITH-AES-128-GCM-SHA256
- TLS-RSA-WITH-AES-256-GCM-SHA384
- TLS-DHE-RSA-WITH-AES-128-CBC-SHA (forward proxy only)
- TLS-DHE-RSA-WITH-AES-256-CBC-SHA (forward proxy only)
- TLS-DHE-RSA-WITH-AES-128-GCM-SHA256 (forward proxy only)
- TLS-DHE-RSA-WITH-AES-256-GCM-SHA256 (forward proxy only)
- TLS-ECDHE-RSA-WITH-AES-128-CBC-SHA (forward proxy only)
- TLS-ECDHE-RSA-WITH-AES-256-CBC-SHA (forward proxy only)
- TLS-ECDHE-RSA-WITH-AES-128-GCM-SHA256 (forward proxy only)
- TLS-ECDHE-RSA-WITH-AES-256-GCM-SHA256 (forward proxy only)
PAN-OS 7.0
- TLS-RSA-WITH-3DES-EDE-CBC-SHA
- TLS-RSA-WITH-RC4-128-MD5
- TLS-RSA-WITH-RC4-128-SHA
- TLS-RSA-WITH-AES-256-CBC-SHA
- TLS-RSA-WITH-AES-128-CBC-SHA
- TLS-RSA-WITH-AES-128-CBC-SHA256
- TLS-RSA-WITH-AES-256-CBC-SHA256
- TLS-RSA-WITH-AES-128-GCM-SHA256
- TLS-RSA-WITH-AES-256-GCM-SHA384
PAN-OS 6.0
- TLS-RSA-WITH-AES-256-CBC-SHA
- TLS-RSA-WITH-AES-128-CBC-SHA
- TLS-RSA-WITH-3DES-EDE-CBC-SHA
- TLS-RSA-WITH-RC4-128-MD5
- TLS-RSA-WITH-RC4-128-SHA
- TLS-RSA-WITH-AES-128-CBC-SHA256
- TLS-RSA-WITH-AES-256-CBC-SHA256
Fix Unsupported Certificates
If you administer a server that does not support the ciphers above, you may want to edit it to allow a supported cipher.
In my case, running Apache, I edited the file /etc/httpd/conf.d/ssl.conf and modified the following line that controls the ciphers used. I had only two ciphers supported. Both Eliptic Curve Diffe Hellman (EECDH).
SSLCipherSuite AES256+EECDH:AES256+EDH
I had to add a RSA cipher by adding one of the following to the end of the line (prefixed with a colon).
- AES256+RSA
- AES128+RSA
- RC4+RSA (RC4 is now considered weak. Using it is not advised)
- 3DES+RSA (3DES is now considered weak. Using it is not advised)
Following the edit, my config line looked like this
SSLCipherSuite AES256+EECDH:AES256+EDH:AES256+RSA
All I had to do was restart the Apache web server and the Palo started decrypting data correctly.
Testing Certificate Protocols
To see what SSL/TLS protocls are in use on a server, use this command
nmap --script ssl-enum-ciphers -p 443 1.1.1.1
Inbound Inspection Limitation
Prior to PAN-OS 8.0, Palo could not do inbound inspection on DHE or ECDHE.
If you place the Palo between the internet and (say) F5 load balancers that terminated the SSL connections, you may haveto add the following to the cipher configuration of the appliances. <pre>:!DHE+AES-GCM:!DHE+AES:!DHE+3DES:!ECDHE+AES-GCM:!ECDHE+AES:!ECDHE+3DES</pre>
Chrome CN Field
Support for the CN field in certificates was removed from Chrome in version 58.
Many people don’t know that the “Common Name” field of an SSL certificate, which contains the domain name the certificate is valid for, was actually phased-out via RFC nearly two decades ago. Instead, the SAN (Subject Alternative Name) field is the proper place to list the domain(s).
However, this has been ignored and for many years the Common Name field was exclusively used. Chrome is finally fed up with the field that refuses to die. In Chrome 58, the Common Name field is now ignored entirely.
This means certificates that were exclusively using that field to indicate the valid domain name are no longer supported. Publicly-trusted SSL certificates have been supporting both fields for years, ensuring maximum compatibility with all software – so you have nothing to worry about if your certificate came from a trusted CA.
This change will only affect private PKIs and other software that have not been following spec. If you notice any sites returning the error NET::ERR_CERT_COMMON_NAME_INVALID it's likely due to the certificate not using SANs properly. Users of products like Sophos' HTTPS interception are now finding out that their software is not RFC-compliant. Eric Lawrence wrote more about this topic.
Windows Server 2016 Forward Proxy Certificate
You may find issues when using Windows Server 2016 to generate a Root CA for decryption.
https://www.reddit.com/r/paloaltonetworks/comments/88dvzy/ssl_decrypt_using_windows_server_2016_pki/
https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClotCAC
The answer given by one Reddit user was I've had to upgrade the PKI infrastructure in an AD environment a few times. There's a lot of LOB and other apps that require certs signed with SHA1 (or better)
Simplest method is to run these two commands on your CA, in an elevated command prompt.
certutil -setreg csp\alternatesignaturealgorithm 0
certutil -setreg csp\cnghashalgorithm sha1
Then reboot, or just restart the Certificate Service (certsvc).
You will need to reissue your root certificate after that. If you can line this operation up with a renewal of your CA/root cert anyway, you can kill two birds with one stone. The process of renewing the root cert is fairly easy and well documented.
You do not need to re-issue existing certificates. They will continue to work. But once the algorithm is changed you can issue a new Sub-CA cert and import that into the firewall
This operation is also supported by MS. The Live article is out of date in that regards. For example, Lync Server 2013+ doesn't supported certificates using RSASSA-PSS either, so MS recommends you make the changes suggested above.
Generate Certificates on CLI
To generate certificates or Certificate Signing Requests on the CLI, use the
request certificate generate
command.
The options are as follows
country-code- <text> Should be a two letter country code (e.g. GB, DE, etc)state- <text>locality- <text>organization- <text>organization-unit- <text>
email- <text>filename- <text> I don't know what this is used for. I have yet to see how this affects the certificates.
ocsp-responder-url- [<name of OCSP object in Palo firewall>]ca- [yes|no]signed-by- [external|<name of certificate in Palo firewall>] (Leave this blank if you have said yes to 'ca' and this doesn't need to be a subordinate CA (i.e. you are creating a self-signed root cert).hostname- This should take the form[hostname1 hostname2 etc]]
ip- This should take the form[ ip1 ip2 etc]]
alt-email- This should take the form[ email1 email2 etc ]]
days-till-expiry- <number> (default is 365)digest- This should be one of the following [md5|sha1|sha256|sha384] (default is SHA256)algorithm[ECDSA ecdsa-nbits {256|384}|RSA rsa-nbits {512|1024|2048|3072|4096}] (suggested defaults are RSA-2048 or ECDSA-256)name- <text> (This is the Common Name CN of the certificate)certificate-name<text> (This is the name that will appear in the GUI. This can be changed. It must be at least 6 characters long if you use the GUI but the CLI allows for fewer characters).
Only algorighm, name and certificate-name are actually needed
So, to create a self-signed CA certificate, run the following
request certificate generate country-code GB state London locality Westminster organization ACME organization-unit IT ca yes days-till-expiry 40 ip [ 192.168.1.1 ] hostname [ hostname.example.com hostname privatehostname 192.168.1.1 ] algorithm RSA rsa-nbits 2048 name hostname.example.com certificate-name CERT_MYCERT
To create a 'Certificate Signing Request (CSR), run the following
request certificate generate signed-by external country-code GB state London locality Westminster organization ACME organization-unit IT ca yes days-till-expiry 40 ip [ 192.168.1.1 ] hostname [ hostname.example.com hostname privatehostname 192.168.1.1 ] algorithm RSA rsa-nbits 2048 name hostname.example.com certificate-name CERT_MYCERT
SSL Decryption Errors
In PAN-OS 10.0+ you can use the following commands to help troubleshoot based on decryption logs.
Version error bitmask values identify mismatches between the TLS protocol versions that the client and server use and also identify TLS protocol mismatches between the client and the Decryption profile applied to the traffic. The CLI command to convert version error bitmasks is the following. The command returns the TLS version that matches the bitmask.
debug dataplane show ssl-decrypt bitmask-version <bitmask-value>
Cipher error bitmask values identify encryption and other mismatches between the client and the Decryption profile applied to the traffic. The command returns the cipher that matches the bitmask.
debug dataplane show ssl-decrypt bitmask-cipher <bitmask-value>
Office365 Certificates
Microsoft list their root and intermediate certificates here.
