Remember, if you manually whitelist a site, any specific sub-pages that are normally classed as malware will be allowed through (even though the list of categories will mark it as cust-list, malware).

PAN-OS 9.0 introduced multi-category URL Filtering.

Best practice for URL risk categories is (source)

• High-risk - alert
• Medium-risk - alert
• Low-risk - allow
• Newly-registered domains - block

When it comes to multiple categories, PAN-OS lists all categories in the “URL Category List” but only one category in the “Category” column.

How does PAN-OS determine which of the selected categories goes by into the category column? Alphabetical order!

Thus, everything from abortion to legal gets put as low-risk in the category column and everything from military onwards is classified as the actual category. (assuming the sites are low-risk)

I don’t think any site is classified with just a risk-category.

Also, it seems that malware, phishing, command-and-control and grayware sites do not have a risk category. I’m assuming this is because they are considered obviously risky.

Remember, if you set low-risk to allow, then all sites will have their actual category in the category column and low-risk will still be visible in the “URL Category List” Column.

On the other hand, if you set all categories to allow and set low-risk to alert then the “Category” column will show ‘low-risk’ for everything and malware, phishing, command-and-control and grayware will not appear as they will not have low-risk associated with them and so will be allow and not log.

Other notes:

• Unknown domains are classified as high-risk until PAN-DB completes site analysis and categorization.
• Unknown IP addresses are categorized as medium-risk until PAN-DB completes site analysis and categorization.
• Online-storage-and-backup always as the risk category set to medium-risk (or higher).
• (I think) dynamic-dns is always high-risk. I could be wrong on that though.

Here is a really nice KB article on multi-category URL Filtering.

Here is a simple Linux script to test all categories. I think (I've not tried it) that on Windows you need to replace /dev/null with NUL (CMD) or $null (PowerShell)

curl http://urlfiltering.paloaltonetworks.com/test-grayware > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-malware > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-phishing > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-command-and-control > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-abortion > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-abused-drugs > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-adult > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-alcohol-and-tobacco > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-auctions > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-business-and-economy > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-computer-and-internet-info > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-content-delivery-networks > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-copyright-infringement > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-cryptocurrency > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-dating > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-dynamic-dns > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-educational-institutions > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-entertainment-and-arts > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-extremism > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-financial-services > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-gambling > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-games > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-government > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-hacking > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-health-and-medicine > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-home-and-garden > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-hunting-and-fishing > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-insufficient-content > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-internet-communications-and-telephony > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-internet-portals > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-job-search > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-legal > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-military > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-motor-vehicles > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-music > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-news > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-nudity > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-online-storage-and-backup > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-parked > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-peer-to-peer > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-personal-sites-and-blogs > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-philosophy-and-political-advocacy > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-private-ip-addresses > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-proxy-avoidance-and-anonymizers > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-questionable > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-real-estate > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-recreation-and-hobbies > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-reference-and-research > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-religion > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-search-engines > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-sex-education > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-shareware-and-freeware > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-shopping > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-social-networking > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-society > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-sports > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-stock-advice-and-tools > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-streaming-media > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-swimsuits-and-intimate-apparel > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-training-and-tools > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-translation > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-travel > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-unknown > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-weapons > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-web-advertisements > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-web-hosting > /dev/null
curl http://urlfiltering.paloaltonetworks.com/test-web-based-email > /dev/null

Based on this article.

• Create a URL Filtering profile that blocks the unwanted HTTP and HTTPS websites. Create a Security Policy with an action of “allow” and then link the URL Filtering profile to it.
• Response pages must be enabled.
• A certificate to be used for Forward Trust on the Palo Alto Networks device.
• A certificate to be used for Forward Untrust on the Palo Alto Networks device.
• Check the BrighCloud information if using BrightCloud.

Once the above requirements have been met, enable the Palo Alto Networks device's ability to inject URL filtering response pages within an HTTPS session with the following configuration command. This command works with either BrightCloud or PAN-DB URL filter: (this setting will automatically by synced in HA).

set deviceconfig setting ssl-decrypt url-proxy yes

Note that this only works for URL filtering. If you block an application (e.g. Twitter) without decryption, you will just get a native browser error (e.g. Security Connection Failed).

You can check a configuration to see if this is set by searching for

<url-proxy>yes</url-proxy>

When a continue page is displayed, a log with action block-continue will be created. If the user clicks continue, a log with action continue< will be created.

When a override page is displayed, a log with action block-override will be created. If the user clicks override, a log with action override will be created.

*.paloaltonetworks.com matches www.paloaltonetworks.com and www.paloaltonetworks.com.uk.

*.paloaltonetworks.com/ matches www.paloaltonetworks.com but not www.paloaltonetworks.com.uk.

In PANOS 9.0+, the URL category 'medium-risk' exists. This is for sites that have previously been associated with Malware but nothing has been seen for up to 60 days. The exception is online-storage-and-backup. This will always be medium-risk.

Requires PAN-OS 9.0.4 or later.

Enable the firewall to hold an initial web request as it looks up a website’s URL category with PAN-DB.

When a user visits a website, a firewall with URL Filtering enabled checks its local cache of URL categories to categorize the site. If the firewall doesn’t find the URL’s category in the cache, it performs a lookup in PAN-DB. By default, the firewall allows the user’s web request during this cloud lookup and enforces policy when the server responds.

But when you choose to hold web requests, the firewall blocks the request until it either finds the URL category or times out. If the lookup times out, the firewall considers the URL category not-resolved.

configure

set deviceconfig setting ctd hold-client-request yes

commit

Sometimes you want to block a domain but allow specific pages. Let's use YouTube as an example. If you have a custom profile 'whitelist' and a custom profile 'blacklist' and they both contain '*.youtube.com', you will find that block takes prescendence over allow/alert. If you put 'www.youtube.com/watch?v=4lm75v4Ndlg' into the whitelist, you will find the block list still take prescendence over it.

To allow the traffic, you need to create a rule that uses 'whitelist' in the match criteria of the rule and then just 'alerts' (or, at least, doesn't block the blacklist).

See this link.

On PANOS 8.1+ you can use HTTP Header Insertion to enforce YouTube safe search (specifically, enforce YouTube restricted mode). This is set in the URL Profile. However, because safe search enforcement is done before HTTP Header Insertion, YouTube safe search enforcement doesn't work well with the main “Enforce Safe Search” feature as what will happen is that restricted mode will appear to be on but the safe search will think it is off and the end user will have no way of “switching it on” as the HTTP Header Insertion does that on the fly. The solution is to create a dedicated rule for the YouTube application ID (or possibly YouTube URLs) and use HTTP Header Insertion without enforcing safe search.

You will need to ensure that you are decrypting the sessions.

You will need to block access to the 'search-engine' category as we can only secure Google, Bing, Yahoo and Yandex. However, I only allow Google.

Add the following to the list of URL exceptions (I'm pretty sure some of them are also required for Google Maps but I'm not sure which ones)

• google.com
• google.co.uk
• www.google.com
• www.google.co.uk
• www.google.co.uk/*
• accounts.google.com
• accounts.google.co.uk
• notifications.google.com
• history.google.com
• myactivity.google.com
• contacts.google.com
• productforums.google.com
• support.google.com
• ogs.google.com
• feedproxy.google.com
• supl.google.com
• kh.google.com
• pki.google.com
• developers.google.com (Used for help documents)
• apis.google.com (needed for some features - e.g Google Keep's menu)
• www.googleapis.com
• maps.googleapis.com
• clients1.google.com
• clients2.google.com
• clients3.google.com
• clients4.google.com
• clients5.google.com
• clients6.google.com
• 0.client-channel.google.com
• 1.client-channel.google.com
• 2.client-channel.google.com
• people-pa.clients6.google.com/v2/people/
• people-pa.clients6.google.com/v2/people/lookup
• *.client-channel.google.com (needed for Google Hangouts)
• sb-ssl.google.com (safe browsing API Lookup)
• dl.google.com (serves binary downloads for Chrome, Earth, the Android SDK, and thousands of other files)
• cloudconsole-pa.clients6.google.com (needed for loading console.cloud.google.com/apis/library?q=youtube)
• servicemanagement.clients6.google.com (needed to activate YouTube API v3 console.cloud.google.com/apis/library?q=youtube)
• cloudusersettings-pa.clients6.google.com (needed to manage YouTube API v3)
• apikeys.clients6.google.com (needed to manage YouTube API v3)
• iam.clients6.google.com (needed to manage YouTube API v3)
• monitoring.clients6.google.com (needed to manage YouTube API v3)
• clientauthconfig.clients6.google.com (needed to get YouTube API v3 Credential Key)
• consent.google.com
• policies.google.com
• keep.google.com

The following is from these two pages on Palo's documentation.

<html>
  <head>
    <title>Search Blocked</title>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
    <meta http-equiv="pragma" content="no-cache">
    <meta name="viewport" content="initial-scale=1.0">

    <style>
      #content {
      border:3px solid#aaa;
      background-color:#fff;
      margin:1.5em;
      padding:1.5em;
      font-family:Tahoma,Helvetica,Arial,sans-serif;
      font-size:1em;
      }

      h1 {
      font-size:1.3em;
      font-weight:bold;
      color:#196390;
      }

      b {
      font-weight:normal;
      color:#196390;
      }

    </style>
  </head>
  <body bgcolor="#e7e8e9">
    <div id="content">
      <h1>Search Blocked</h1>
      <p>
        <b>User:</b><user/>
      </p>
      <p>Your search results have been blocked because your search settings are not in accordance with company policy.  In order to continue, please update your search settings so that Safe Search is set to the strictest setting. If you are currently logged into your account, please also lock Safe Search and try your search again.
	  </p>
      <p>
        For more information, please refer to: 
        <a href="<ssurl/>">
          <ssurl/>
        </a>
      </p>
      <p id="java_off"> Please enable JavaScript in your browser.<br></p>
      <p><b>Please contact your system administrator if you believe this message is in error.</b></p>
    </div>
  </body>

  <script>
    // Grab the URL that's in the browser.
    var s_u = location.href;

    // Bing
    // Matches the forward slashes in the beginning, anything, then ".bing." then anything followed by a non greedy slash. Hopefully the first forward slash.
    var b_a = /^.*\/\/(.+\.bing\..+?)\//.exec(s_u);
    if (b_a) {
        s_u = s_u + "&adlt=strict";
        window.location.replace(s_u);
        document.getElementById("java_off").innerHTML = 'You are being redirected to a safer search!';
    }

    // Google
    // Matches the forward slashes in the beginning, anything, then ".google." then anything followed by a non greedy slash. Hopefully the first forward slash.
    var g_a = /^.*\/\/(.+\.google\..+?)\//.exec(s_u);
    if (g_a) {
        s_u = s_u.replace(/&safe=off/ig,"");
        s_u = s_u + "&safe=active";
        window.location.replace(s_u);
        document.getElementById("java_off").innerHTML = 'You are being redirected to a safer search!';
    }
	
    // Yahoo
    // Matches the forward slashes in the beginning, anything, then ".yahoo."" then anything followed by a non greedy slash. Hopefully the first forward slash.
    var y_a = /^.*\/\/(.+\.yahoo\..+?)\//.exec(s_u);
    if (y_a) {
        s_u = s_u.replace(/&vm=p/ig,"");
        s_u = s_u + "&vm=r";
        window.location.replace(s_u);
        document.getElementById("java_off").innerHTML = 'You are being redirected to a safer search!';
    }
	
    document.getElementById("java_off").innerHTML = ' ';
  </script>
</html>

To make the Chrome Omni bar work correctly (without requiring a redirect), I had to go to Chrome settings and add the following as a search provider and then make that provider the default

https://www.google.co.uk/search?q=%s&safe=active

List of URL Categories

'Clean' List of URL Categories

Test a Site

*.360safe.com
  *.360.cn

Another test page.