Differences

This shows you the differences between two versions of the page.

--- infoblox:tide [2023/11/16 12:49] – [Your Uploaded Data] bstafford
+++ infoblox:tide [2024/12/27 15:12] (current) – removed bstafford
@@ Line 1: / Line 1: @@
-====== TIDE ======
-TIDE is Threat Intelligence Data Exchange. It is effectively a very large Threat Database that is managed by the Infoblox Threat Intelligence Team. From its database, the Infoblox RPZ feeds are generated.
-===== Active Indicators =====
-In the CSP > Research > Active Indicators, we can't see RPZ membership of threats. This is expected as this page is a simple GUI interface to the Threat Intel database. RPZ membership can be found in Dossier lookups or (ideally) via API lookups.
-===== Test RPZ Data =====
-Infoblox has harmless test domains that are in various RPZ feeds. This allows you to test that a given RPZ feed is active and working as well as generate example logs, etc.
-Test RPZ data [[infoblox:test_domains|here]].
-===== Get Custom RPZ Data=====
-Get all custom RPZ feeds and Country Based RPZ feeds
-<code>curl -s -X GET "https://csp.infoblox.com/api/custom_feeds/v1/feeds" -H "Authorization: Token token=APIKEY_HERE" -H "Content-Type: application/json"</code>
-You will get a lot of data back. Each feed has an ID. Use that to limit further queries to pull just that feed.
-<code>curl -s -X GET "https://csp.infoblox.com/api/custom_feeds/v1/feeds/<ID>" -H "Authorization: Token token=APIKEY_HERE" -H "Content-Type: application/json"</code>
-===== What is In the Custom RPZ Feed Options =====
-  * IID
-  * AISCOMM
-  * EmergingThreats
-  * FarsightSecurity
-  * SURBL (no longer available?)
-  * 0011A00002BcDwEFG
-Infoblox Curated Data
-  * IID:ANALYST
-  * IID:CRIME
-  * IID:DTQ_IP
-  * IID:IID
-  * IID:IID Internal
-  * IID:IID_BH_DECLARED
-  * IID:IID_IRD
-  * IID:IID_TTL
-  * IID:LookalikeDomains
-  * IID:OTHER
-  * IID:PARTNER
-  * IID:PORTAL_AK
-  * IID:POWERSHARK
-  * IID:RateShark
-  * IID:IB_NOC
-  * IID:IB_PM
-  * IID:IB_DS
-  * IID:IB_SE
-Infoblox 3rd Party Data
-  * AISCOMM:AIS-COMMERCIAL (Department of Homeland Security's (DHS) Automated Indicator Sharing (AIS))
-  * EmergingThreats:Hostnames_Feed (ProofPoint)
-  * EmergingThreats:IP_Feed (ProofPoint)
-  * EmergingThreats:URL_Feed (ProofPoint)
-  * FarsightSecurity:NOD (Farsight Security - now part of DomainTools)
-  * SURBL:FRESH
-  * SURBL:SURBL_multi_list
-SUBRL is SPAM URI (Uniform Resource Identifier) Real-time Block List.
-=====Your Uploaded Data=====
-  * 0011A00001AaA1aAAA:name-of-custom-data-profile
-You can upload data to TIDE via the BloxOne portal (Manage > TIDE Data > Data Upload )
-Documentation [[https://docs.infoblox.com/space/BloxOneThreatDefense/35434535|here]].
-Example XML file here:
-<code><feed>
- <profile>SampleProfile</profile>
- <record_type>host</record_type>
- <record>
-   <host>www.testtesttest0.com</host>
-   <property>Phishing_Phish</property>
-   <detected>20230101T154742Z</detected>
- </record>
- <record>
-   <host>www.testtesttest1.com</host>
-   <property>Phishing_Phish</property>
-   <detected>20230101T154742Z</detected>
- </record>
-</feed></code>
-JSON FILE
-Notice that the second domain doesn't have a duration.
-<code>{
- "feed": {
- "record_type": "host",
- "record": [
- {"host": "www.baddomain.com", "property": "Scanner_Generic", "detected": "19980927T154242Z", "duration": "1y0m0w0d42h"},
- {"host": "www.example.com", "property": "Phishing_Phish", "detected": "20170602T154742Z"}  ]
- }
-}</code>
-Upload with CURL
-<code> curl -H "AUTHORIZATION:TOKEN APIKEY" -H 'Content-Type: application/json;' -X POST "https://csp.infoblox.com/tide/api/data/batches?profile=SampleDataProfile" --data-binary '@/home/name/DATA_FILE_NAME.json'</code>
-The recommended limit for the number of records in a given data submission is 50,000. The maximum number of records should be no more than 60,000 at this point in time.
-Documentation [[https://docs.infoblox.com/space/BloxOneThreatDefense/35434535/TIDE+Data+Submission+Overview|here]], [[https://docs.infoblox.com/space/BloxOneThreatDefense/236028376/Submitting+Threat+Indicators|here]], and [[https://docs.infoblox.com/space/BloxOneThreatDefense/236159529/Data+Profiles|here]].
-===== Palo Alto Networks Integration =====
-<code>curl -X GET "https://csp.infoblox.com/tide/api/data/threats?type=host&threat_level_from=80&class=suspicious&field=host,threat_level&period=24h"  -H "Authorization: Token token=PUT_KEY_HERE" > tide-output</code>
-<code>cat tide-output | sed s/},{\"host\":\"/\\t/g | sed s/\",\"threat_level\":/\\t/g | sed s/\\t80//g | sed s/\\t100//g | sed s/\\t/\\r\\n/g | sed s/\{\"threat\"\:\\[\{\"host\":\"//g | sed s/\}\],\"record_count\"//g | sed s/\:.*$//g > palo-list</code>
-<code>sudo cp ./palo-list /var/www/html/</code>
-Make sure NGINX is installed and serving up /var/www/html. Edit as needed. Add certificates as needed.#
-On PAN-OS Web UI
-  * Objects > External Dynamic Lists > Add :
-  * Type = Domain List (So we can use it in Anti-Spyware Profiles)
-  * Tick "Automatically expand to include sub-domains"
-  * Set "Check for updates" as required
-  * Click "Test Source URL"
-===== Palo Alto Networks Integration Direct =====
-You can update the External Dynamic List in Palo Alto Networks to use the following URL and download directly from CSP
-<code>https://apikey:API-KEY-HERE@csp.infoblox.com/tide/api/data/threats?data_format=csv&field=domain&rlimit=10&type=host&class=cryptocurrency</code>
-Note the use of the username ''apikey'' and also an API key.
-If you set a Server Certificate in the EDL you can also user username/password (e.g. ''apiuser'' and APIKEY) in addition to <code>csp.infoblox.com/tide/api/data/threats?data_format=csv&field=domain&rlimit=10&type=host&class=cryptocurrency</code>
-I got this working for domains but not IP feeds. URL feeds don't work directly from Infoblox CSP because the Palo Alto can't handle the inclusion of <code>https://</code> at the start when setting a URL feed.
-===== API Examples =====
-==== Get Latest Threats From TIDE ====
-<code>PROPERTY=Suspicious_Generic
-LIMIT=100
-MIN_THREAT_LEVEL=80
-SOURCE=iid
-AGE=7d
-APIKEY=APIKEY
-curl -X GET -H "Authorization: Token token=$APIKEY" "https://csp.infoblox.com/tide/api/data/threats/state?type=host&profile=$SOURCE&property=$PROPERTY&threat_level_from=$MIN_THREAT_LEVEL&period=$AGE&rlimit=$LIMIT&field=host,detected" | sed s/,/\\n/g | grep host | awk -F "\"" '{print $4}' > suspicious-domains-last-week.txt</code>
-==== Get Bad Nameservers ====
-Get all bad nameservers from the last year.
-<code>PROPERTY=MaliciousNameserver_Generic
-OUTPUT="threat-domain-$PROPERTY.txt"
-LIMIT=100
-MIN_THREAT_LEVEL=80
-SOURCE=iid
-AGE=1y
-APIKEY=SET_APIKEY
-curl -X GET -H "Authorization: Token token=$APIKEY" "https://csp.infoblox.com/tide/api/data/threats/state?type=host&profile=$SOURCE&property=$PROPERTY&threat_level_from=$MIN_THREAT_LEVEL&period=$AGE&rlimit=$LIMIT&field=host,detected" | sed s/,/\\n/g | grep host | awk -F "\"" '{print $4}' > $OUTPUT</code>
-==== Get Newly Observed Domains====
-<code>curl -s -X GET -H "Authorization: Token token=$API_KEY" "https://csp.infoblox.com/tide/api/data/threats/host/hourly?property=Policy_NewlyObservedDomains&data_format=csv&field=host" > new-domains.txt</code>
-==== Recent Data ====
-<code>{{TIDE_url}}/data/threats/host/recent?fields=detected,property,host&data_format=csv</code>
-<code>{{TIDE_url}}/data/threats/host/daily?fields=detected,property,host&data_format=csv</code>
-<code>/tide/api/data/threats?type=host&class=suspicious&field=host,threat_level,detected,notes&period=24h&rlimit=50000</code>
-==== Get Top 10 Suspicious Emergent Domains ====
-Using GREP, show top 10 suspicious emergent domains from Infoblox that have been added in the last 48 hours. This list can be fed to External Dynamic List for third party security vendors.
-<code>curl -X GET -H "Authorization: Token token=APIKEY" "https://csp.infoblox.com/tide/api/data/threats/state?type=host&profile=iid&property=Suspicious_EmergentDomain&threat_level_from=80&period=48h&rlimit=10&field=host,detected" | sed s/,/\\n/g | grep host | awk -F "\"" '{print $4}' > emergent-domains-last-48h.txt</code>
-===== Summary of Count of Each Threat Type =====
-<code>https://csp.infoblox.com/tide/api/v1/iid_atp/dossier/threat_counts</code>
-==== Get All Critical MalwareC2DGA Threats ====
-Show all critical MalwareC2DGA threats detected in the last 48 hours. Limit results to 10 items.
-<code>curl -X GET -H "Authorization: Token token=APIKEY" "https://csp.infoblox.com/tide/api/data/threats/state?type=host&profile=iid&class=MalwareC2DGA&threat_level_from=100&rlimit=10&period=48h&field=host,property,threat_level,detected" > critical-malware-threats-last-2days.txt</code>
-Main bit
-<code>curl -X GET -H "Authorization: Token token=APIKEY" "https://csp.infoblox.com/tide/api/data/threats/state?type=..." > critical-malware-threats-last-2days.txt</code>
-... bit - in this case we change class to property = MalwareC2DGA_Locky
-<code>host&profile=iid&property=MalwareC2DGA_Locky&threat_level_from=100&rlimit=10&period=48h&field=host,property,threat_level,detected</code>
-  * Type = Host (type=host)
-  * Profile = IID (profile=iid)
-  * Threat Level >= 100 (threat_level_from=100)
-  * Max Results = 10 (rlimit=10)
-  * Period = 24h (Results added to TIDE in last 24 Hours)
-  * Field = detected,host,property (Data to include in result)
-  * Property = MalwareC2DGA_Locky (Property to detect)
-  * Class = MalwareC2DGA (Class to detect)
-Format is the following. Replace ''type=...'' with examples below<code>curl -X GET 'https://csp.infoblox.com/tide/api/data/threats?type=...'  -H 'Authorization: Token token=<your token>'</code>
-==== Get Notes ====
-Retrieve notes in TIDE for a domain example.com:
-<code>type=host&profile=IID&host=example.com</code>
-Retrieve new suspicious indicators:
-<code>type=host&profile=IID&class=suspicious&rlimit=2</code>
-==== Get Notes ====
-(Possibly) "Absolutely safe to block" and almost certainly not available via any other source or vendor today as verified. However, this is only true of the results where the threat_level >= 50.
-<code>type=host&class=suspicious&field=host,detected,threat_level,notes&period=24h</code>
-<code>/tide/api/services/intel/lookup/indicator/host?value=example.com&wait=true&data_format=json</code>
-====== Dossier ======
-Dossier [[https://docs.infoblox.com/space/BloxOneThreatDefense/271975091/Dossier+Source+Descriptions|sources]].
-  * iSIGHT is available as a separate subscription and is not automatically included with Dossier.
-===== RPZ =====
-Remember, if you only have Business Cloud and a threat exists in one of the "Advanced only" RPZ feeds, you won't see those RPZ feeds be identified if you send a Dossier query for the domain.
-In the event that you query Dossier for a domain that is in both AntiMalware and (e.g.) Suspicious-Lookalikes, then, as a Business Cloud customer, you will only see AntiMalware in the "Matched RPZ" list. You will, however, see the relevant "suspicious" tag in the activity timeline which gives you a clue it might also be in suspicious. Only Advanced customers will see (in Dossier) that the domain is in both AntiMalware and Suspicious Lookalikes RPZ feeds.
-===== API =====
-Simply query for
-  * type = HOST
-  * profile = IID Data
-  * class= MalwareC2
-  * property = MalwareC2_BackdoorRAT
-  * threatlevel = 80+
-  * period = (last) 24h
-==== Get MalwareC2 Threats from Last 24 Hours ====
-<code>curl -X GET -H "Authorization: Token token=APIKEY" "https://csp.infoblox.com/tide/api/data/threats?type=host&profile=iid&class=malwarec2&property=malwarec2_backdoorrat&threat_level_from=100&period=24h"</code>
-==== Check Which RPZ Feeds a Domain is In ====
-<code>curl -s -X POST "https://csp.infoblox.com/tide/api/services/intel/lookup/jobs?wait=true" -H "Authorization: Token token=APIKEY" -H "Content-Type: application/json" -d '{"target": {"one": {"type": "host", "target": "google.com","sources":["rpz_feeds"]}}}'</code>
-Same command but use GREP to filter out everything but the feed names.
-<code>curl -s -X POST "https://csp.infoblox.com/tide/api/services/intel/lookup/jobs?wait=true" -H "Authorization: Token token=APIKEY" -H "Content-Type: application/json" -d '{"target": {"one": {"type": "host", "target": "google.com","sources":["rpz_feeds"]}}}' | grep feed_name | grep -v ib- | awk -F ":" '{print $2}' | sed 's/\ //g' | sed 's/,//g' | sed 's/"//g'</code>
-Sam command but with BASH
-<code>#!/bin/bash
-DOMAIN=$1
-APIKEY="PUT_KEY_HERE"
-COMMAND="{\"target\": {\"one\": {\"type\": \"host\", \"target\": \"$1\",\"sources\":[\"rpz_feeds\"]}}}"
-echo "List of RPZ feeds that contain $1"
-curl -s -X POST "https://csp.infoblox.com/tide/api/services/intel/lookup/jobs?wait=true" -H "Authorization: Token token=$APIKEY" -H "Content-Type: application/json" -d "$COMMAND" | grep feed_name | grep -v ib- | awk -F ":" '{print $2}' |  sed 's/,//g' | sed 's/"//g'</code>
-==== Get Infoblox Web Category for Domain ====
-<code>#!/bin/bash
-DOMAIN=$1
-APIKEY="PUT_KEY_HERE"
-COMMAND="{\"target\": {\"one\": {\"type\": \"host\", \"target\": \"$DOMAIN\",\"sources\":[\"infoblox_web_cat\"]}}}"
-echo "Infoblox Web Category for $DOMAIN: "
-curl -s -X POST "https://csp.infoblox.com/tide/api/services/intel/lookup/jobs?wait=true" -H "Authorization: Token token=$APIKEY" -H "Content-Type: application/json" -d "$COMMAND" | grep name | awk -F ":" '{print $2}'</code>
-==== List of Dossier Fields for Domain ====
-<code>"acs","activity","atp","ccb","custom_lists","dns","gcs","geo","gsb","infoblox_web_cat","inforank","isi
-ght","malware_analysis","malware_analysis_v3","pdns","ptr","rlabs","rpz_feeds","rwhois","whitelist","whois","s
-sl_cert","urlhaus","nameserver","threatfox"</code>
-==== Values for ACS ====
-<code>batch_id
-class
-confidence
-confidence_score
-confidence_score_rating
-confidence_score_vector
-detected
-dga
-domain
-expiration
-extended  (can contain {"notes":"notes put here"})
-full_profile
-host
-id
-imported
-profile
-property
-received
-risk_score
-risk_score_rating
-risk_score_vector
-threat_level
-threat_score
-threat_score_rating
-threat_score_vector
-tld
-type
-up</code>
-===== Script To Check RPZ Feeds (Python) =====
-<code>#!/usr/bin/env python3
-# vim: tabstop=8 expandtab shiftwidth=4 softtabstop=4
-import requests
-import json
-# Needed to read CLI parameters
-import sys
-# Needed to check the valid domain name regex
-import re
-# Needed to manipulate the data
-from datetime import datetime
-APIKEY = "API_KEY"
-def is_valid_domain_name(domain_name):
-    # Regular expression pattern to match a valid domain name
-    pattern = r"^(?!:\/\/)(?!www\.)(?:[a-zA-Z0-9](?:[a-zA-Z0-9\-]{0,61}[a-zA-Z0-9])?\.)+(?:[a-zA-Z]{2,})$"
-    # Check if the domain name matches the pattern
-    if re.match(pattern, domain_name):
-        return True
-    else:
-        return False
-def is_date_in_future(date_str):
-    current_date = datetime.now().date()
-    date = datetime.strptime(date_str, '%Y-%m-%d').date()
-    if date > current_date:
-        return True
-    else:
-        return False
-if len(sys.argv) < 1:
-    print("Usage: python script_name.py domain-to-test.corp.")
-    sys.exit(1)  # Exit the script with a non-zero status code indicating an error
-parameter1 = sys.argv[1]  # The first parameter
-if not is_valid_domain_name(parameter1):
-    print("Error: Parameter ",parameter1," is not a valid domain name.")
-    sys.exit(1)
-DATA = "{\"target\": {\"one\": {\"type\": \"host\", \"target\": \"" + parameter1 + "\",\"sources\":[\"rpz_feeds\"]}}}"
-# Make the cURL request
-url = "https://csp.infoblox.com/tide/api/services/intel/lookup/jobs?wait=true"
-headers = {
-    "Content-Type": "application/json",
-    "Authorization": "Token " + APIKEY
-}
-response = requests.post(url, data=DATA, headers=headers)
-# Check if the request was successful (status code 200)
-if response.status_code == 200:
-    data = response.json()
-    results = data["results"]
-    for result in results:
-        records = result['data']['records']
-        if len(records) == 0:
-            print("RPZ for ",parameter1,":NONE")
-        else:
-            print("RPZ for ",parameter1,":")
-            for record in records:
-                feed_name = record["feed_name"]
-                feed_name = feed_name.ljust(27)
-                class_name = record["class"]
-                properties = record["property"]
-                detected = record["detected"]
-                expiration = record["expiration"]
-                detected = datetime.strptime(detected, "%Y-%m-%dT%H:%M:%SZ")
-                detected = detected.strftime("%Y-%m-%d")
-                expiration = datetime.strptime(expiration, "%Y-%m-%dT%H:%M:%SZ")
-                expiration = expiration.strftime("%Y-%m-%d")
-                result = is_date_in_future(expiration)
-                if result is True:
-                    active = "Active"
-                else:
-                    active = "Expired"
-                print(feed_name,"\t(",properties,")\t(",active,")",detected,"to",expiration)
-else:
-    print("Error:", response.status_code)
-DATA = "{\"target\": {\"one\": {\"type\": \"host\", \"target\": \"" + parameter1 + "\",\"sources\":[\"infoblox_web_cat\"]}}}"
-# Make the cURL request
-url = "https://csp.infoblox.com/tide/api/services/intel/lookup/jobs?wait=true"
-headers = {
-    "Content-Type": "application/json",
-    "Authorization": "Token " + APIKEY
-}
-response = requests.post(url, data=DATA, headers=headers)
-# Check if the request was successful (status code 200)
-if response.status_code == 200:
-    data = response.json()
-    results = data["results"]
-    print("")
-    for result in results:
-        records = result['data']['results']
-        if len(records) == 0:
-            print("WebCat for ",parameter1,":NONE")
-        else:
-            for record in records:
-                cat_name = record["name"]
-                print("WebCat for ",parameter1,":",cat_name)
-else:
-    print("Error:", response.status_code)
-DATA = "{\"target\": {\"one\": {\"type\": \"host\", \"target\": \"" + parameter1 + "\",\"sources\":[\"atp\"]}}}"
-# Make the cURL request
-url = "https://csp.infoblox.com/tide/api/services/intel/lookup/jobs?wait=true"
-headers = {
-    "Content-Type": "application/json",
-    "Authorization": "Token " + APIKEY
-}
-response = requests.post(url, data=DATA, headers=headers)
-# Check if the request was successful (status code 200)
-if response.status_code == 200:
-    data = response.json()
-    results = data['results'][0]['data']['threat']
-    print("")
-    if len(results) == 0:
-        print("No Threat Data for ",parameter1)
-    else:
-        print("Threat Data for ",parameter1)
-        print("FullProfile\tProperty\t\t\tExpiration\tConfidence\tConfidenceScore\t\tThreatLevel\tThreatScore\tRiskScore")
-        for result in results:
-            # Keep this IF statement if you only want to inlucde Infoblox data and not SURBL or FARSIGHT
-            var_profile = "NA"
-            var_confidence = "NA"
-            var_expiration = "NA"
-            var_full_profile = "NA"
-            var_property = "NA"
-            var_threat_level = "NA"
-            var_confidence_score_rating = "NA"
-            var_risk_score_rating = "NA"
-            var_threat_score_rating = "NA"
-            if 'profile' in result:
-                var_profile = result['profile']
-            if 'confidence' in result:
-                var_confidence = result['confidence']
-            if 'expiration' in result:
-                var_expiration = result['expiration']
-                var_expiration = datetime.strptime(var_expiration, "%Y-%m-%dT%H:%M:%S.%fZ")
-                var_expiration = var_expiration.strftime("%Y-%m-%d")
-                dateres = is_date_in_future(var_expiration)
-                if dateres is False:
-                    var_expiration = "Expired"
-            if 'full_profile' in result:
-                var_full_profile = result['full_profile']
-            if 'property' in result:
-                var_property = result['property']
-            if len(var_property) < 22:
-                var_property += "\t"
-            if 'threat_level' in result:
-                var_threat_level = result['threat_level']
-            if 'confidence_score_rating' in result:
-                var_confidence_score_rating = result['confidence_score_rating']
-            if 'risk_score_rating' in result:
-                var_risk_score_rating = result['risk_score_rating']
-            if 'threat_score_rating' in result:
-                var_threat_score_rating = result['threat_score_rating']
-            if result['profile'] == 'IID':
-                if var_expiration != 'Expired':
-                    print(var_full_profile,"\t",var_property,"\t",var_expiration,"\t",var_confidence,"\t\t",var_confidence_score_rating,"\t\t\t",var_threat_level,"\t\t",var_threat_score_rating,"\t\t",var_risk_score_rating)
-else:
-    print("Error:", response.status_code)
-</code>