Pulsedive Lookup with MSTICPy

Author: Thomas Roccia | @fr0gger_

The Pulsedive Lookup module in MSTICPy allows you to easily access the Pulsedive threat intelligence platform from within your Python scripts. With this module, you can search, scan, and enrich IPs, URLs, domains, and other Indicator of Compromise (IOC) from Open Source Intelligence (OSINT) feeds or even submit your own data.

This notebook serves as a demonstration of how to use the Pulsedive module in MSTICpy to perform threat enrichment tasks. With this module, you can quickly and easily retrieve information about known threats, scan indicators for potential threats, and explore the Pulsedive database using various search criteria.

To learn more about the Pulsedive API and the various functions available through the module, please visit the official Pulsedive API documentation at https://pulsedive.com/api/.

The Pulsedive support in MSTICPy has two components

  • A regular TI Provider invoked using TILookup or Pivot functions
  • The PDLookup class, which has other operations and lookup types available.

Pulsedive configuration and API Key

Before you can use the Pulsedive API you must create an account at the site and obtain and API key. You can use the API key interactively or configure a Pulsedive entry in your msticpyconfig.yaml

TIProviders:
  ...
  Pulsedive:
    Args:
      AuthKey: your_pd_api_key
    Primary: True
    Provider: Pulsedive

MSTICPy TILookup and Pulsedive

Adding a configuration for Pulsedive to msticpyconfig.yaml also adds Pulsedive as a regular MSTICPy TI Provider that you can use as follows:

ti_lookup = TILookup()
single_result = ti_lookup.lookup_ioc("21.3.4.5", providers=["Pulsedive"])

# multiple observables - you can include multiple observable types
# in the same list
ip_list = ["21.3.4.5", "19.8.7.6", "201.45.178.2", ...]
ti_lookup.lookup_iocs(my_ip_list, providers=["Pulsedive"])

If you supply no value for providers, all configured providers will be used.

You can also use the MSTICPy pivot interface to lookup specific observable types:

IpAddress.ti.lookup_ip(ip_list)

Initializing the PDlookup class

The PDLookup class lets you perform operations in Pulsedive, beyond simple indicator lookup, such as: threat lookup, domain scanning, and the Pulsedive query explorer interface.

In [ ]:
# import the Pulsedive module
from msticpy.context.tiproviders.pulsedive import PDlookup
In [2]:
# Use the PDlookup class to get more details about the IOC. Specify your API key here.
pdlookup = PDlookup(pd_key='')

Lookup IOC

The lookup_ioc function is used to request data from the Pulsedive API by providing an observable and a Pulsedive type. The observable is the specific data being requested, such as a domain name or IP, and the Pulsedive type corresponds to the type of data being requested.

You can use lookup_ioc to perform all types of lookup operations by supplying the appropriate value for pd_type. The default for lookup_ioc is "indicator".

Alteratively you can use the direct methods:

  • PDlookup.lookup_threat
  • PDlookup.explore
  • PDlookup.scan

The available Pulsedive types are:

  • 'indicator': Retrieves information about a specific Indicator of Compromise (IOC), such as a domain name or IP.
  • 'threat': Retrieves information about a specific threat, such as a threat actor name or malware family. (e.g: ryuk, zeus)
  • 'explore': Queries the Pulsedive database using specific keywords.(e.g: "ioc=pulsedive.com or threat=ryuk")
  • 'scan': Scans a domain name or IP.

Indicator

In [3]:
pddetail = pdlookup.lookup_ioc(observable = "alvoportas.com.br", pd_type = "indicator")
display(pddetail)
qid iid indicator type risk risk_recommended manualrisk retired stamp_added stamp_updated ... properties.banners.22 properties.banners.21 properties.dns.a properties.dns.ns properties.dns.rname properties.dns.mname properties.dns.mx properties.dns.txt properties.dns.soa properties.dom.screenshot
0 None 2 alvoportas.com.br domain none none 0 None 2017-09-27 18:11:38 2023-01-19 03:01:59 ... SSH-2.0-OpenSSH_7.4\n 220---------- Welcome to Pure-FTPd [privsep] [... 162.241.60.111 [nspro15.hostgator.com.br, nspro14.hostgator.c... root@sh-pro14.hostgator.com.br nspro14.hostgator.com.br mail.alvoportas.com.br v=spf1 ip4:69.49.252.72 ip4:162.241.60.107 a m... nspro14.hostgator.com.br. root.sh-pro14.hostga... https://sandbox.pulsedive.com/screenshots/b590...

1 rows × 79 columns

Threats

In [4]:
pddetail = pdlookup.lookup_threa(observable="zeus")
display(pddetail)
tid threat category othernames risk description wikisummary wikireference retired stamp_added ... summary.attributes.technology.Nginx.indicators summary.attributes.technology.Nginx.aid summary.risk.unknown summary.risk.low summary.risk.medium summary.risk.high summary.risk.critical summary.risk.retired summary.risk.total summary.risk.none
0 1 Zeus malware [Zbot] high Zeus, ZeuS, or Zbot is a Trojan horse malware... https://en.wikipedia.org/wiki/Zeus_(malware) None 2017-09-27 18:11:38 ... 23 65095 42 21 10 2 0 744 1000 181

1 rows × 85 columns

Explore

The query language allows you to search across the dataset with boolean logic and wildcards. You can search indicators by value, type, risk, last seen timestamp, threat, feed, attribute, and property, or any combination. More details here: https://pulsedive.com/explore/

Search Fields

The below tables show the field that can be used for this request:

Search Field Description Example Query
ioc Search by indicator value. Default if the search field is omitted. ioc=pulsedive.com, ioc=pulsedive*
attribute Search by attribute. Includes ports, protocols, and technologies. port=443, protocol=http*, technology=apache, port=80 and port=443
property Search by property. dns.a=45.55.106.210, meta=pulsedive, content-type=text/html, ssl="let's encrypt*"
threat Search by threat name or alias. threat=ryuk, threat=zeus
feed Search by feed name or organization. feed=urlhaus, feed=abuse.ch
type Search by indicator type. type=url, type=domain,ip,ipv6
risk Search by indicator risk. risk=critical, risk=low,medium, risk=high+, risk=low-
seen Search by Last Seen timestamp. (UTC) seen=day, seen=week, seen=month, seen=2020-01-01, seen=2022-01-01+, seen=2021-12-31-, seen=2020-01-01-2020-12-31
active, retired Search by active or retired status. active=true, retired=0

Boolean Logic and Wildcards

Explore queries allow for AND, OR, and NOT operations. Wildcards are also allowed.

Operation Operator Example Query
AND &, &&, and pulsedive.com && type=domain
pulsedive.com type=domain
OR or google.com or pulsedive*
(pulsedive and type=domain) threat=phishing
NOT != risk!=medium-
Wildcard * pulsedive
In [5]:
pddetail = pdlookup.explore(query="ioc=pulsedive.com or threat=AgentTesla")
pddetail
Out[5]:
iid indicator type risk stamp_added stamp_updated stamp_seen summary.properties.whois.++privacy summary.properties.geo.country summary.properties.geo.countrycode ... summary.properties.http.++content-type summary.attributes summary.domainiid summary.properties.whois.++gdpr summary.domain summary.properties.geo.org summary.properties.dns.ptr stamp_retired summary.properties.http summary
0 53929 pulsedive.com domain none 2017-10-04 01:20:55 2023-01-19 02:20:24 2023-01-19 02:20:24 1 Canada CA ... text/html NaN NaN NaN NaN NaN NaN NaN NaN NaN
1 102 dns10.parkpage.foundationapi.com domain none 2017-09-27 18:11:57 2023-01-17 20:18:41 2023-01-17 20:18:41 United States of America US ... text/html [] 103.0 foundationapi.com NaN NaN NaN NaN NaN
2 107 dns9.parkpage.foundationapi.com domain none 2017-09-27 18:11:57 2023-01-17 20:18:41 2023-01-17 20:18:41 United States of America US ... text/html [] 103.0 foundationapi.com NaN NaN NaN NaN NaN
3 116 209.99.40.222 ip none 2017-09-27 18:11:58 2023-01-17 07:09:04 2023-01-17 04:14:28 NaN VG US ... text/html NaN NaN NaN NaN CENTURYLINK-LEGACY-LVLT-203 209-99-40-222.fwd.datafoundry.com NaN NaN NaN
4 117 ns2.parkingcrew.net domain none 2017-09-27 18:11:58 2023-01-18 16:00:04 2023-01-18 16:00:04 1 Germany DE ... text/html [] 118.0 1 parkingcrew.net NaN NaN NaN NaN NaN
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
95 3673600 http://www.agentfalco.xyz/Webl/word.exe url medium 2018-12-02 17:44:38 2021-11-03 23:43:16 2021-08-02 05:32:05 NaN NaN NaN ... NaN [] 4340111.0 NaN www.agentfalco.xyz NaN NaN 2021-11-03 23:43:16 NaN NaN
96 3673823 https://f.coka.la/EJ6Q7V.jpg url medium 2018-12-02 17:45:12 2021-11-03 23:43:16 2021-08-02 05:36:52 NaN NaN NaN ... text/html [] 3655358.0 NaN f.coka.la NaN NaN 2021-11-03 23:43:16 NaN NaN
97 3687390 23.20.239.12 ip none 2018-12-05 05:47:57 2023-01-17 07:57:32 2023-01-12 05:56:22 NaN United States of America US ... text/html NaN NaN NaN NaN AMAZON-AES ec2-23-20-239-12.compute-1.amazonaws.com NaN NaN NaN
98 3687391 18.211.9.206 ip none 2018-12-05 05:47:57 2023-01-17 05:07:37 2022-06-01 01:47:02 NaN United States of America US ... NaN NaN NaN NaN NaN AMAZON-AES ec2-18-211-9-206.compute-1.amazonaws.com 2022-09-03 02:44:42 NaN NaN
99 3694195 http://amsi.co.za/zzam/cjz.exe url medium 2018-12-06 06:33:12 2021-10-01 06:13:44 2021-07-01 05:07:00 NaN REDACTED ZA ... text/html [] 3164623.0 1 amsi.co.za NaN NaN 2021-10-01 06:13:44 NaN NaN

100 rows × 23 columns

Scan

This type allows you to analyze a specific indicator such as a domain name or IP address by sending a request to the Pulsedive API. The function returns the result in the form of a Pandas DataFrame, however, it should be noted that this process can take some time as the indicator is being analyzed. This function is useful for analyzing indicators that have not yet been seen in the Pulsedive database and can provide additional information and context.

In [6]:
pdscan = pdlookup.scan(observable= "alvoportas.com.br")
pdscan
processing
Out[6]:
qid iid indicator type risk risk_recommended manualrisk retired stamp_added stamp_updated ... properties.dns.txt properties.dns.soa properties.dom.screenshot links.Active DNS links.Sources links.Mail Servers links.Name Servers links.Related Domains links.Redirects links.Related URLs
0 722111366 2 alvoportas.com.br domain none none 0 None 2017-09-27 18:11:38 2023-01-19 03:04:22 ... v=spf1 ip4:69.49.252.72 ip4:162.241.60.107 a m... nspro14.hostgator.com.br. root.sh-pro14.hostga... https://sandbox.pulsedive.com/screenshots/b590... [{'iid': 6, 'indicator': 'ns2.emidhost4.com.br... [{'iid': 4549968, 'indicator': 'https://www.al... [{'iid': 37438712, 'indicator': 'mail.alvoport... [{'iid': 11175125, 'indicator': 'nspro14.hostg... [{'iid': 4238057, 'indicator': 'www.alvoportas... [{'iid': 37438713, 'indicator': 'http://alvopo... [{'iid': 2839792, 'indicator': 'http://alvopor...

1 rows × 86 columns