Carbon Black 4.0+ ships with support for threat intelligence feeds. The Indicators of Compromise (IOCs) contained in the feeds are compared to the sensor data as it arrives on the server. Any activity matching an IOC is tagged; users can search for the tags and, optionally, register for e-mail alerts.

Feeds allow Carbon Black servers to use freely available threat intelligence, proprietary customer threat data, and provides a mechanism to feed threat indicators from on-premise analytic sources to Carbon Black for verification, detection, visibility and analysis.

The Carbon Black 4.0+ server supports three types of indicators:

• Binary MD5s
• IPv4 addresses
• DNS names

The Carbon Black 5.0+ server adds support for two new types of indicators:

• Process Queries (Process Searches)
• Binary Queries (Binary Searches)

Please note that query IOC types have significant differences as compared to MD5s, IPv4 addresses, and DNS names. Please see notes below regarding their usage.

The feed format, described in the "Feed Structure" section below, is designed for simplicity. This should make it easy to add support for feed data from any input source.

Example feed creation scripts are included. See the 'Examples' section in this document for a listing of the examples.

The Carbon Black Feeds API (CBFAPI) is found on github at:

https://github.com/carbonblack/cbfeeds

The CBFAPI is a collection of documentation, example scripts, and a helper library to help create and validate Carbon Black feeds. It is not required in order to build a Carbon Black feed - a feed can be created in any language that allows for building JSON, or even built by hand. The feed file itself must match the feed structure, or schema, defined in the "Feed Structure" section below.

[root@localhost carbonblack]# yum install git
...

[root@localhost carbonblack]# git clone https://github.com/carbonblack/cbfeeds.git
Initialized empty Git repository in /root/repos/carbonblack/cbfeeds/.git/
remote: Reusing existing pack: 80, done.
remote: Counting objects: 25, done.
remote: Compressing objects: 100% (25/25), done.
Receiving objects: 100% (105/105), 38.03 KiB | 17 KiB/s, done.
Resolving deltas: 100% (50/50), done.
remote: Total 105 (delta 10), reused 0 (delta 0)

[root@localhost carbonblack]# ls
cbfeeds
[root@localhost carbonblack]# cd cbfeeds/
[root@localhost cbfeeds]# ls
cbfeeds  generate_feed_from_raw_iocs.py  generate_tor_feed.py  images  README.md  setup.py  validate_feed.py

[root@localhost cbfeeds]# python generate_tor_feed.py example_tor_feed.feed
[root@localhost cbfeeds]# ls -l example_tor_feed.feed 
-rw-r--r--. 1 root root 2179084 Mar 25 08:09 example_tor_feed.feed

[root@localhost cbfeeds]# python validate_feed.py --feedfile example_tor_feed.feed 
-> Validated that file exists and is readable
-> Validated that feed file is valid JSON
-> Validated that the feed file includes all necessary CB elements
-> Validated that all element values are within CB feed 

• Feed: a Carbon Black feed
  • FeedInfo: Feed metadata: name, description, etc
  • Reports: a list of report
    • Report metadata: title, id, URL
    • IOCs for this report

A feed is a JSON structure with two entries:

• feedinfo
• reports

The feedinfo structure is a list of basic feed metadata. reports is a list of report structures.
Each report has report metadata and a list of IOCs.

feedinfo is a JSON structure with the following entries:

Notes:

The 'name' field must not include spaces or special characters. Typically, it should be unique per-feed on a single server.

An example feedinfo structure, from the example_tor.py script:

  "feedinfo": {
    "provider_url": "http://www.dan.me.uk",
    "display_name": "Tor Exit Nodes",
    "name": "tor",
    "tech_data": "There are no requirements to share any data to receive this feed.",
    "summary": "This feed is a list of Tor Node IP addresses, updated every 30 minutes.",
    "version": 1,
    "icon": "...."
   }

A report is a JSON structure with the following entries:

CB 4.0 ships with feeds version 1 and supports four kinds of IOCs:

• IPv4 addresses
• domain names
• md5s
• query - this contains query related to modules or events

iocs is a structure with one or more of these entries:

An example reports list with two report structures, each with one IPv4 IOC, from the example_tor.py script:

  "reports": [
    {
      "timestamp": 1380773388,
      "iocs": {
        "ipv4": [
          "100.2.142.8"
        ]
      },
      "link": "https://www.dan.me.uk/tornodes",
      "id": "TOR-Node-100.2.142.8",
      "title": "As of Wed Oct  2 20:09:48 2013 GMT, 100.2.142.8 has been a TOR exit for 26 days, 0:44:42. Contact: Adam Langley <[email protected]>"
    },
    {
      "timestamp": 1380773388,
      "iocs": {
        "ipv4": [
          "100.4.7.69"
        ]
      },
      "link": "https://www.dan.me.uk/tornodes",
      "id": "TOR-Node-100.4.7.69",
      "title": "As of Wed Oct  2 20:09:48 2013 GMT, 100.4.7.69 has been a TOR exit for 61 days, 2:07:23. Contact: GPG KeyID: 0x1F40CBDC Jeremy <[email protected]>"
    }
  ]

Another example with "query" IOC:

"reports": 
[
    {
      "title": "Notepad processes", 
      "timestamp": 1388538906, 
      "iocs": {
        "query": [
          {
            "index_type": "events",
            "search_query": "cb.urlver=1&q=process_name%3Anotepad.exe"
          }
        ]
      }, 
      "score": 50, 
      "link": "http://www.myfeedserver/feed/report/notepad_proc",
      
      "id": "notepad_proc"
    },
    {
      "title": "Newly loaded modules", 
      "timestamp": 1388570000, 
      "iocs":
      {
        "query": [
          {
            "index_type": "modules",
            "search_query": "cb.urlver=1&q=is_executable_image%3Afalse"
          }
        ]
      }, 
      "score": 50,
       
      "link": "http://www.dxmtest1.org/02",
      "id": "new_mod_loads"
    }
]

Following conditions apply for "query" IOC reports

• the "iocs" element can only contain one "query" element
• only "events" and "modules" are valid values for "index_type" element
• a report with a query CANNOT also have other IOCs

The "search_query" syntax is particularly noteworthy. The following conditions apply for the "search_query" field:

• the "search_query" syntax is described in CB Enterprise Server Query Overview documentation
• the query itself should be prepended with a q=
• the query should be percent-encoded. This can be accomplished in several ways, including:
  • by copying a query from the Carbon Black UI
  • by using a quoting library such as included with python in urllib.
  • by using the included percent_encode_query.py script

As with all feeds, it is highly recommended to provide initial validation of the feed with the included validate_feed.py script. For any feeds that include query IOCs, it is recommended to run feed_query_validate.py in the cbapi github repo.

Queries IOCs impose a much higher performance cost on the Carbon Black Enterprise Server than md5, dns, and ip IOCs. Furthermore, the relative costs of queries can very signficantly. As a general rule, 'events' queries are more expensive than 'modules' queries. The use of wildcards, long paths, or multiple terms is also expensive.

It is recommended that feed developers take care in constructing query IOCs and test against representative server prior to deploying in production.

Several example scripts are included in the 'example' subdirectory. These example scripts illustrate using the Carbon Black cbfeeds API to generate Carbon Black feeds from a variety of data sources.