Threat Intelligence Feeds Part I. - Using Security Onion and Snort

Currently there many companies offering Threat Intelligence feeds which can be integrated easily in SEAM platforms like Splunk, Arcsight or similar. Some of this companies uses Open Source Intelligence IOCs feeds as part of their intelligence, but also they create their own intelligence feeds base on their own research.

It exists several standards and tools defined to share intel feeds, like CybOX, openIOC, STIX, TAXII or MISP.

During this post I'm going to describe how I created a simple platform to consume OSINT IOCs base on malicious IP which can be used to analyse traffic. In the next post I'll do the same but using URL and Domains OSINT IOCs. This setup can be very useful in a small environment where there is not a real SEAM. Also, for incident response and malware analysis this setup can help as any captured traffic can be matched against the indicators from the differents OSINT list. In the end of the post I'll show a real example of a Locky infection which is going on at the moment.

In essence, the main idea behind is to create a file which contains the list of IP indicators which is being used by Snort to match the traffic. 

Requirements

• Security Onion:  This Linux distribution contains all the necessary tools, like snort IDS, squert, ELSA, etc. A good resource to install Security Onion is its Wiki 
• OSINT IP Intel Feeds: There are plenty of intel feeds around which can be used. You need always to check the license of the feeds as some of them might not be used for commercial purposes. Also, some feeds require to subscribe, even if they are free. The list of feeds I use are the following:
• https://ransomwaretracker.abuse.ch/downloads/RW_IPBL.txt
• https://zeustracker.abuse.ch/blocklist.php?download=badips
• https://zeustracker.abuse.ch/blocklist.php?download=ipblocklist
• https://reputation.alienvault.com/reputation.generic
• https://www.openbl.org/lists/base.txt
• https://lists.blocklist.de/lists/bots.txt
• http://cinsscore.com/list/ci-badguys.txt
• http://myip.ms/files/blacklist/general/latest_blacklist.txt
• http://myip.ms/files/blacklist/general/latest_blacklist_users_submitted.txt
• https://www.autoshun.org/download/?api_key=XXXXXXXXXX&format=csv (requieres registration and an API key)
• http://wget-mirrors.uceprotect.net/rbldnsd-all/dnsbl-1.uceprotect.net.gz
• https://malc0de.com/bl/IP_Blacklist.txt
• http://www.urlvir.com/export-ip-addresses/ 
• http://www.malwaredomainlist.com/hostslist/ip.txt
• http://osint.bambenekconsulting.com/feeds/c2-ipmasterlist.txt
• http://osint.bambenekconsulting.com/feeds/c2-ipmasterlist-high.txt
• http://www.ciarmy.com/list/ci-badguys.txt

Installation

I assume Security Onion is already setup and running.
The first step that needs to be done is to include the blacklist files which contains the indicators. This is done through the snort.conf, which is located in /etc/nsm/securityonion-eth0/snort.conf. 

The preprocessor reputation needs to include the files where the indicators are stored. The default configuration from snort uses a file "black_list.rules" which contains some bad indicators. This default file is obtained from http://labs.snort.org/feeds/ip-filter.blf and can be retrieve through pulledpork

# Reputation preprocessor. For more information see README.reputation

var WHITE_LIST_PATH /etc/nsm/rules
var BLACK_LIST_PATH /etc/nsm/rules

...

...

preprocessor reputation: \
   memcap 500, \
   priority whitelist, \
   nested_ip inner, \
   whitelist $WHITE_LIST_PATH/white_list.rules, \
   blacklist $BLACK_LIST_PATH/black_list.rules, \
   blacklist $BLACK_LIST_PATH/black_list_custom.rules

The second file, "black_list_custom.rules" is the one we will use to include our own indicators. It is possible to create as many files as wished.

To fill that file, I've created a shell script which retrieves all the OSINT IP indicators from the different URLs, parse the fields to extract only the IP, and dump in a file. Then Snort is restarted to include the new indicators. It is possible always to include the script in a cron task. 

(The script could be improved or rewritten in other language, but I needed something quick to perform some analysis. I let you improve it ;-)) )

#!/bin/bash


mkdir /tmp/watchlist
cd /tmp/watchlist

wget https://ransomwaretracker.abuse.ch/downloads/RW_IPBL.txt -O ransomware-abuse-ch.txt
 cat ransomware-abuse-ch.txt | grep -v "#" > ransomware-abuse-ch_clean.txt
wget https://zeustracker.abuse.ch/blocklist.php?download=badips -O zeus-abuse-ch.txt
 cat zeus-abuse-ch.txt | grep -v "#" > zeus-abuse-ch_clean.txt
wget https://zeustracker.abuse.ch/blocklist.php?download=ipblocklist -O zeus-abuse-block-ch.txt
 cat zeus-abuse-block-ch.txt | grep -v "#" > zeus-abuse-block-ch_clean.txt
wget https://reputation.alienvault.com/reputation.generic -O alienvault.txt
 cat alienvault.txt | cut -d "#" -f 1 > alienvault_clean.txt
wget https://www.openbl.org/lists/base.txt -O openbl.txt
 cat openbl.txt | grep -v "#" > openbl_clean.txt
wget https://lists.blocklist.de/lists/bots.txt -O blocklist.txt
 cat blocklist.txt | cut -d "#" -f 1 > blocklist_clean.txt
wget http://cinsscore.com/list/ci-badguys.txt -O cisscore.txt
 cat cisscore.txt | cut -d "#" -f 1 > cisscore_clean.txt
wget http://myip.ms/files/blacklist/general/latest_blacklist.txt -O myip.txt
 cat myip.txt | cut -d "#" -f 1 | grep -v ":" > myip_clean.txt
wget http://myip.ms/files/blacklist/general/latest_blacklist_users_submitted.txt -O myip_2.txt
 cat myip_2.txt | cut -d "#" -f 1 | grep -v ":" > myip_2_clean.txt

  ## Need registration to get the Api Key
wget "https://www.autoshun.org/download/?api_key=------&format=csv" -O autoshun.txt
 if [ -s autoshun.txt ]
 then
  cat autoshun.txt | grep -v "#"  | cut -d "," -f1  >  autoshun_clean.txt
 fi

wget http://wget-mirrors.uceprotect.net/rbldnsd-all/dnsbl-1.uceprotect.net.gz -O dnsbl-1.uceprotect.net.gz
 gunzip -f dnsbl-1.uceprotect.net.gz
  cat dnsbl-1.uceprotect.net | grep -v "#" | grep -v "!" | grep -v "127.0.0.2" | grep -v "SOA" > dnsbl-1.uceprotect_clean.txt
wget https://malc0de.com/bl/IP_Blacklist.txt -O malc0de.txt
 cat malc0de.txt | grep -v "/" > malc0de_clean.txt
wget http://www.urlvir.com/export-ip-addresses/ -O urlvir.txt
 cat urlvir.txt | grep -v "#" >urlvir_clean.txt
wget http://www.malwaredomainlist.com/hostslist/ip.txt -O malwaredomain.txt
 cp malwaredomain.txt malwaredomain_clean.txt
wget http://osint.bambenekconsulting.com/feeds/c2-ipmasterlist.txt -O bambenek-c2.txt
 cat bambenek-c2.txt | cut -d "#" -f 1 | cut -d "," -f 1 > bambenek-c2_clean.txt
wget http://osint.bambenekconsulting.com/feeds/c2-ipmasterlist-high.txt -O bambenek-c2-high.txt
 cat bambenek-c2-high.txt | cut -d "#" -f 1 | cut -d "," -f 1 > bambenek-c2-high_clean.txt
wget http://www.ciarmy.com/list/ci-badguys.txt -O ciarmy.txt
 cat ciarmy.txt | grep -v "#" >  ciarmy_clean.txt

cat *clean.txt | grep -v ":" | sort | uniq > black_list_custom.rules

cp /tmp/watchlist/black_list_custom.rules /etc/nsm/rules/black_list_custom.rules


nsm_sensor_ps-restart --only-snort-alert

We can see the amount of indicators included in the logs /var/log/nsm/securityonion-eth0/snortu-1.log. In this case there are around 900k indicators


 "Reputation entries loaded: 905941, invalid: 1, re-defined: 8510 (from file /etc/nsm/rules/black_list_custom.rules)"

Analysis of some network traffic

Now, it is possible to check any network traffic against the indicators. You could put Security Onion in a live environment. But also you use it to perform forensic analysis on captured traffic. For that, the tcpreplay command is very handy as the traffic can be replied in security onion.

The command used is: "sudo tcpreplay --intf1=eth0 suspicious_traffic.pcap"

All the alerts can be seen now in the 'squert' portal. The triggered signature is "reputation: Packet is blacklisted"




There are two different flows involved. Actually, there are 2 different IP indicators

Going deeper in the analysis, I can see the traffic that generated the first alert, if I click in the event ID (3.12368)


The event is opened in capme. This tools shows the full traffic generating the alert. In this case, I see it is some HTTP traffic. The GET requests a file in htXXp://newjobdool.top/admin.php?f=1 which it is a DOS MZ executable file. 




The EXE file (9c2ee870c074a08a6e9211aa5c06a20db1897854e3b9d461d3468c31902dbefb) downloaded has been reported today.






The other event traffic is also showed in capme and it is also an HTTP session to the C&C. The C&C is hosted in hxxp://185.51.247.211/php/upload.php





It is very easy to configure a full functional setup of OSINT IOC list which can be feed into Snort and perform real network traffic analysis. Besides, with the tools from Security Onion the visualisation of the incidents generated by the IOC give a lot of information and a full analysis of the traffic.


Next post I'll explain how to do the same with URL/Domain indicators.