It exists several standards and tools defined to share intel feeds, like CybOX, openIOC, STIX, TAXII or MISP.
During this post I'm going to describe how I created a simple platform to consume OSINT IOCs base on malicious IP which can be used to analyse traffic. In the next post I'll do the same but using URL and Domains OSINT IOCs. This setup can be very useful in a small environment where there is not a real SEAM. Also, for incident response and malware analysis this setup can help as any captured traffic can be matched against the indicators from the differents OSINT list. In the end of the post I'll show a real example of a Locky infection which is going on at the moment.
In essence, the main idea behind is to create a file which contains the list of IP indicators which is being used by Snort to match the traffic.
Requirements
- Security Onion: This Linux distribution contains all the necessary tools, like snort IDS, squert, ELSA, etc. A good resource to install Security Onion is its Wiki
- OSINT IP Intel Feeds: There are plenty of intel feeds around which can be used. You need always to check the license of the feeds as some of them might not be used for commercial purposes. Also, some feeds require to subscribe, even if they are free. The list of feeds I use are the following:
- https://ransomwaretracker.abuse.ch/downloads/RW_IPBL.txt
- https://zeustracker.abuse.ch/blocklist.php?download=badips
- https://zeustracker.abuse.ch/blocklist.php?download=ipblocklist
- https://reputation.alienvault.com/reputation.generic
- https://www.openbl.org/lists/base.txt
- https://lists.blocklist.de/lists/bots.txt
- http://cinsscore.com/list/ci-badguys.txt
- http://myip.ms/files/blacklist/general/latest_blacklist.txt
- http://myip.ms/files/blacklist/general/latest_blacklist_users_submitted.txt
- https://www.autoshun.org/download/?api_key=XXXXXXXXXX&format=csv (requieres registration and an API key)
- http://wget-mirrors.uceprotect.net/rbldnsd-all/dnsbl-1.uceprotect.net.gz
- https://malc0de.com/bl/IP_Blacklist.txt
- http://www.urlvir.com/export-ip-addresses/
- http://www.malwaredomainlist.com/hostslist/ip.txt
- http://osint.bambenekconsulting.com/feeds/c2-ipmasterlist.txt
- http://osint.bambenekconsulting.com/feeds/c2-ipmasterlist-high.txt
- http://www.ciarmy.com/list/ci-badguys.txt
Installation
I assume Security Onion is already setup and running.
The first step that needs to be done is to include the blacklist files which contains the indicators. This is done through the snort.conf, which is located in /etc/nsm/securityonion-eth0/snort.conf.
The preprocessor reputation needs to include the files where the indicators are stored. The default configuration from snort uses a file "black_list.rules" which contains some bad indicators. This default file is obtained from http://labs.snort.org/feeds/ip-filter.blf and can be retrieve through pulledpork
The first step that needs to be done is to include the blacklist files which contains the indicators. This is done through the snort.conf, which is located in /etc/nsm/securityonion-eth0/snort.conf.
The preprocessor reputation needs to include the files where the indicators are stored. The default configuration from snort uses a file "black_list.rules" which contains some bad indicators. This default file is obtained from http://labs.snort.org/feeds/ip-filter.blf and can be retrieve through pulledpork
# Reputation preprocessor. For more information see README.reputation
var WHITE_LIST_PATH /etc/nsm/rules
var BLACK_LIST_PATH /etc/nsm/rules
...
...
preprocessor reputation: \ memcap 500, \ priority whitelist, \ nested_ip inner, \ whitelist $WHITE_LIST_PATH/white_list.rules, \ blacklist $BLACK_LIST_PATH/black_list.rules, \ blacklist $BLACK_LIST_PATH/black_list_custom.rules
The second file, "black_list_custom.rules" is the one we will use to include our own indicators. It is possible to create as many files as wished.
To fill that file, I've created a shell script which retrieves all the OSINT IP indicators from the different URLs, parse the fields to extract only the IP, and dump in a file. Then Snort is restarted to include the new indicators. It is possible always to include the script in a cron task.
(The script could be improved or rewritten in other language, but I needed something quick to perform some analysis. I let you improve it ;-)) )
#!/bin/bash mkdir /tmp/watchlist cd /tmp/watchlist wget https://ransomwaretracker.abuse.ch/downloads/RW_IPBL.txt -O ransomware-abuse-ch.txt cat ransomware-abuse-ch.txt | grep -v "#" > ransomware-abuse-ch_clean.txt wget https://zeustracker.abuse.ch/blocklist.php?download=badips -O zeus-abuse-ch.txt cat zeus-abuse-ch.txt | grep -v "#" > zeus-abuse-ch_clean.txt wget https://zeustracker.abuse.ch/blocklist.php?download=ipblocklist -O zeus-abuse-block-ch.txt cat zeus-abuse-block-ch.txt | grep -v "#" > zeus-abuse-block-ch_clean.txt wget https://reputation.alienvault.com/reputation.generic -O alienvault.txt cat alienvault.txt | cut -d "#" -f 1 > alienvault_clean.txt wget https://www.openbl.org/lists/base.txt -O openbl.txt cat openbl.txt | grep -v "#" > openbl_clean.txt wget https://lists.blocklist.de/lists/bots.txt -O blocklist.txt cat blocklist.txt | cut -d "#" -f 1 > blocklist_clean.txt wget http://cinsscore.com/list/ci-badguys.txt -O cisscore.txt cat cisscore.txt | cut -d "#" -f 1 > cisscore_clean.txt wget http://myip.ms/files/blacklist/general/latest_blacklist.txt -O myip.txt cat myip.txt | cut -d "#" -f 1 | grep -v ":" > myip_clean.txt wget http://myip.ms/files/blacklist/general/latest_blacklist_users_submitted.txt -O myip_2.txt cat myip_2.txt | cut -d "#" -f 1 | grep -v ":" > myip_2_clean.txt ## Need registration to get the Api Key wget "https://www.autoshun.org/download/?api_key=------&format=csv" -O autoshun.txt if [ -s autoshun.txt ] then cat autoshun.txt | grep -v "#" | cut -d "," -f1 > autoshun_clean.txt fi wget http://wget-mirrors.uceprotect.net/rbldnsd-all/dnsbl-1.uceprotect.net.gz -O dnsbl-1.uceprotect.net.gz gunzip -f dnsbl-1.uceprotect.net.gz cat dnsbl-1.uceprotect.net | grep -v "#" | grep -v "!" | grep -v "127.0.0.2" | grep -v "SOA" > dnsbl-1.uceprotect_clean.txt wget https://malc0de.com/bl/IP_Blacklist.txt -O malc0de.txt cat malc0de.txt | grep -v "/" > malc0de_clean.txt wget http://www.urlvir.com/export-ip-addresses/ -O urlvir.txt cat urlvir.txt | grep -v "#" >urlvir_clean.txt wget http://www.malwaredomainlist.com/hostslist/ip.txt -O malwaredomain.txt cp malwaredomain.txt malwaredomain_clean.txt wget http://osint.bambenekconsulting.com/feeds/c2-ipmasterlist.txt -O bambenek-c2.txt cat bambenek-c2.txt | cut -d "#" -f 1 | cut -d "," -f 1 > bambenek-c2_clean.txt wget http://osint.bambenekconsulting.com/feeds/c2-ipmasterlist-high.txt -O bambenek-c2-high.txt cat bambenek-c2-high.txt | cut -d "#" -f 1 | cut -d "," -f 1 > bambenek-c2-high_clean.txt wget http://www.ciarmy.com/list/ci-badguys.txt -O ciarmy.txt cat ciarmy.txt | grep -v "#" > ciarmy_clean.txt cat *clean.txt | grep -v ":" | sort | uniq > black_list_custom.rules cp /tmp/watchlist/black_list_custom.rules /etc/nsm/rules/black_list_custom.rules nsm_sensor_ps-restart --only-snort-alert
We can see the amount of indicators included in the logs /var/log/nsm/securityonion-eth0/snortu-1.log. In this case there are around 900k indicators
"Reputation entries loaded: 905941, invalid: 1, re-defined: 8510 (from file /etc/nsm/rules/black_list_custom.rules)"
Analysis of some network traffic
Now, it is possible to check any network traffic against the indicators. You could put Security Onion in a live environment. But also you use it to perform forensic analysis on captured traffic. For that, the tcpreplay command is very handy as the traffic can be replied in security onion.
The command used is: "sudo tcpreplay --intf1=eth0 suspicious_traffic.pcap"
All the alerts can be seen now in the 'squert' portal. The triggered signature is "reputation: Packet is blacklisted"
There are two different flows involved. Actually, there are 2 different IP indicators
Going deeper in the analysis, I can see the traffic that generated the first alert, if I click in the event ID (3.12368)
The event is opened in capme. This tools shows the full traffic generating the alert. In this case, I see it is some HTTP traffic. The GET requests a file in htXXp://newjobdool.top/admin.php?f=1 which it is a DOS MZ executable file.
The EXE file (9c2ee870c074a08a6e9211aa5c06a20db1897854e3b9d461d3468c31902dbefb) downloaded has been reported today.
The other event traffic is also showed in capme and it is also an HTTP session to the C&C. The C&C is hosted in hxxp://185.51.247.211/php/upload.php
It is very easy to configure a full functional setup of OSINT IOC list which can be feed into Snort and perform real network traffic analysis. Besides, with the tools from Security Onion the visualisation of the incidents generated by the IOC give a lot of information and a full analysis of the traffic.
Next post I'll explain how to do the same with URL/Domain indicators.
There are two different flows involved. Actually, there are 2 different IP indicators
Going deeper in the analysis, I can see the traffic that generated the first alert, if I click in the event ID (3.12368)
The event is opened in capme. This tools shows the full traffic generating the alert. In this case, I see it is some HTTP traffic. The GET requests a file in htXXp://newjobdool.top/admin.php?f=1 which it is a DOS MZ executable file.
The EXE file (9c2ee870c074a08a6e9211aa5c06a20db1897854e3b9d461d3468c31902dbefb) downloaded has been reported today.
The other event traffic is also showed in capme and it is also an HTTP session to the C&C. The C&C is hosted in hxxp://185.51.247.211/php/upload.php
It is very easy to configure a full functional setup of OSINT IOC list which can be feed into Snort and perform real network traffic analysis. Besides, with the tools from Security Onion the visualisation of the incidents generated by the IOC give a lot of information and a full analysis of the traffic.
Next post I'll explain how to do the same with URL/Domain indicators.
