Managing IDS/IPS signatures can be a difficult task. Even with trained security professionals who are knowledgeable about their organization’s normal traffic pattern, most organizations configurations are continuously changing. New services and machines are put into place, creating new traffic patterns. While network IDS/IPS serve the function of finding evidence of nefarious activities, at large organization the volume of alerts can be a bit daunting. Signatures are being updated regularly to deal with changing attack vectors, adding to the number of alerts. Time to analyze is always limited. Organizations will often find themselves with too much information and too many alerts to investigate.
The use of blacklists can help. Ron Gula in his post “Event Analysis Training — Working with BlackLists” discussed how in an organization aggregating network IDS/IPs events with 10,000 to 1,000,000 events per day, Tenable has observed on a heavy day of blacklist correlation 10 to 30 hits are often generated. While those blacklist hits require minimal effort to analyze, they provide potentially very useful information. By following the interactions between blacklisted IP addresses with hosts in an organization’s network, one can find:
- systems on the organization’s network that have been compromised and are being used to send SPAM.
- possible vulnerabilities on systems within the organization’s network based on targeted ports.
- successful phishing attacks, if the organization’s IP addresses are visiting IPs associated with phishing scams.
- systems within the organization that are part of botnet, since blacklists can identify IPs that are part of the botnet command and control.
Blacklists fall into two categories: global worst offender lists (GWOL) and local worst offender lists (LWOL). LWOL will be built by an organization based on the organization’s firewalls and network activities. For GWOL, three invaluable sources providing up-to-date information pulled from various sources that are known to be used to propagate malware and spyware are:
- DNS-BH Malware Domain Blocklist: provides information maintained as part of the DNS-BH project and represents a list of domains that are known to be used to propagate malware and spyware.
- Global Watchlist: C.S. Lee describes what he and Spoonfork did in his posting “The Harimau Watchlist.” He describes the list as a “list of suspected malicous IPs/Net ranges from different sources such as SANS DShield, Arbor atlas and so forth, then putting all of them in one place.”
- Ninja Chimp Strike Force Blacklist: created on an hourly basis from data Arbor Networks, Project Honeypot, Shadowserver, and about 24+ hosts.
Martin Roesch recently posted “IP Blacklisting Version 2 for Snort 2.8.4.1 available” where he discusses version 2 of a patch created for Snort 2.8.4.1 allowing IP blacklisting.
We will walk through the process of installing Snort 2.8.4.1 with the patch in order to allow the use of blacklists. We will follow that up with creating a process that will generated the blacklist file for Snort. We will go through enough of the setup to allow the user to start using blacklists.
Required Packages
The first step is to make sure the system has the required software installed, which includes:
The aclocal, automake, and autoconf packages should be on your system. I wanted to mention them here because the version installed on your system may end up causing problems. On one system, I had a most difficult time because of the version. Since I could not update the software on the problem system, I ended up creating a local area under the Snort directory and installing the versions I required there.
The packages that are needed will be dependent on your installation base. I will walk through a few package installations.
Libdnet
Make sure Libdnet is in your library path:
root# /sbin/ldconfig -p | grep -i libdnet libdnet32.so.1 (libc6) => /usr/lib/libdnet32.so.1 libdnet32.so (libc6) => /usr/lib/libdnet32.so |
If you do not get a path returned, you will need to install libdnet (use –prefix if it needs to be installed in a special location):
root# cd /usr/local/src /usr/local/src root# wget \ http://libdnet.googlecode.com/files/libdnet-1.12.tgz /usr/local/src root# tar xzf libdnet.cvs.tgz /usr/local/src root# cd libdnet-1.12 /usr/local/src root# ./configure /usr/local/src root# make /usr/local/src root# make install /usr/local/src root# cp include/dnet/sctp.h /usr/local/include/dnet |
If you have installed libdnet in a special location, make sure to include its path in /etc/ld.so.conf.
PCRE (pcre-7.8)
The PCRE library is a set of functions that implement regular expression pattern matching using the same syntax and semantics as Perl 5. If you can install PCRE via a port specific to your operating system, that is the best way to install in order to avoid having to keep the software up-to-date. Below are the instructions for installing the software from source.
root# cd /usr/local/src /usr/local/src root# wget \ http://downloads.sourceforge.net/sourceforge/pcre/\ pcre-7.9.tar.gz?use_mirror=internap /usr/local/src root# wget \ http://downloads.sourceforge.net/sourceforge/pcre/\ pcre-7.9.tar.gz.sig?use_mirror=internap /usr/local/src root# gpg --verify pcre-7.9.tar.gz.sig pcre-7.9.tar.gz gpg: Signature made Sat 11 Apr 2009 10:33:38 AM EDT using RSA key ID FB0F43D8 gpg: Good signature from " onclick="javascript:_gaq.push(['_trackEvent','outbound-article','http://downloads.sourceforge.net']);"Philip Hazel <ph10@hermes.cam.ac.uk>" gpg: aka "Philip Hazel <ph10@cam.ac.uk>" gpg: aka "Philip Hazel <ph10@cus.cam.ac.uk>" gpg: WARNING: This key is not certified with a trusted signature! gpg: There is no indication that the signature belongs to the owner. Primary key fingerprint: 45F6 8D54 BBE2 3FB3 039B 46E5 9766 E084 FB0F 43D8 /usr/local/src root# tar xzf pcre-7.9.tar.gz /usr/local/src root# cd pcre-7.9 /usr/local/src/pcre-7.8 root# ./configure --prefix=/usr/local/pcre /usr/local/src/pcre-7.8 root# make /usr/local/src/pcre-7.8 root# make test /usr/local/src/pcre-7.8 root# make install |
Libpcap and Large Files Support
Under some OSs, you need to compile libpcap and Snort to support large files (files large than 2G). Since the source code of libpcap will be needed for this configuration, we are going to install the resulting libpcap under /usr/local/snort. If the libpcap installed on your system does not produce an error, skip this step. You will want to follow these steps only if you get the following error when running Snort with a large file:
root# ls -lh /data/ids/full2.pcap -rw-r--r-- 1 root root 12G Oct 26 10:01 /data/ids/full2.pcap root# /usr/local/snort/bin/snort -o -A none -c \ /usr/local/snort/conf/snort.conf -l /logs/snort/logs \ -r /data/ids/full2.pcap Error getting stat on pcap file: /data/ids/full2.pcap: Value too large for defined data type ERROR: Error getting pcaps Fatal Error, Quitting.. |
First, we need to compile large file support into libpcap. As mentioned above, we will install the libraries under /usr/local/snort.
root# cd /usr/local/src /usr/local/src root# wget http://www.tcpdump.org/release/\ libpcap-1.0.0.tar.gz /usr/local/src root# wget http://www.tcpdump.org/release/\ libpcap-1.0.0.tar.gz.sig /usr/local/src root# wget http://www.tcpdump.org/tcpdump-workers.asc /usr/local/src root# gpg --import tcpdump-workers.asc gpg: key 89E917F3: "tcpdump.org (SIGNING KEY) |
In the next section, we will discuss how to install Snort with blacklist, large file, and MySQL support.
Snort
Below we will get the software, verify, configure, and install the software under the /usr/local/snort area. Please adjust this to your environment. Reminder to Mac OS X and FreeBSD users, use the md5 command instead of md5sum.
root# cd /usr/local/src /usr/local/src root# wget http://dl.snort.org/snort-current/snort-2.8.4.1.tar.gz /usr/local/src root# wget http://dl.snort.org/snort-current/snort-2.8.4.1.tar.gz.md5 /usr/local/src root# cat snort-2.8.4.1.tar.gz.md5 63f4e76ae96a2d133f4c7b741bad5458 snort-2.8.4.1.tar.gz /usr/local/src root# md5sum snort-2.8.4.1.tar.gz 63f4e76ae96a2d133f4c7b741bad5458 snort-2.8.4.1.tar.gz /usr/local/src root# tar xzf snort-2.8.4.1.tar.gz /usr/local/src root# cd snort-2.8.4.1 |
At this point, we will add the blacklist path and regenerate the configuration and make files. If you do not have prelude installed, the following will occur:
/usr/local/src/snort-2.8.4.1 root# wget http://www.snort.org/users/roesch/code/iplist.patch.v2.tgz /usr/local/src/snort-2.8.4.1 root# tar xzf iplist.patch.v2.tgz /usr/local/src/snort-2.8.4.1 root# patch -p1 < iplist.patch /usr/local/src/snort-2.8.4.1 root# aclocal aclocal:configure.in:1050: warning: macro `AM_PATH_LIBPRELUDE' not found in library /usr/local/src/snort-2.8.4.1 root# autoconf configure.in:1050: error: possibly undefined macro: AM_PATH_LIBPRELUDE If this token and others are legitimate, please use m4_pattern_allow. See the Autoconf documentation. /usr/local/src/snort-2.8.4.1 root# automake |
Eoin Miller has been very active and helpful in resolving problems with the patch. In a recent post, he resolved the problem of ” warning: macro `AM_PATH_LIBPRELUDE’ not found in library” by commenting out those lines pertaining to Prelude in the configure file. The exact line number will differ between systems.
#if test "x$enable_prelude" = "xyes"; then
# AM_PATH_LIBPRELUDE(0.9.6, use_prelude="yes", use_prelude="no")
# if test "$use_prelude" = "yes"; then
# LDFLAGS="${LDFLAGS} ${LIBPRELUDE_LDFLAGS}"
# LIBS="$LIBS ${LIBPRELUDE_LIBS}"
# CFLAGS="$CFLAGS ${LIBPRELUDE_PTHREAD_CFLAGS}"
#cat >>confdefs.h <<\_ACEOF
##define HAVE_LIBPRELUDE
#_ACEOF
#
# fi
#fi
|
You may also need to change the definition of LIBDNET and DNETFLAGS in the configuration file. The configure file will have these variables set by calling dumbnet-config without any path. If your system does not know the location of dumbnet-config, this will cause problems. Instead, figure out the values and replace the calls to dumbnet-config with those values. To find the values that should be used, issue the command:
/usr/local/src/snort-2.8.4.1 root# /usr/local/bin/dnet-config --libs -L/usr/local/lib -ldnet /usr/local/src/snort-2.8.4.1 root# /usr/local/bin/dnet-config --cflags -I/usr/local/include |
In the above case, you should change all occurrences in the configure file from having:
LIBDNET="`dumbnet-config --libs`" DNETFLAGS="`dumbnet-config --cflags`" |
to (depending on the results of the above dnet-config results):
LIBDNET="-L/usr/local/lib -ldnet" DNETFLAGS="-I/usr/local/include" |
The post by Martin Roesch, suggest a replacement for the function IpListEval. You may want to replace that in file spp_iplist.c.
/usr/local/src/snort-2.8.4.1 root# vi ./src/preprocessors/spp_iplist.c |
We are going to add in support to place alerts into a MySQL database. If MYSQL is installed on the system, you can use the “–with-mysql” configuration option to specify where. In a previous post, “Introduction to MySQL,” we went through the installation of MySQL into the /usr/local/mysql directory. For such an installation, the –with-mysql-includes=/usr/local/mysql/include and –with-mysql-libraries=/usr/local/mysql/lib command options must be used. In order to use the dynamic plugin libraries, Snort needs to be able to find libmysqlclient.so. On some operating systems, you may have problems. Adding LDFLAGS=”-L/usr/local/mysql/lib/mysql” should work.
You may want to consider configuring Snort to allow decoder and preprocessor rule eventing. This allows you to enable and disable decoder and preprocessor events on a rule by rule bases. It also allow you to specify the rule type or action of a decoder or preprocessor event on a rule by rule basis. Enable this configuration option with the configuration option using –enable-iplist.
We will also be adding in large file support. To allow enable blacklist files, we must include the configuration option –enable-iplist. If you had to install libdnet in a special location, you will need to specify that location with the “–with-dnet-includes=” and “–with-dnet-libraries=” configuration options.
We will configure Snort with the following command:
/usr/local/src/snort-2.8.4.1 root# CFLAGS="-D_LARGEFILE_SOURCE \ -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64" \ ./configure --prefix=/usr/local/snort --with-libpcap-includes=/usr/local/snort/include \ --with-libpcap-libraries=/usr/local/snort/lib \ --with-libpcre-includes=/usr/local/pcre/include \ --with-libpcre-libraries=/usr/local/pcre/lib \ --with-mysql-includes=/usr/local/mysql/include\ --with-mysql-libraries=/usr/local/mysql/lib \ --enable-iplist \ --enable-decoder-preprocessor-rules |
After you configure Snort, you continue to make and install it.
/usr/local/src/snort-2.8.4.1 root# make /usr/local/src/snort-2.8.4.1 root# make check /usr/local/src/snort-2.8.4.1 root# make install /usr/local/src/snort-2.8.4.1 root# mkdir -p /usr/local/snort/etc /usr/local/src/snort-2.8.4.1 root# cp etc/* /usr/local/snort/etc /usr/local/src/snort-2.8.4.1 root# mkdir -p /usr/local/snort/preproc_rules /usr/local/src/snort-2.8.4.1 root# cp preproc_rules/*.rules /usr/local/snort/preproc_rules /usr/local/src/snort-2.8.4.1 root# /usr/local/snort/bin/snort -V ,,_ -*> Snort! <*- o" )~ Version 2.8.4.1 (Build 38) '''' By Martin Roesch & The Snort Team: http://www.snort.org/team.html Copyright (C) 1998-2009 Sourcefire, Inc., et al. Using PCRE version: 7.8 2008-09-05 |
You may need to move the lib_sfdynamic_example_rule.so, if you receive errors. Just issue the command:
/usr/local/src/snort-2.8.4.1 root# mv /usr/local/snort/lib/snort_dynamicrules/\ lib_sfdynamic_example_rule.so \ /usr/local/snort/lib/snort_dynamicrules/lib_sfdynamic_example_rule.so.broke |
Rules
Now we need some rules. For this example we will get the rules from the Snort and the Emerging Threats site. You will need to register for the rules at the Snort site. Do consider subscribing for the latest up-to-date rules. Registered users can only access rules 30 days after their release.
root# cd /usr/local/snort/rules /usr/local/snort/rules root# wget http://www.emergingthreats.net/rules/emerging-all.rules /usr/local/snort/rules root# cd /usr/local/src /usr/local/src root# wget \ http://www.snort.org/pub-bin/downloads.cgi/\ Download/vrt_os/snortrules-snapshot-CURRENT.tar.gz /usr/local/src root# md5sum snortrules-snapshot-CURRENT.tar.gz 184aed405da3f1043b82d81c98122237 snortrules-snapshot-CURRENT.tar.gz /usr/local/src root# mv snortrules-snapshot-CURRENT.tar.gz /usr/local/snort/ /usr/local/src root# cd /usr/local/snort/ /usr/local/snort root# tar xzf snortrules-snapshot-CURRENT.tar.gz /usr/local/snort root# rm snortrules-snapshot-CURRENT.tar.gz /usr/local/snort root# vi /usr/local/snort/etc/snort.conf |
Modify /usr/local/snort/etc/snort.conf to your environment. Make sure the RULE_PATH is set to /usr/local/snort/rules. If you configured Snort to enable decoder and preprocessor rules, you will need to add a line specifying the location of those files. Define PREPROC_RULE_PATH with the line:
var PREPROC_RULE_PATH ../preproc_rules |
Later in the snort.conf file include the lines (before other rule lists are included):
include $PREPROC_RULE_PATH/preprocessor.rules include $PREPROC_RULE_PATH/decoder.rules |
If you wish to use the emerging threat rules, add:
include $RULE_PATH/emerging-all.rules |
in the /usr/local/snort/etc/snort.conf file. Do not forget to adjust dynamicpreprocessor file and dynamicengine path. Mac OS X users will need to use the dynamic libraries. Uncomment the Mac OS X lines in the Snort configuration file.
Dumbpig
Leon Ward has released a Perl program, Dumbpig, which will check Snort rules for badly formatted entires and incorrect usage. He has even added blacklist support (see posting “ET RBN Blacklists with Snort and DumbPig“). To pull down dumbpig.pl, the required Perl modules, and run it against the Emerging Threats rule set:
root# cd /home/snort/perl
/home/snort/perl root# wget rm-rf.co.uk/downloads/dumbpig.pl
/home/snort/perl root# chmod u+x ./dumbpig.pl
/home/snort/perl root# cpan -e "Parse::snort"
/home/snort/perl root# cpan -e "LWP::Simple"
/home/snort/perl root# ./dumbpig.pl -r /usr/local/snort/rules/emerging-all.rules
DumbPig version 0.5 - leon.ward@sourcefire.com
Because I hate looking for the same dumb problems with snort rule-sets
__,, ( Dumb-pig says )
~( oo ---( "ur rulz r not so )
'''' ( gud akshuly" * )
Config
----------------------
* Sensivity level - 3/3
* Blacklist outputi : Disabled
* Processing File - /home/snort/rules//emerging-all.rules
* Check commented out rules : Disabled
* Pause : Disbled
* ForceFail : Disabled
* Censor : Disabled
* Quite mode : Disabled
----------------------
Issue 1
1 Problem(s) found with rule on line 59 of /home/snort/rules//emerging-all.rules
alert tcp $HOME_NET any -> \
[75.126.138.202,72.249.118.38,208.78.69.70,204.13.249.70,208.78.68.70] $HTTP_PORTS ( \
msg:"ET CURRENT_EVENTS Possible Downadup/Conficker-A Infection Checking Geographical Location"; \
flow:to_server; \
classtype:trojan-activity; \
reference:url,www.microsoft.com/security/portal/Entry.aspx?Name=Worm%3aWin32%2fConficker.A; \
reference:url,www.f-secure.com/v-descs/worm_w32_downadup_a.shtml; \
threshold:type both, count 5, seconds 60, track by_src; \
reference:url,doc.emergingthreats.net/bin/view/Main/2008803; \
reference:url,www.emergingthreats.net/cgi-bin/cvsweb.cgi/sigs/CURRENT_EVENTS/CURRENT_Conficker; \
sid:2008803; \
rev:3; \
)
- TCP/UDP rule with no deep packet checks? This rule looks more suited to a firewall or blacklist
alert tcp $HOME_NET any ->
[75.126.138.202,72.249.118.38,208.78.69.70,204.13.249.70,208.78.68.70] $HTTP_PORTS (msg:"ET CURRENT_
EVENTS Possible Downadup/Conficker-A Infection Checking Geographical
Location"; flow:to_server; classtype:trojan-activity;
reference:url,www.microsoft.com/security/portal/Entry.aspx?Name=Worm%3aWin32%2fConficker.A;
reference:url,www.f-secure.com/v-descs/worm_w32_downadup_a.shtml;
threshold:type both, count 5, seconds 60, track by_src;
reference:url,doc.emergingthreats.net/bin/view/Main/2008803;
reference:url,www.emergingthreats.net/cgi-bin/cvsweb.cgi/sigs/CURRENT_EVENTS/CURRENT_Conficker;
sid:2008803; rev:3;)
|
There are many more issues reported. I included the first issue, because it brings up the point that rules can be rewritten to use blacklists. In Leon’s post, he found rewriting the Russian Business Network rules dropped processing time by over 40 percent. The Dumbpig program, along with the blacklist patch, can help tremendously with performance.
Configuration File
In the /usr/local/snort/etc/snort.conf file, make sure to add iplist preprocessor command. For example, if we are going to combine the three separate source for blacklists into one blacklist file, we might want to do:
preprocessor iplist: blacklist watchlist /usr/local/snort/conf/combo.blacklist \ whitelist /usr/local/snort/conf/default.whitelist |
Testing Snort Using Attack Data
Whenever installing a new version of Snort, it is a good idea to test it. Leon Ward has made available a pcap file containing attacks that occurred back in 2001 against a honeypot. If you have other data that will produce interesting results, please feel free to use that.
root# mkdir -p /data/ids/tcpdump root# cd /data/ids/tcpdump /data/ids/tcpdump root# wget http://rm-rf.co.uk/downloads/Honeynet-RFP-iis.tgz /data/ids/tcpdump root# tar xzf Honeynet-RFP-iis.tgz /data/ids/tcpdump root# /usr/local/snort/bin/snort -c /usr/local/snort/etc/snort.conf \ -A fast -l /data/ids/tcpdump -r ./Honeynet-RFP-iis.pcap /data/ids/tcpdump root# ls /data/ids/tcpdump/alert /data/ids/tcpdump/snort.log.* |
Two results files should get created. The file /data/ids/tcpdump/alert will contain the alerts and /data/ids/tcpdump/snort.log.<date>, which contains the pcaps of the detected events.
Blacklists
We will pull down upates from DNS-BH Malware Domain Blocklist, the Global Watchlist, and Ninja Chimp Strike Force Blacklist. The file format for the DNS-BH blocklist is:
# domain type original_reference-why_it_was_listed dateadded seqnum note--pound sign=comment 1 # notice notice duplication is not permitted 2 00.devoid.us malware www.cyber-ta.org/releases/malware-analysis/public/SOURCES/DNS.Cumulative.Summary 20090321 3 032439.com malware www.malwaredomainlist.com 20080822 9 0503.pass.as malware www.threatexpert.com/report.aspx?md5=21d60d5e9b8c9353b1b55994dfa1b11e 20080503 11 |
The Global Watchlist uses the file format:
# watchlist.security.org.my, contact mel@hackinthebox.org # ip/net, source, comment, name, last update (GMT+8) 10.50.50.50, http://www.dshield.org/ipsascii.html, Dshield: Top IPs, dshield-top-ips, 2009/07/20 00:00:11 61.184.255.175, http://www.dshield.org/ipsascii.html, Dshield: Top IPs, dshield-top-ips, 2009/07/20 00:00:11 222.82.249.235, http://www.dshield.org/ipsascii.html, Dshield: Top IPs, dshield-top-ips, 2009/07/20 00:00:11 |
Ninja Chimp uses the file format:
# This is a compiled list of dirty hosts associated # with bruteforcing attempts, spam, botnets, etc. # The list is comprised of data from Arbor Networks, # Project Honeypot, Shadowserver, and about 24+ hosts # I maintain. It is sorted on an hourly basis to keep # information current and is consistently changing Sun Jul 19 12:59:03 CDT 2009 99.254.50.139 99.248.26.177 |
In order to track the data and be able to generate various reports, we will take the data and place it into a databases. The database schema can be fairly simple. The “ip” feild represents the ip or network address. We will add an “end_ip” to reduce calculation time and allow use to search if an IP appears within ip/cidr.
mysql> describe watchlist;
+------------+------------------+------+-----+---------------------+-------+
| Field | Type | Null | Key | Default | Extra |
+------------+------------------+------+-----+---------------------+-------+
| ip | int(10) unsigned | NO | PRI | 0 | |
| cidr | int(2) unsigned | NO | | 0 | |
| source | varchar(100) | YES | | NULL | |
| comment | varchar(50) | YES | | NULL | |
| name | varchar(50) | YES | | NULL | |
| lastupdate | datetime | YES | | 0000-00-00 00:00:00 | |
| domain | varchar(50) | YES | | NULL | |
| reported | datetime | YES | | 0000-00-00 00:00:00 | |
| active | enum('yes','no') | YES | | no | |
| end_ip | int(10) unsigned | YES | MUL | 0 | |
+------------+------------------+------+-----+---------------------+-------+
|
We will create two Perl programs. The first program to pull the data down from the websites and the second program parses the files and load the information into the tables. The different blacklist sites are pulling information from the same sources. Overlap is expected. Each IP and network will get only one entry into the database and be listed once in the Snort blacklist file. The first program to pull data down from the sites might look like:
#!/usr/local/bin/perl -w
use LWP::UserAgent;
sub readpage {
my($url) = @_;
my $ua = new LWP::UserAgent;
# Go out and retrieve page
my $req = new HTTP::Request('GET', $url);
my $res = $ua->request($req);
# Check if the requested webpage is there and return results
if ($res->is_success) { # Request successful
return(1,$res->content)
}
else {
return(0,"");
}
}
sub pulldate {
my($url,$inputfn) = @_;
my ($pjstatus,$page) = &readpage($url);
if ($pjstatus) {
open(OUTFILE, ">$inputfn") || die "ERROR: Can't open $inputfn: $!\n";
print OUTFILE $page;
close(OUTFILE);
}
return($pjstatus);
}
# Main
my $base_dir = "/home/snort/projects/blacklists";
my $datadir = $base_dir . "/data/";
chdir $datadir || die "ERROR: Data directory $datadir does not exist: $!\n";
&pulldate("http://watchlist.security.org.my/all.txt",$datadir . 'watchlist.dat');
&pulldate("http://www.malwaredomains.com/files/domains.txt",$datadir . 'domains.dat');
&pulldate("http://www.infiltrated.net/blacklisted",$datadir . 'blacklisted.dat');
|
The second Perl program to parse the blacklists, is a bit longer. Initially I planned in including it, but this post is already quite long. I will leave the program to the reader’s creative mind. You know the format of the data sources and you know the database. It is not difficult to write a program to parse the files and place the data into the database. Just make sure to check if the IP/network is already in the watchlist table. If there is already an entry, only update the lastupdate feild. You will use that to track IPs and networks that may be dropped from the blacklist files.
You should note that the various blacklist will report the IPs and domains differently. The DNS-BH blocklist provides the domainname. You may want to convert that into IP/CIDR notation. Ninja Chimp blacklist contains only a list of IPs. The Global watchlist will provide IPs or networks. You may choose to use the Net::DNS and/or the Geo::IP Perl modules, depending on what information you want to store. If you use these modules, you should also be aware that such operations may end up warning the hostel domains that your site is trying to lookup information on those IPs. There is always the option to tunnel the request to an an IP that will not resolve back to your organization.
Since I like to gather information on the IPs and networks for use in evaluating the potential threat level, I keep the following information for each IP/network:
mysql> describe watchips; +--------------+------------------+------+-----+---------------------+-------+ | Field | Type | Null | Key | Default | Extra | +--------------+------------------+------+-----+---------------------+-------+ | ip | int(10) unsigned | NO | PRI | 0 | | | hostname | varchar(50) | YES | | NULL | | | entered | datetime | YES | | 0000-00-00 00:00:00 | | | lastseen | datetime | YES | | 0000-00-00 00:00:00 | | | asn | int(5) unsigned | NO | MUL | 0 | | | asn_comp | varchar(50) | YES | | NULL | | | asn_desc | varchar(75) | YES | | NULL | | | network | int(10) unsigned | NO | | 0 | | | cidr | int(2) unsigned | YES | | NULL | | | ccode | varchar(2) | NO | MUL | NULL | | | rir | varchar(10) | YES | | NULL | | | rir_moddate | date | YES | | 0000-00-00 | | | ccode3 | varchar(2) | YES | | NULL | | | country_name | varchar(50) | YES | | NULL | | | region | varchar(25) | YES | | NULL | | | region_name | varchar(50) | YES | | NULL | | | city | varchar(25) | YES | | NULL | | | postal_code | varchar(25) | YES | | NULL | | | latitude | decimal(7,4) | NO | | 0.0000 | | | longitude | decimal(7,4) | NO | | 0.0000 | | | time_zone | varchar(25) | YES | | NULL | | | area_code | varchar(5) | YES | | NULL | | | continent | varchar(2) | YES | | NULL | | | metro_code | int(3) unsigned | NO | | 0 | | +--------------+------------------+------+-----+---------------------+-------+ |
Creating Blacklist Files for Snort
The files Snort uses for blacklists are specified in the snort.conf file (mentioned above). The format for these files are:
# This is a blacklist file, there are many like it but this one is mine # Comments are supported #10.1.1.0/24 192.168.0.0/16 # I can do inline comments too and put # multiple CIDR blocks on one line 10.50.50.50/32 |
A Perl program to pull the data from the tables and generate the blacklist files in the format Snort expects can be as simple as:
#!/usr/local/bin/perl -w
use DBI;
my $base_dir = "/usr/local/snort/conf";
my $snortfile = $base_dir . "/combo.blacklist";
my $db = "badips";
my $mysql_user = "secretpig";
my $mysql_passwd = "secretpigpassword";
my $db_host = 'localhost';
my $results = "";
local($dbh) = DBI->connect("DBI:mysql:$db:$db_host",
$mysql_user, $mysql_passwd) || die "ERROR: Connecting: $DBI::errstr\n";
my $start_time = `/bin/date -d '7 day ago' +"%Y-%m-%d %H:%M:%S"`;
chomp $start_time;
my $sql = qq{ SELECT inet_ntoa(ip), cidr, comment, source
FROM watchlist WHERE lastupdate >= ? AND active=? };
$sth = $dbh->prepare( $sql );
$rc = $sth->execute($start_time,"YES");
if ($rc) {
while (my($ip,$cidr,$ocomment,$source) = $sth->fetchrow_array()) {
my $comment = "$source";
if ($ocomment ne "") {
$comment = $ocomment . " ($source)";
}
$results .= sprintf "%-20s # $comment\n","$ip/$cidr";
}
}
open(OUTFILE,">$snortfile");
print OUTFILE $results;
close(OUTFILE);
exit;
|
Final Thoughts
The above program will generate a large number of blacklisted IPs and domains. It might be helpful to add a process that helps evaluates the threat a little further. This was the motivation behind the work done by Jian Zhang (SRI), Phillip Porras (SRI), and Johannes Ullrich (SANS Institute). They attempted to develop a highly predictive blacklist (HPB). The technology is similar to Google’s PageRank algorithm using a multiphase approach which uses a relevance ranking and severity assessment to produce blacklists potentially unique for each organization or organization type participating in the HPB process.
While I do believe in tapping into the collaborative intelligence of the security community, one needs to verify the integrity of the information before it can be trusted. It is ill advised to automatically block IPs or networks listed in any blacklist unless your organization can accept the consequences. This holds true for commercial products as well. For example, image your organization HQ’s IP, through an accident or hackers, ending up on the blacklist. That is a large part of the motivation behind the work I have been doing with the Threat Observation, Tracking, and Evaluation Model (TOTEM). While I encourage organizations to take advantage of such options as sharing their organization’s information with DShield and receiving tailored HPBs, always verify before taking actions. Develop some form of an additional assessment method within your organization. Good security solutions involve layers. Blacklists can be a very useful tool in an organization’s security arsenal.
[...] IDS 1.4″ [3], “Snort 3: The Next Generation” [4], “Blacklisting with Snort”, [17], “IDS/IPS: The Mark Twain of the Security World” [5], and “IDS” [6]). I [...]