Recent Controversy

There has been a bit of a verbal flare up between the folks behind Snort and Suricata. Matt Jonkman, founder of Emerging Threats and OISF’s president, recent statement that “[Intrusion detection technology] has been stagnant for the last five years” [7] did not sit well with Martin Roesch, Snort’s creator. Roesch questioned Suricata’s stated benefits when he responded, “OISF has wrapped Suricata in some cool computer science concepts, but they have not delivered on their vision. [Suricata] offers a sub-set of Snort’s functionality at a fraction of its performance.” [8] Roesch went on to say, “They’ve produced a clone of Snort that performs worse at taxpayer’s expense.”

Matt Olney, a senior research engineer for the Sourcefire Vulnerability Research Team (VRT) addressed the multi-threaded benefit when he wrote, “Trust me, if multi-threading were the answer, the industry would have moved there in short order.” [10] Olney went on to quote results of an internal test pitting Snort against Suricata, “With rules loaded, Suricata runs up to about 200MB per second. Snort, with rules, hits 894MB per second with no drops.”

Jonkman questioned those finding when he wrote, “Those stats are ridiculous, and they refuse to publish details of the equipment and configuration used.” Jonkman goes on to explain, “We know that we’re not, right now, cycle for cycle, faster than Snort … but we’re getting six times the performance as Snort on the same hardware, with version 1.0.” Victor Julien, lead developer of Suricata, explain [14], “Is Suricata faster than Snort on a single core cycle for cycle, tick for tick? No. It’s pretty clear we aren’t, I didn’t expect us to be either. But we scale. We’ve had reports of running on a 32 core box and scaling to use all cores.” Russ McRee, a senior security analyst / researcher and founder of holisticinfosec.org, adds, “Consider that an unnamed military body has tested Suricata versus Snort on a large scale platform (24 processors and 128GB of RAM) and saw a very clear 6-fold speed increase over a tuned Snort implementation on the same platform.”

Features

Russ McRee article on Suricata in August’s ISSA Journal [18] contained a table comparing features, which we will add Bro information to:

Seth Hall, Information Security Detection-Response Architect at GE (and one of the top Bro developers), addressed the above table and pointed out some of the strong features of Bro. Seth writes:

• Multithreaded processing: Work is ongoing on this, but nothing releasable yet. Bro does have a fully functional cluster deployment model which helps users to scale support on a single box and/or across multiple boxes.
• IPv6 Support: Due to a bug, which hopefully will be addressed by the next release, IPv6 support is unusable in large scale production.
• IP reputation: You could say that Bro has IP reputation, it’s easy to utilize lists of addresses at least. I’m going to be working heavily on an intelligence sources framework for Bro soon too which will be able to consume a wide range of intelligence sources including IP addresses.
• Automated protocol detection: There’s even an academic paper about it [19] if you’d like to find out exactly how it works.
• Global variables/flowbits: Bro support for this sort of thing is far beyond what anything else has inherently because Bro has a complete programming language.
• GeoIP lookups: I added that myself several years ago. Bro supports IPv6 geoip lookups in addition to IPv4 and ASN lookups using another database for libGeoIP.
• Advanced HTTP Parsing: Bro has had it for years.
• HTTP Access Logging: Definitely. My script [20] will be included in the next release too.
• SMB Access Logging: This is something that I’m planning on tackling soon. I don’t know what the level of support for SMB is currently, but there is a parser already.
• HTTP Blocklist lookups: Yes, I consider this similar to the IP reputation and it’s going to be included in the intelligence sources framework. Some usage of URL lists is already included in a script that I distribute separately [21] but which will be in the next release of Bro.
• Free: Bro is under the BSD license, so in my opinion it’s actually more free than Snort or Suricata which are both under the GPL and much more difficult to share code with.

While Snort and Suricata have been a bit in the public spotlight recently, the developers of Bro have stayed clear of the recent verbal debate. Bro is the third open source IDS/IPS engine we will be working with. It is primarily funded by the National Science Foundation’s Strategic Technologies for the Internet program. Robin Sommer this week announced [26] that the International Computer Science Institute (ICSI) and the National Center for Supercomputing Applications (NCSA) have been awarded a grant of almost $3M for extensive Bro development. To quote Robin:

The funded project aims specifically at addressing much of the feedback that we have received from Bro users over the years. It will enable us to refine many of the rough edges that the system has accumulated over time[*], improve Bro’s performance significantly, and also make it much easier for the community to contribute to the project.

Expect some interesting work from the Bro camp. Some of Bro’s current stated features and benefits [15] include:

• Network Based: Bro is a network-based IDS. It collects, filters, and analyzes traffic that passes through a specific network location. A single Bro monitor, strategically placed at a key network junction, can be used to monitor all incoming and outgoing traffic for the entire site. Bro does not use or require installation of client software on each individual, networked computer.
• Rich Application-Layer Analysis: A primary feature of Bro is that it includes detailed, parser-driven analysis of many popular application protocols. The output of these analyzers is a stream of events that describe observed activity in semantically rich, high-level terms. These events themselves do not constitute security alerts, but rather provide the input for further, stateful processing using Bro’s custom scripting language.
• Custom Scripting Language: Bro policy scripts are programs written in the Bro language. They contain the “rules” that describe what sorts of activities are deemed troublesome. They analyze the network activity and initiate actions based on the analysis. Although the Bro language takes some time and effort to learn, once mastered, the Bro user can write or modify Bro policies to detect and alert on virtually any type of network activity.
• Pre-written Policy Scripts: Bro comes with a rich set of policy scripts designed to detect the most common Internet attacks while limiting the number of false positives, i.e., alerts that confuse uninteresting activity with the important attack activity. These supplied policy scripts will run “out of the box” and do not require knowledge of the Bro language or policy script mechanics.
• Powerful Signature Matching Facility: Bro policies incorporate a signature matching facility that looks for specific traffic content. For Bro, these signatures are expressed as regular expressions, rather than fixed strings. Bro adds a great deal of power to its signature-matching capability because of its rich language. This allows Bro to not only examine the network content, but to understand the context of the signature, greatly reducing the number of false positives. Bro comes with a set of high value signatures policies, selected for their high detection and low false positive characteristics.
• Network Traffic Analysis: Bro not only looks for signatures, but can also analyze network protocols, connections, transactions, data amounts, and many other network characteristics. It has powerful facilities for storing information about past activity and incorporating it into analyses of new activity.
• Detection Followed by Action: Bro policy scripts can generate output files recording the activity seen on the network (including normal, non-attack activity). They can also generate problem alerts to event logs, including the operating system syslog facility. In addition, scripts can execute programs, which can, in turn, send e-mail messages, page the on-call staff, automatically terminate existing connections, or, with appropriate additional software, insert access control blocks into a router’s access control list. With Bro’s ability to execute programs at the operating system level, the actions that Bro can initiate are only limited by the computer and network capabilities that support Bro.

In my previous post, “Snort 3: The Next Generation” [4], Marty provided a roadmap of where Snort is heading. While changes have since been made as Snort develops, the philosophy remains the same. Sourcefire is moving forward on a solid security framework. See the recent work on the Sourcefire’s Razorback framework [24] and [25]. Olney described Razorback in this way [10], “It isn’t Snort, it isn’t ClamAV, and it isn’t Suricata. It’s a new approach to the detection problem, and was built from the ground up in close collaboration with groups that are facing APT-level threats. It may not be perfect, it may not even be the right answer (but we think it is), but it is truly innovative.”

More immediate, examine the the improved features Snort 2.9.0 Beta [16]:

• Updates to HTTP Inspect to extract and log IP addresses from X-Forward-For and True-Client-IP header fields when Snort generates events on HTTP traffic.
• Updates to SMTP preprocessor to support MIME attachment decoding across multiple packets.
• Updates to the Snort packet decoders for IPv6 for improvements to anomaly detection.

The new features include:

• Feature rich IPS mode including improvements to Stream for inline deployments. Additionally a common active response API is used for all packet responses, including those from Stream, Respond, or React. A new response module, respond3, supports the syntax of both resp & resp2, including strafing for passive deployments. When Snort is deployed inline, a new preprocessor
  has been added to handle packet normalization to allow Snort to interpret a packet the same way as the receiving host.
• Use of a Data Acquisition API (DAQ) that supports many different packet access methods including libpcap, netfilterq, IPFW, and afpacket. For libpcap, version 1.0 or higher is now required. The DAQ library can be updated independently from Snort and is a separate module that Snort links to.
• A new rule option ‘byte_extract’ that allows extracted values to be used in subsequent rule options for isdataat, byte_test, byte_jump, and content distance/within/depth/offset.
• Two new rule options to support base64 decoding of certain pieces of data and inspection of the base64 data via subsequent rule options.
• Added a new pattern matcher that supports Intel’s Quick Assist Technology for improved performance on supported hardware platforms. Visit http://www.intel.com to find out more about Intel Quick Assist.

Last, but not least, several characteristics OSIF report Suricata has to handle today’s threat are [12]:

• An open source engine. The power of the community works well
  within IT security defenses, as a community is more effective than a
  single organization at capturing characteristics of emerging threats.
• Multi-threaded. A multi-threaded architecture allows the engine
  to take advantage of the multiple core and multiple processor
  architectures of today’s systems.
• Supports IP reputation. By incorporating reputation and
  signatures into its engine, Suricata can flag traffic from known
  nefarious origins.
• Automated protocol detection. Preprocessors automatically
  identify the protocol used in a network stream and apply the
  appropriate rules, regardless of numerical port.

For additional background information, I have written several past posts on IDS/IPS (see “Suricata: A Next Generation IDS/IPS Engine” [2], “Installing Bro IDS 1.4″ [3], “Snort 3: The Next Generation” [4], “Blacklisting with Snort”, [17], “IDS/IPS: The Mark Twain of the Security World” [5], and “IDS” [6]). I will not repeat that information in this post.

Version

We will be setting these software packages up to be used on a development machine. The exchange between the Snort and Suricata was focused on timing and features. For that reason, I am interested in maximizing features verses stability. We will be using the latest software, which may mean beta or even CVS versions.

Supporting Software

The three IDS/IPS engines share most of the supporting software requirements, depending on configuration options. Below are a few required libraries and software packages:

Below are a few libraries and software packages that are not required, but you should consider installing. The packages, except GeoIP and Google Perftools, should have binaries available for your OS. Use these ports to install the packages and save yourself the trouble of having to keep the software updated. We will go through through the installation of GeoIP and Google Perftools from source code.

Installing Supporting Software

Which libraries and supporting software you install will be dependent on which options you use in configuring your IDS/IPS engines. If you can install packages (and not source), carefully consider this option. It will make maintenance easier. We will walk through the source installation for demonstration purposes.

Libcap-ng

For Linux users, the libcap-ng will be required for dropping privileges.

root# cd /usr/local/src
/usr/local/src root# wget http://people.redhat.com/sgrubb/libcap-ng/libcap-ng-0.6.4.tar.gz
/usr/local/src root# tar libcap-ng-0.6.4.tar.gz
/usr/local/src  root# cd libcap-ng-0.6.4
/usr/local/src/libcap-ng-0.6.4 root# ./configure
/usr/local/src/libcap-ng-0.6.4 root# make
/usr/local/src/libcap-ng-0.6.4 root# make install

Libdnet

Make sure Libdnet is in your library path:

 root# /sbin/ldconfig -p | grep -i libdnet
libdnet32.so.1 (libc6) => /usr/lib/libdnet32.so.1
libdnet32.so (libc6) => /usr/lib/libdnet32.so

If you do not get a path returned, you will need to install libdnet (use –prefix if it needs to be installed in a special location). We will pull down it down the CVS version, because we will need the sctp.h file to be installed.

root# cd /usr/local/src
/usr/local/src root# wget svn checkout http://libdnet.googlecode.com/svn/trunk/ libdnet-cvs
/usr/local/src root# cd libdnet-cvs
/usr/local/src/libdnet-cvs root# ./configure
/usr/local/src/libdnet-cvs root# make
/usr/local/src/libdnet-cvs root# make install
/usr/local/src/libdnet-cvs root# cp include/dnet/sctp.h /usr/local/include/dnet

If you have installed libdnet in a special location, make sure to include its path in /etc/ld.so.conf.

Libnet

The library libnet will be required for packet-injecting.

root# cd /usr/local/src
/usr/local/src root# wget http://github.com/sam-github/libnet/tarball/libnet-1.1.4 \
-O  libnet-1.1.4.tgz
/usr/local/src root# tar xzf libnet-1.1.4.tgz
/usr/local/src root# cd sam-github-libnet-d2bedb5
/usr/local/src/sam-github-libnet-d2bedb5 root# ./autogen.sh
/usr/local/src/sam-github-libnet-d2bedb5 root# ./configure
/usr/local/src/sam-github-libnet-d2bedb5 root# make
/usr/local/src/sam-github-libnet-d2bedb5 root# make install

Libnfnetlink and Libnetfilter

If you plan on using the IPS capabilities (inline support), you will need to install libnfnetlink and libnfnetlink-queue.

root# cd /usr/local/src
/usr/local/src root# wget \
ftp://ftp.netfilter.org/pub/libnfnetlink/snapshot/libnfnetlink-20100823.tar.bz2
/usr/local/src root# bunzip2 libnfnetlink-20100823.tar.bz2
/usr/local/src root# tar xf libnfnetlink-20100823.tar
/usr/local/src root# cd libnfnetlink-20100823
/usr/local/src/libnfnetlink-20100823 root# ./autogen.sh
/usr/local/src/libnfnetlink-20100823 root# ./configure
/usr/local/src/libnfnetlink-20100823 root# make
/usr/local/src/libnfnetlink-20100823 root# make check
/usr/local/src/libnfnetlink-20100823 root# make install
/usr/local/src/libnfnetlink-20100823 root# cd /usr/local/src
/usr/local/src root# wget \
ftp://ftp.netfilter.org/pub/libnetfilter_queue/snapshot/libnetfilter_queue-20100824.tar.bz2
/usr/local/src root# md5sum libnetfilter_queue-20100824.tar.bz2
69ce1eb24632bfed050cd936e0fe660c  libnetfilter_queue-20100824.tar.bz2
/usr/local/src root# bunzip2 libnetfilter_queue-20100824.tar.bz2
/usr/local/src root# tar xf libnetfilter_queue-20100824.tar
/usr/local/src root# cd libnetfilter_queue-20100824
/usr/local/src/libnetfilter_queue-20100824 root# ./autogen.sh
/usr/local/src/libnetfilter_queue-20100824 root# PKG_CONFIG_PATH=/usr/local/lib/pkgconfig/ ./configure
/usr/local/src/libnetfilter_queue-20100824 root# make
/usr/local/src/libnetfilter_queue-20100824 root# make check
/usr/local/src/libnetfilter_queue-20100824 root# make install

Libpcap and PF_RING

If you are running on a system with a Linux kernels 2.6.x or greater, you will want to install PF_RING on your system. PF_RING is a network socket that can greatly improve packet capture speed. PF_RING polls packets from NICs by means of Linux NAPI. NAPI (“New API”) is a modification to the device driver packet processing framework, which is designed to improve the performance of high-speed networking. NAPI copies packets from the NIC to PF_RING circular buffer. The application then reads packets from the ring. PF_RING can distribute incoming packets to multiple rings (hence multiple applications) simultaneously. Please see “Exploiting Commodity Multicore Systems for Network Traffic Analysis” [23] for additional information.

We are going to walk through a specific example with certain ethernet card drivers and a particular linux kernel. Please make sure not to copy the commands blindly. Adjust to your system. The below is for demonstration purposes. These steps are based on Gunjan Bansal blog [22]. Please see Gunjan’s blog for additional explanation.

First step, provide some basic information on the system.

root# /sbin/ifconfig -a
eth0      Link encap:Ethernet  HWaddr 00:00:00:00:00:00
root# /sbin/ethtool -i eth0
driver: e1000e
version: 1.0.2-k3.1
firmware-version: 1.3-1
bus-info: 0000:00:19.0
root# /bin/uname -r
2.6.18-194.8.1.el5PAE

In this example I will be working off the ethernet interface eth0 and the ethernet driver e1000e. The kernel release is 2.6.18-194.8.1.el5PAE.

Second step is to download the PF_RING software from ntop through the SVN repository, configure, compile, and install. The “/sbin” directory will need to be in your PATH or you will get a complaint about “ldconfig: Command not found.”

root# cd /usr/local/src
root# PATH=$PATH:/sbin
/usr/local/src root# mkdir pf_ring && cd pf_ring
/usr/local/src/pf_ring root# svn co https://svn.ntop.org/svn/ntop/trunk/PF_RING/
/usr/local/src/pf_ring root# cd PF_RING/kernel
/usr/local/src/pf_ring/PF_RING/kernel root# make
/usr/local/src/pf_ring/PF_RING/kernel root# make install
/usr/local/src/pf_ring/PF_RING/kernel root# cd ../userland/lib
/usr/local/src/pf_ring/PF_RING/userland/lib root# make
/usr/local/src/pf_ring/PF_RING/userland/lib root# make install

Under some OSs, you need to compile libpcap to support large files (files large than 2G). We are going to install the resulting libpcap under /usr/local. Large file support is required if the following kind of error is produced:

root# ls -lh /data/ids/full2.pcap
-rw-r--r-- 1 root root 12G Oct 26 10:01 /data/ids/full2.pcap
root# /usr/local/snort/bin/snort -o -A none -c \
/usr/local/snort/conf/snort.conf -l /logs/snort/logs \
-r /data/ids/full2.pcap
Error getting stat on pcap file: /data/ids/full2.pcap:
Value too large for defined data type
ERROR: Error getting pcaps
Fatal Error, Quitting..

To compile large file support into libpcap:

/usr/local/src/pf_ring/PF_RING/userland/lib root# cd ../libpcap-1.0.0-ring/
/usr/local/src/pf_ring/userland/libpcap-1.0.0-ring root# ./configure --prefix=/usr/local \
CFLAGS="-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE \
-D_FILE_OFFSET_BITS=64"
/usr/local/src/pf_ring/userland/libpcap-1.0.0-ring root# make
/usr/local/src/pf_ring/userland/libpcap-1.0.0-ring root# make shared
/usr/local/src/pf_ring/userland/libpcap-1.0.0-ring root# make install
/usr/local/src/pf_ring/userland/libpcap-1.0.0-ring root# make install-shared

By configuring tcpdump with support for PF_RING, all applications (tcpdump and our IDS/IPS engines) will be able to access simultaneously the PF_RING circular buffer.

/usr/local/src/pf_ring/userland/libpcap-1.0.0-ring root# cd ../tcpdump-4.0.0
/usr/local/src/pf_ring/userland/tcpdump-4.0.0 root# CFLAGS="-D_LARGEFILE_SOURCE \
-D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64" LDFLAGS="-lpfring -lpcap" LD_LIBRARY_PATH="/usr/lib:/usr/local/lib" ./configure
/usr/local/src/pf_ring/userland/tcpdump-4.0.0 root# make
/usr/local/src/pf_ring/userland/tcpdump-4.0.0 root# make install

Replace the ethernet driver.

/usr/local/src/pf_ring/userland/tcpdump-4.0.0 root# cd ../../drivers/intel/e1000e-1.0.15/src
/usr/local/src/pf_ring/PF_RING/drivers/intel/e1000e-1.0.15/src root# make
/usr/local/src/pf_ring/PF_RING/drivers/intel/e1000e-1.0.15/src root# make install

The third step is to activate PF_RING if its not already activated. Use lsmod to check if pf_ring is started or not.

/usr/local/src/pf_ring/PF_RING/drivers/intel/e1000e-1.0.15/src root# /sbin/lsmod | grep pf_ring
pf_ring                46680  0
/usr/local/src/pf_ring/PF_RING/drivers/intel/e1000e-1.0.15/src root# cd \
/lib/modules/2.6.18-194.8.1.el5PAE/kernel/net/pf_ring
/lib/modules/2.6.18-194.8.1.el5PAE/kernel/net/pf_ring root# /sbin/insmod \
pf_ring.ko transparent_mode=1
/lib/modules/2.6.18-194.8.1.el5PAE/kernel/net/pf_ring root# cd \
../../drivers/net/e1000e

Step four, you will unload the ethernet card driver (e1000e) and load the new driver. Keep in mind, unloading the driver means ethernet access will be lost. It is wise not to issue this command remotely.

/lib/modules/2.6.18-194.8.1.el5PAE/kernel/drivers/net/e1000e root# /sbin/rmmod \
e1000e ; /sbin/insmod e1000e.ko

You now are PF_RING enabled.

LibYAML

The yaml library will be required for parsing Suricata configuration file.

root# cd /usr/local/src
/usr/local/src root# wget http://pyyaml.org/download/libyaml/yaml-0.1.3.tar.gz
/usr/local/src root# tar xzf yaml-0.1.3.tar.gz
/usr/local/src root# cd yaml-0.1.3
/usr/local/src/yaml-0.1.3 root# ./configure
/usr/local/src/yaml-0.1.3 root# make
/usr/local/src/yaml-0.1.3 root# make check
/usr/local/src/yaml-0.1.3 root# make install

GeoIP Installation and Configuration

MaxMind GeoIP is a collection of APIs for looking up the location of an IP address. There is a collection of free GeoLite databases, which are not as accurate as the GeoIP databases, but will do for starting out and testing with Bro. To setup GeoIP for use with Bro, please follow the commands below.

root# cd /usr/local/src
/usr/local/src root# wget http://www.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
/usr/local/src root# gunzip GeoLiteCity.dat.gz
/usr/local/src root# mkdir -p /usr/local/share/GeoIP
/usr/local/src root# mv GeoLiteCity.dat /usr/local/share/GeoIP/GeoIPCity.dat
/usr/local/src root# wget http://www.maxmind.com/download/geoip/api/c/GeoIP.tar.gz
/usr/local/src root# tar xzf GeoIP.tar.gz
/usr/local/src root# cd  GeoIP-1.4.6
/usr/local/src/GeoIP-1.4.6 root# ./configure
/usr/local/src/GeoIP-1.4.6 root# make
/usr/local/src/GeoIP-1.4.6 root# make check
/usr/local/src/GeoIP-1.4.6 root# make install

Make sure /usr/local/lib is placed into your library path.

Google Perftools Installation and Configuration

Google’s perftools is a collection of a high-performance multi-threaded malloc() implementation and some performance analysis tools. Google’s perftools have replaced mpatrol for leak-checking and heap-profiling. We will compile Bro with –enable-perftools. By default, perftools will install under /usr/local directory. With perftools compiled into Bro, there are two command-line options made available:

To help with the installation of Google’s perftool, the ICSI Networking Group has written a post “Making Sure Your Bro Code Does Not Leak.” The post will provide additional information. The basic steps to install perftools are:

root# cd /usr/local/src
/usr/local/src root# wget http://google-perftools.googlecode.com/files/google-perftools-1.6.tar.gz
/usr/local/src root# tar xzf google-perftools-1.6.tar.gz
/usr/local/src root# cd google-perftools-1.6
/usr/local/src/google-perftools-1.6 root# ./configure
/usr/local/src/google-perftools-1.6 root# make
/usr/local/src/google-perftools-1.6 root# make check
/usr/local/src/google-perftools-1.6 root# make install

PCRE (pcre-8.10)

The PCRE library is a set of functions that implement regular expression pattern matching using the same syntax and semantics as Perl 5. If you can install PCRE via a binary specific to your operating system, that is the best way to install in order to avoid having to keep the software up-to-date. Below are the instructions for installing the software from source.

root# cd /usr/local/src
/usr/local/src root# wget \
http://downloads.sourceforge.net/project/pcre/pcre/8.10/pcre-8.10.tar.gz
/usr/local/src root# wget \
http://sourceforge.net/projects/pcre/files/pcre/8.10/pcre-8.10.tar.gz.sig/download
/usr/local/src root# gpg --verify pcre-8.10.tar.gz.sig pcre-8.10.tar.gz
/usr/local/src root# tar xzf pcre-8.10.tar.gz
/usr/local/src root# cd pcre-8.10
/usr/local/src/pcre-8.10 root# ./configure --prefix=/usr/local/pcre
/usr/local/src/pcre-8.10 root# make
/usr/local/src/pcre-8.10 root# make test
/usr/local/src/pcre-8.10 root# make install

XML Analyzer

The XML analyzer is highly-experimental code written by Tobias Kiesling. Installation of Xerces-C++ and XQilla is required to use the XML analyzer. The code allows you to be able to easily adjust analysis capabilities to specific XML data formats. Xerces-C++ is a validating XML parser written in a portable subset of C++. XQilla is an XQuery and XPath 2 library and command line utility written in C++.

root# cd /usr/local/src
/usr/local/src root# wget http://downloads.sourceforge.net/xqilla/XQilla-2.2.4.tar.gz
/usr/local/src root# wget http://mirror.its.uidaho.edu/pub/apache/xerces/c/3/sources/xerces-c-3.1.1.tar.gz
/usr/local/src root#  md5sum xerces-c-3.1.1.tar.gz
6a8ec45d83c8cfb1584c5a5345cb51ae  xerces-c-3.1.1.tar.gz
/usr/local/src root# tar xzf xerces-c-3.1.1.tar.gz
/usr/local/src root# tar xzf XQilla-2.2.4.tar.gz
/usr/local/src root# ln -s XQilla-2.2.4 xqilla
/usr/local/src root# cd  xerces-c-3.1.1
/usr/local/src/xerces-c-3.1.1 root# ./configure
/usr/local/src/xerces-c-3.1.1 root# make
/usr/local/src/xerces-c-3.1.1 root# make check
/usr/local/src/xerces-c-3.1.1 root# make install

With Xerces-C++, configure and install XQilla.

root# cd /usr/local/src/xqilla/
/usr/local/src/xqilla root# ./configure --with-xerces=/usr/local/src/xerces-c-3.1.1/
/usr/local/src/xqilla root# make
/usr/local/src/xqilla root# make install

Bro Setup

We will be working off the instructions previously posted in “Installing Bro IDS 1.4″ [3], just updating the material to reflect the requirements of the current Bro software. There a few options when installing Bro. Bro was not developed for the PHB. Advance security software provides the power to the user, with all the options to adapt it to your environment. To quote the Bro site, “Bro has been developed primarily as a research platform for intrusion detection and traffic analysis. It is not intended for someone seeking an ‘out of the box’ solution. Bro is designed for use by Unix experts who place a premium on the ability to extend an intrusion detection system with new functionality as needed, which can greatly aid with tracking evolving attacker techniques as well as inevitable changes to a site’s environment and security policy requirements.” With the Unix experts in mind, we will go through the steps involved to install both the stable and the development versions of Bro.

Current Stable Version

The current version should be the most stable. To install, follow these commands:

root# cd /usr/local/src
/usr/local/src root# wget ftp://bro-ids.org/bro-1.5-release.tar.gz
/usr/local/src root# tar xzf bro-1.5-release.tar.gz
/usr/local/src root# cd bro-1.5.1

The configuration and installations appears below.

Subversion Trunk

Reading the posts on the Bro mailing list, reveals that modifications have already been made to the current release. Fixes are being made continuously. These changes, while fixing problems, might introduce new problems. You do have the option of getting the most up-to-date code possible through the subversion repository. The Bro development team has made available two subparts of the repository: the trunk and development branches. The trunk is the main development head from which releases are made on a regular basis. It should be fairly stable with changes passing a regression suite to ensure the code do not break existing functionality. It is still considered experimental and not suitable for critical deployment. Below is how to download code from the trunk.

root# cd /usr/local/src
/usr/local/src root# mkdir bro-cvs
/usr/local/src/bro-cvs root# cd bro-cvs
/usr/local/src/bro-cvs root# svn checkout http://svn.icir.org/bro/trunk/bro
/usr/local/src/bro-cvs root# mv bro bro-1.5.1.cvs
/usr/local/src/bro-cvs root# cd bro-1.5.1.cvs
/usr/local/src/bro-cvs/bro-1.5.1.cvs root# ./autogen.sh

Robin’s Development Branch

The developers merge their work into the the Bro subversion trunk. Robin Sommer has a separate branch which contains experimental code for:

• the Bro Cluster framework
• NetFlow support (by Bernhard Ager)
• a BitTorrent analyzer (by Nadi Sarrar and Bernhard Ager)
• an XML analyzer (by Tobias Kiesling)
• Python bindings for Broccoli
• restructured logic for taking drop decisions via Bro’s notice framework (by Brian Tierney and Robin Sommer)
• a test-suite for Bro’s communication & serialization subsystems
• various tweaks and bugfixes

If you want the latest work done by Robin and others mentioned above, you can get access to the code with the following commands.

root# cd /usr/local/src
/usr/local/src root# mkdir bro-cvs
/usr/local/src root# cd bro-cvs
/usr/local/src/bro-cvs root# svn checkout http://svn.icir.org/bro/branches/robin/work
/usr/local/src/bro-cvs root# mv work bro-1.5.1.robin
/usr/local/src/bro-cvs root# cd bro-1.5.1.robin
/usr/local/src/bro-cvs/bro-1.5.1.robin root# ./autogen.sh

Configure and Install

Because of the various bug fixes and the additional features which add interesting options, we are going to step through installation of Robin’s branch. Please use the version of Bro appropriate for your operation.

root# cd /usr/local/src/bro-cvs/bro-1.5.1.robin
/usr/local/src/bro-cvs/bro-1.5.1.robin root# CFLAGS="-D_LARGEFILE_SOURCE \
-D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64" LDFLAGS="-lpfring -lpcap" \
LD_LIBRARY_PATH="/usr/lib:/usr/local/lib" ./configure --prefix=/usr/local/bro  \
--enable-debug --enable-perftools
/usr/local/src/bro-cvs/bro-1.5.1.robin root# make
/usr/local/src/bro-cvs/bro-1.5.1.robin root# make check
/usr/local/src/bro-cvs/bro-1.5.1.robin root# make install

If you run into any problems, go to back to the stable version of Bro and see if you can get it to compile. Then you may want to try the subversion trunk code.

Snort Setup

We will be following the direction posted previously in “Blacklisting with Snort″ [17]. Below we will get the software, verify, configure, and install the software under the /usr/local/snort area. Please adjust this to your environment. Reminder to Mac OS X and FreeBSD users, use the md5 command instead of md5sum.

 root# cd /usr/local/src
/usr/local/src root# wget wget http://www.snort.org/downloads/116 -O snort-2.8.6.1.tar.gz
/usr/local/src root# wget http://www.snort.org/downloads/116/show_md5
/usr/local/src root# cat show_md5
“b1119396a32e9df0d80404e4b6c49166”
/usr/local/src root# md5sum snort-2.8.6.1.tar.gz
b1119396a32e9df0d80404e4b6c49166  snort-2.8.6.1.tar.gz
/usr/local/src root# tar xzf snort-2.8.6.1.tar.gz
/usr/local/src root# cd snort-2.8.6.1

We are going to add in support to place alerts into a MySQL database. If MYSQL is installed on the system, you can use the “–with-mysql” configuration option to specify where. In a previous post, “Introduction to MySQL,” we went through the installation of MySQL into the /usr/local/mysql directory. For such an installation, the –with-mysql-includes=/usr/local/mysql/include and –with-mysql-libraries=/usr/local/mysql/lib command options must be used. In order to use the dynamic plugin libraries, Snort needs to be able to find libmysqlclient.so. On some operating systems, you may have problems. Adding LDFLAGS=”-L/usr/local/mysql/lib/mysql” should work.

You may want to consider configuring Snort to allow decoder and preprocessor rule eventing. This allows you to enable and disable decoder and preprocessor events on a rule by rule bases. It also allow you to specify the rule type or action of a decoder or preprocessor event on a rule by rule basis. Enable this configuration option with the configuration option using –enable-decoder-preprocessor-rules.

We will also be adding in large file support. If you had to install libdnet in a special location, you will need to specify that location with the “–with-dnet-includes=” and “–with-dnet-libraries=” configuration options.

We will configure Snort with the following command:

/usr/local/src/snort-2.8.6.1 root# CFLAGS="-D_LARGEFILE_SOURCE \
-D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64" LDFLAGS="-lpfring -lpcap" \
 LD_LIBRARY_PATH="/usr/lib:/usr/local/lib" \
./configure --prefix=/usr/local/snort --with-libpcap-includes=/usr/local/include \
--with-libpcap-libraries=/usr/local/lib \
--with-libpcre-includes=/usr/local/pcre/include \
--with-libpcre-libraries=/usr/local/pcre/lib \
--with-mysql-includes=/usr/local/mysql/include \
--with-mysql-libraries=/usr/local/mysql/lib \
--enable-decoder-preprocessor-rules --enable-zlib

Check config.log if you had any problems or just want to make sure Snort configured everything correctly. After you configure Snort, you continue to make and install it.

/usr/local/src/snort-2.8.6.1 root# make
/usr/local/src/snort-2.8.6.1 root# make check
/usr/local/src/snort-2.8.6.1 root# make install
/usr/local/src/snort-2.8.6.1 root# mkdir -p /usr/local/snort/etc
/usr/local/src/snort-2.8.6.1 root# cp etc/* /usr/local/snort/etc
/usr/local/src/snort-2.8.6.1 root# mkdir -p /usr/local/snort/preproc_rules
/usr/local/src/snort-2.8.6.1 root# cp preproc_rules/*.rules /usr/local/snort/preproc_rules
/usr/local/src/snort-2.8.6.1 root# /usr/local/snort/bin/snort -V

,,_ -*> Snort! <*-
o" )~ Version 2.8.6.1 (Build 39)
'''' By Martin Roesch & The Snort Team: http://www.snort.org/team.html
Copyright (C) 1998-2010 Sourcefire, Inc., et al.
Using PCRE version: 8.10 2010-06-25

Rules

Now we need some rules. For this example we will get the rules from the Snort and the Emerging Threats site. You will need to register for the rules at the Snort site. Do consider subscribing for the latest up-to-date rules. Registered users can only access rules 30 days after their release.

 root# cd /usr/local/snort/rules
/usr/local/snort/rules root# wget http://www.emergingthreats.net/rules/emerging-all.rules
/usr/local/snort/rules root# cd /usr/local/src
/usr/local/src root# wget \
https://www.snort.org/downloads/83 \
-O snortrules-snapshot-CURRENT.tar.gz
/usr/local/src root# md5sum snortrules-snapshot-CURRENT.tar.gz
/usr/local/src root# mv snortrules-snapshot-CURRENT.tar.gz /usr/local/snort/
/usr/local/src root# cd /usr/local/snort/
/usr/local/snort root# tar xzf snortrules-snapshot-CURRENT.tar.gz
/usr/local/snort root# rm snortrules-snapshot-CURRENT.tar.gz
/usr/local/snort root# vi /usr/local/snort/etc/snort.conf

Modify /usr/local/snort/etc/snort.conf to your environment. Make sure the RULE_PATH is set to /usr/local/snort/rules. If you configured Snort to enable decoder and preprocessor rules, you will need to add a line specifying the location of those files. Define PREPROC_RULE_PATH with the line:

 var PREPROC_RULE_PATH ../preproc_rules

Later in the snort.conf file include the lines (before other rule lists are included):

 include $PREPROC_RULE_PATH/preprocessor.rules
include $PREPROC_RULE_PATH/decoder.rules

If you wish to use the emerging threat rules, add:

 include $RULE_PATH/emerging-all.rules

in the /usr/local/snort/etc/snort.conf file. Do not forget to adjust dynamicpreprocessor file and dynamicengine path. Mac OS X users will need to use the dynamic libraries. Uncomment the Mac OS X lines in the Snort configuration file.

Dumbpig

Leon Ward has released a Perl program, Dumbpig, which will check Snort rules for badly formatted entries and incorrect usage. He has even added blacklist support (see posting “ET RBN Blacklists with Snort and DumbPig“). To pull down dumbpig.pl, the required Perl modules, and run it against the Emerging Threats rule set:

 root# cd /home/snort/perl
/home/snort/perl root# wget http://dumbpig.googlecode.com/files/dumbpig-0.9.tgz
/home/snort/perl root# tar xzf dumbpig-0.9.tgz
/home/snort/perl root# chmod u+x ./dumbpig.pl
/home/snort/perl root# cpan -e "Parse::Snort"
/home/snort/perl root# cpan -e "LWP::Simple"
/home/snort/perl root# ./dumbpig.pl -r /usr/local/snort/rules/emerging-all.rules
DumbPig version 0.9 - leon.ward@sourcefire.com
Because I hate looking for the same dumb problems with snort rule-sets

        __,,    ( Dumb-pig says     )
      ~(  oo ---( "ur rulz r not so )
        ''''    ( gud akshuly" *    )

Config
----------------------
* Sensivity level - 3/3
* Blacklist outputi : Disabled
* Processing File - /home/snort/rules//emerging-all.rules
* Check commented out rules : Disabled
* Pause : Disbled
* ForceFail : Disabled
* Censor : Disabled
* Quite mode : Disabled
----------------------
Issue 1
1 Problem(s) found with rule on line 59 of /home/snort/rules//emerging-all.rules

alert tcp $HOME_NET any -> \
[75.126.138.202,72.249.118.38,208.78.69.70,204.13.249.70,208.78.68.70] $HTTP_PORTS  ( \
  msg:"ET CURRENT_EVENTS Possible Downadup/Conficker-A Infection Checking Geographical Location"; \
  flow:to_server; \
  classtype:trojan-activity; \
  reference:url,www.microsoft.com/security/portal/Entry.aspx?Name=Worm%3aWin32%2fConficker.A; \
  reference:url,www.f-secure.com/v-descs/worm_w32_downadup_a.shtml; \
  threshold:type both, count 5, seconds 60, track by_src; \
  reference:url,doc.emergingthreats.net/bin/view/Main/2008803; \
  reference:url,www.emergingthreats.net/cgi-bin/cvsweb.cgi/sigs/CURRENT_EVENTS/CURRENT_Conficker; \
  sid:2008803; \
  rev:3; \
)
- TCP/UDP rule with no deep packet checks? This rule looks more suited to a firewall or blacklist
alert tcp $HOME_NET any ->
[75.126.138.202,72.249.118.38,208.78.69.70,204.13.249.70,208.78.68.70] $HTTP_PORTS (msg:"ET CURRENT_
EVENTS Possible Downadup/Conficker-A Infection Checking Geographical
Location"; flow:to_server; classtype:trojan-activity;
reference:url,www.microsoft.com/security/portal/Entry.aspx?Name=Worm%3aWin32%2fConficker.A;
reference:url,www.f-secure.com/v-descs/worm_w32_downadup_a.shtml;
threshold:type both, count 5, seconds 60, track by_src;
reference:url,doc.emergingthreats.net/bin/view/Main/2008803;
reference:url,www.emergingthreats.net/cgi-bin/cvsweb.cgi/sigs/CURRENT_EVENTS/CURRENT_Conficker;
sid:2008803; rev:3;)

Suricata Setup

The Open Information Security Foundation ( OISF) developed Suricata to be on the leading edge in IDS/IPS software. It is very much still in development. Is that not what makes it so interesting? Before you begin, be aware of some known issues. Check out the development roadmap for upcoming scheduled releases. If you run into problems, see if it is a known issue and share your experiences with the community of developers. That is the best way to make sure Suricata improves.

With required and options software on the system, installing Suricata is pretty straight forward. Pull down the source, configure, compile, and install.

 root# cd /usr/local/src
/usr/local/src root# wget  http://www.openinfosecfoundation.org/download/suricata-1.0.1.tar.gz
/usr/local/src root# wget http://www.openinfosecfoundation.org/download/suricata-1.0.1.tar.gz.sig
/usr/local/src root# wget http://www.openinfosecfoundation.org/download/OISF.asc
/usr/local/src root# gpg --import OISF.asc
/usr/local/src root# gpg --verify suricata-1.0.1.tar.gz.sig suricata-1.0.1.tar.gz
gpg: Signature made Thu 29 Jul 2010 02:34:58 PM EDT using RSA key ID 051CC261
gpg: Good signature from "OISF "
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 3332 6BF5 5751 35CC 24F5  D2AF A30C 431D 051C C261
/usr/local/src root#  tar xzf suricata-1.0.1.tar.gz
/usr/local/src root# cd suricata-1.0.1
/usr/local/src/suricata-1.0.1 root# mkdir /usr/local/suricata
/usr/local/src/suricata-1.0.1 root#  LD_RUN_PATH="/usr/lib:/usr/local/lib" \
./configure --enable-pfring --with-libpfring-libraries=/usr/local/lib \
--with-libpfring-includes=/usr/local/include --with-libpcap-libraries=/usr/local/lib \
--with-libpcap-includes=/usr/local/include --enable-nfqueue--enable-unittests \
--enable-unified-native-timeval  --enable-profiling  --prefix=/usr/local/suricata
/usr/local/src/suricata-1.0.1 root# make
/usr/local/src/suricata-1.0.1 root# make check
/usr/local/src/suricata-1.0.1 root# make install
/usr/local/src/suricata-1.0.1 root# mkdir /usr/local/suricata/log
/usr/local/src/suricata-1.0.1 root# mkdir /usr/locall/suricata/etc
/usr/local/src/suricata-1.0.1 root# cp classification.config suricata.yaml /usr/local/suricata/etc
/usr/local/src/suricata-1.0.1 root# mkdir /usr/local/suricata/rules
/usr/local/src/suricata-1.0.1 root# cd /usr/local/suricata/rules
/usr/local/suricata/rules root# wget http://www.emergingthreats.net/rules/emerging-attack_response.rules
/usr/local/suricata/rules root# wget http://www.emergingthreats.net/rules/emerging-scan.rules
/usr/local/suricata/rules root# wget http://www.emergingthreats.net/rules/emerging-exploit.rules
/usr/local/suricata/rules root#wget http://www.emergingthreats.net/rules/emerging-current_events.rules
/usr/local/suricata/rules root#wget http://www.emergingthreats.net/rules/emerging-voip.rules
/usr/local/suricata/rules root#wget http://www.emergingthreats.net/rules/emerging-malware.rules
/usr/local/suricata/rules root#wget http://www.emergingthreats.net/rules/emerging-dos.rules
/usr/local/suricata/rules root#wget http://www.emergingthreats.net/rules/emerging-drop.rules
/usr/local/suricata/rules root#wget http://www.emergingthreats.net/rules/emerging-compromised.rules
/usr/local/suricata/rules root#wget http://www.emergingthreats.net/rules/emerging-dshield.rules
/usr/local/suricata/rules root#wget http://www.emergingthreats.net/rules/emerging-botcc.rules
/usr/local/suricata/rules root#wget http://www.emergingthreats.net/rules/emerging-rbn.rules
/usr/local/suricata/rules root# wget http://www.emergingthreats.net/rules/emerging-virus.rules
/usr/local/suricata/rules root# vi /usr/loca/suricata/etc/suricata.yaml

Modify suricata.yaml to reflect your environment. At this point, you can run Suricata with the command:

root# /usr/local/suricata/bin/suricata -c /usr/local/suricata/etc/suricata.yaml \
-s /usr/local/suricata/etc/classification.config -i eth0

To Be Continued…

Henry J. Kaiser, the father of modern American shipbuilding, once said, “Live daringly, boldly, fearlessly. Taste the relish to be found in competition – in having put forth the best within you” [9] Hopefully Jonkman, Julien, Roesch, and Olney Roesch will relish their competition and the community will enjoy the fruits of their efforts. One powerful benefit of open source is that it allows organizations the flexibility to pull down the source and setup the software in their own environment. One can easily try the packages out and become familiar with the benefits of the different IDS/IPS engines. The more you know, the better you will be at defending your organization.

Setting up the three IDS/IPS engines is only the first step. In later posts, we will continue by examining the configuration and output from Bro, Snort, and Suricata. This should help the reader understand the features each might offer an organization. Bro, Snort, and Suricata are just tools. While they will have different features, it is the person who yields the tool that determines its effectiveness. Determine for yourself what works best. Even if you have a single development box, you can setup and test against small subsets of your own network traffic. Visit Wireshark’s Sample capture page for links to pcap files and additional sources. There is also the OpenPacket’s Capture Repository, which provides the security community the capability to comment and vote on submitted pcap files. Give the IDS/IPS engines a test ride and please feel free to share your experiences.

Links

[1] Marc Ambinder, August 13th 2010, “Pentagon Wants to Secure Dot-Com Domains of Contractors,” http://www.theatlantic.com/politics/archive/2010/08/pentagon-wants-to-secure-dot-com-domains-of-contractors/61456/.
[2] John Gerber, January 5th 2010, “Suricata: A Next Generation IDS/IPS Engine,” http://blog.securitymonks.com/2010/01/05/suricata-a-next-generation-idsips-engine/.
[3] John Gerber, October 29th 2008, “Installing Bro IDS 1.4,” http://blog.securitymonks.com/2008/10/28/installing-bro-ids-14/.
[4] John Gerber, October 20th 2008, “Snort 3: The Next Generation,” http://blog.securitymonks.com/2008/10/20/snort-3-the-next-generation/.
[5] John Gerber, August 9th, 2008, “IDS/IPS: The Mark Twain of the Security World,” http://blog.securitymonks.com/2008/08/09/idsips-the-mark-twain-of-the-security-world/.
[6] John Gerber, June 17th, 2007, “IDS,” http://blog.securitymonks.com/2007/06/17/ids/.
[7] Jaikumar Vijayan, July 20th 2010, “DHS, vendors unveil open source intrusion detection engine,” http://www.computerworld.com/s/article/9179436/DHS_vendors_unveil_open_source_intrusion_detection_engine.
[8] Ellen Messmer, July 20th 2010, “Is open source Snort dead? Depends who you ask ,” http://www.networkworld.com/news/2010/072010-is-snort-dead.html?page=1.
[9] “Wikiquote: Henry J. Kaiser,” http://wapedia.mobi/enwikiquote/Henry_J._Kaiser.
[10] Matt Olney, July 20th 2010, “Innovation — You Keep Using That Word…,” http://vrt-sourcefire.blogspot.com/2010/07/innovation-you-keep-using-that-word.html.
[11] “Razorback”, http://sourceforge.net/projects/razorbacktm/files/.
[12] “Next Generation Open-Source IDS to Address Issues Facing Network Security Industry,” July 19th 2010, http://www.businesswire.com/portal/site/home/permalink/?ndmViewId=news_view&newsId=20100719005970&newsLang=en.
[13] Russ McRee, August 3rd 2010, “Suricata in toolsmith: meet the meerkat,” http://holisticinfosec.blogspot.com/2010/08/suricata-in-toolsmith-meet-meerkat.html.
[14] Victor Julien, July 22nd 2010, “On Suricata performance,” http://www.inliniac.net/blog/2010/07/22/on-suricata-performance.html.
[15] “Bro Features and Benefits,” June 28th 2010, http://www.bro-ids.org/Features.html.
[16] “Snort 2.9.0 Beta,” June 6th 2010, https://s3.amazonaws.com/snort.org/snort-beta/20100727/release_notes_290_beta.txt?AWSAccessKeyId=AKIAJJSHU7YNPLE5MKOQ&Expires=1282246315&Signature=jchbdD6d3G0ncaqOkrQAps6ZV5M%3D.
[17] John Gerber, July 19th 2009, “Blacklisting with Snort,” http://blog.securitymonks.com/2009/07/19/blacklisting-with-snort/.
[18] Russ McRee, August 2010, “Suricata: An Introduction,” http://holisticinfosec.org/toolsmith/pdf/august2010.pdf
[19] “Dynamic Application-Layer Protocol Analysis for Network Intrusion Detection,” Proceedings of the 15th conference on USENIX Security Symposium – Volume 15, 2006, http://www.icir.org/robin/papers/usenix06.pdf.
[20] Seth Hall, August 6th 2010, Github Social Coding, “http://github.com/sethhall/bro_scripts/blob/master/http-ext.bro.
[21] Seth Hall, August 6th 2010, Github Social Coding,http://github.com/sethhall/bro_scripts/blob/master/http-ext.bro#L64.
[22] Gunjan Bansal, June 16th 2010, “Installation Guide for PF_RING,” http://gunjan-bansal.blogspot.com/2010/06/installation-guide-for-pfring.html.
[23] Luca Deri and Francesco Fusco, January 30th 2010, “Exploiting Commodity Multi-core Systems for Network Traffic Analysis,” http://luca.ntop.org/MulticorePacketCapture.pdf.

[24] Mathew Olney and Matthew Watchinski, “Implementing Resource Intensive Detection Techniques With the Razorback Framework,” http://iweb.dl.sourceforge.net/project/razorbacktm/Whitepapers/razorback-whitepaper-0.1.pdf.
[25] Patrick Mullen and Ryan Pentney, Defcon Razorback Presentation, “https://www.defcon.org/images/defcon-18/dc-18-presentations/Mullen-Pentney/DEFCON-18-Mullen-Pentney-Razorback.pdf.
[26] Robin Sommer, August 24th 2010, The ICSI Networking Group Blog, “Major NSF Funding for Bro Development,” http://blog.icir.org/2010/08/major-nsf-funding-for-bro-development.html.

The use of blacklists can help. Ron Gula in his post “Event Analysis Training — Working with BlackLists” discussed how in an organization aggregating network IDS/IPs events with 10,000 to 1,000,000 events per day, Tenable has observed on a heavy day of blacklist correlation 10 to 30 hits are often generated. While those blacklist hits require minimal effort to analyze, they provide potentially very useful information. By following the interactions between blacklisted IP addresses with hosts in an organization’s network, one can find:

• systems on the organization’s network that have been compromised and are being used to send SPAM.
• possible vulnerabilities on systems within the organization’s network based on targeted ports.
• successful phishing attacks, if the organization’s IP addresses are visiting IPs associated with phishing scams.
• systems within the organization that are part of botnet, since blacklists can identify IPs that are part of the botnet command and control.

Blacklists fall into two categories: global worst offender lists (GWOL) and local worst offender lists (LWOL). LWOL will be built by an organization based on the organization’s firewalls and network activities. For GWOL, three invaluable sources providing up-to-date information pulled from various sources that are known to be used to propagate malware and spyware are:

• DNS-BH Malware Domain Blocklist: provides information maintained as part of the DNS-BH project and represents a list of domains that are known to be used to propagate malware and spyware.
• Global Watchlist: C.S. Lee describes what he and Spoonfork did in his posting “The Harimau Watchlist.” He describes the list as a “list of suspected malicous IPs/Net ranges from different sources such as SANS DShield, Arbor atlas and so forth, then putting all of them in one place.”
• Ninja Chimp Strike Force Blacklist: created on an hourly basis from data Arbor Networks, Project Honeypot, Shadowserver, and about 24+ hosts.

Martin Roesch recently posted “IP Blacklisting Version 2 for Snort 2.8.4.1 available” where he discusses version 2 of a patch created for Snort 2.8.4.1 allowing IP blacklisting.

We will walk through the process of installing Snort 2.8.4.1 with the patch in order to allow the use of blacklists. We will follow that up with creating a process that will generated the blacklist file for Snort. We will go through enough of the setup to allow the user to start using blacklists.

Required Packages

The first step is to make sure the system has the required software installed, which includes:

• aclocal
• automake
• autoconf
• Libdnet 1.10 or higher
• A recent Libpcap

The aclocal, automake, and autoconf packages should be on your system. I wanted to mention them here because the version installed on your system may end up causing problems. On one system, I had a most difficult time because of the version. Since I could not update the software on the problem system, I ended up creating a local area under the Snort directory and installing the versions I required there.

The packages that are needed will be dependent on your installation base. I will walk through a few package installations.

Libdnet

Make sure Libdnet is in your library path:

 root# /sbin/ldconfig -p | grep -i libdnet
 libdnet32.so.1 (libc6) => /usr/lib/libdnet32.so.1
 libdnet32.so (libc6) => /usr/lib/libdnet32.so

If you do not get a path returned, you will need to install libdnet (use –prefix if it needs to be installed in a special location):

 root# cd /usr/local/src
 /usr/local/src root# wget \
http://libdnet.googlecode.com/files/libdnet-1.12.tgz
 /usr/local/src root# tar xzf libdnet.cvs.tgz
 /usr/local/src root# cd libdnet-1.12
 /usr/local/src root# ./configure
 /usr/local/src root# make
 /usr/local/src root# make install
 /usr/local/src root# cp include/dnet/sctp.h /usr/local/include/dnet

If you have installed libdnet in a special location, make sure to include its path in /etc/ld.so.conf.

PCRE (pcre-7.8)

The PCRE library is a set of functions that implement regular expression pattern matching using the same syntax and semantics as Perl 5. If you can install PCRE via a port specific to your operating system, that is the best way to install in order to avoid having to keep the software up-to-date. Below are the instructions for installing the software from source.

 root# cd /usr/local/src
 /usr/local/src root# wget \
http://downloads.sourceforge.net/sourceforge/pcre/\
pcre-7.9.tar.gz?use_mirror=internap
 /usr/local/src root# wget \
http://downloads.sourceforge.net/sourceforge/pcre/\
pcre-7.9.tar.gz.sig?use_mirror=internap
 /usr/local/src root# gpg --verify pcre-7.9.tar.gz.sig pcre-7.9.tar.gz
gpg: Signature made Sat 11 Apr 2009 10:33:38 AM EDT using RSA key ID FB0F43D8
gpg: Good signature from "Philip Hazel <ph10@hermes.cam.ac.uk>"
gpg: aka "Philip Hazel <ph10@cam.ac.uk>"
gpg: aka "Philip Hazel <ph10@cus.cam.ac.uk>"
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: 45F6 8D54 BBE2 3FB3 039B 46E5 9766 E084 FB0F 43D8

 /usr/local/src root# tar xzf pcre-7.9.tar.gz
 /usr/local/src root# cd pcre-7.9
 /usr/local/src/pcre-7.8 root# ./configure --prefix=/usr/local/pcre
 /usr/local/src/pcre-7.8 root# make
 /usr/local/src/pcre-7.8 root# make test
 /usr/local/src/pcre-7.8 root# make install

Libpcap and Large Files Support

Under some OSs, you need to compile libpcap and Snort to support large files (files large than 2G). Since the source code of libpcap will be needed for this configuration, we are going to install the resulting libpcap under /usr/local/snort. If the libpcap installed on your system does not produce an error, skip this step. You will want to follow these steps only if you get the following error when running Snort with a large file:

 root# ls -lh /data/ids/full2.pcap
-rw-r--r-- 1 root root 12G Oct 26 10:01 /data/ids/full2.pcap
 root# /usr/local/snort/bin/snort -o -A none -c \
/usr/local/snort/conf/snort.conf -l /logs/snort/logs \
-r /data/ids/full2.pcap
Error getting stat on pcap file: /data/ids/full2.pcap:
Value too large for defined data type
ERROR: Error getting pcaps
Fatal Error, Quitting..

First, we need to compile large file support into libpcap. As mentioned above, we will install the libraries under /usr/local/snort.

 root# cd /usr/local/src
 /usr/local/src root# wget http://www.tcpdump.org/release/\
libpcap-1.0.0.tar.gz
 /usr/local/src root# wget http://www.tcpdump.org/release/\
libpcap-1.0.0.tar.gz.sig
 /usr/local/src root# wget http://www.tcpdump.org/tcpdump-workers.asc
 /usr/local/src root# gpg --import tcpdump-workers.asc
gpg: key 89E917F3: "tcpdump.org (SIGNING KEY) " not changed
gpg: Total number processed: 1
gpg: unchanged: 1
 /usr/local/src root# gpg --verify libpcap-1.0.0.tar.gz.sig libpcap-1.0.0.tar.gz
gpg: Signature made Tue 25 Sep 2007 10:11:56 PM EDT using DSA key ID 89E917F3
gpg: Good signature from "tcpdump.org (SIGNING KEY) "
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: 0227 54EB 4C30 9185 FD31 33A3 464D 3CEB 89E9 17F3
 /usr/local/src root# tar xzf libpcap-1.0.0.tar.gz
 /usr/local/src root# cd libpcap-1.0.0
 /usr/local/src/libpcap-1.0.0 root# ./configure --prefix=/usr/local/snort \
CFLAGS="-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE \
-D_FILE_OFFSET_BITS=64"
 /usr/local/src/libpcap-1.0.0 root# make
 /usr/local/src/libpcap-1.0.0 root# make shared
 /usr/local/src/libpcap-1.0.0 root# make install
 /usr/local/src/libpcap-1.0.0 root# make install-shared

In the next section, we will discuss how to install Snort with blacklist, large file, and MySQL support.

Snort

Below we will get the software, verify, configure, and install the software under the /usr/local/snort area. Please adjust this to your environment. Reminder to Mac OS X and FreeBSD users, use the md5 command instead of md5sum.

 root# cd /usr/local/src
 /usr/local/src root# wget http://dl.snort.org/snort-current/snort-2.8.4.1.tar.gz
 /usr/local/src root# wget http://dl.snort.org/snort-current/snort-2.8.4.1.tar.gz.md5
 /usr/local/src root# cat snort-2.8.4.1.tar.gz.md5
63f4e76ae96a2d133f4c7b741bad5458 snort-2.8.4.1.tar.gz
 /usr/local/src root# md5sum snort-2.8.4.1.tar.gz
63f4e76ae96a2d133f4c7b741bad5458 snort-2.8.4.1.tar.gz
 /usr/local/src root# tar xzf snort-2.8.4.1.tar.gz
 /usr/local/src root# cd snort-2.8.4.1

At this point, we will add the blacklist path and regenerate the configuration and make files. If you do not have prelude installed, the following will occur:

 /usr/local/src/snort-2.8.4.1 root# wget http://www.snort.org/users/roesch/code/iplist.patch.v2.tgz
 /usr/local/src/snort-2.8.4.1 root# tar xzf iplist.patch.v2.tgz
 /usr/local/src/snort-2.8.4.1 root# patch -p1 < iplist.patch
 /usr/local/src/snort-2.8.4.1 root# aclocal
aclocal:configure.in:1050: warning: macro `AM_PATH_LIBPRELUDE' not found in library
 /usr/local/src/snort-2.8.4.1 root# autoconf
configure.in:1050: error: possibly undefined macro: AM_PATH_LIBPRELUDE
 If this token and others are legitimate, please use m4_pattern_allow.
 See the Autoconf documentation.
 /usr/local/src/snort-2.8.4.1 root# automake

Eoin Miller has been very active and helpful in resolving problems with the patch. In a recent post, he resolved the problem of ” warning: macro `AM_PATH_LIBPRELUDE’ not found in library” by commenting out those lines pertaining to Prelude in the configure file. The exact line number will differ between systems.

 #if test "x$enable_prelude" = "xyes"; then
# AM_PATH_LIBPRELUDE(0.9.6, use_prelude="yes", use_prelude="no")
# if test "$use_prelude" = "yes"; then
# LDFLAGS="${LDFLAGS} ${LIBPRELUDE_LDFLAGS}"
# LIBS="$LIBS ${LIBPRELUDE_LIBS}"
# CFLAGS="$CFLAGS ${LIBPRELUDE_PTHREAD_CFLAGS}"

#cat >>confdefs.h <<\_ACEOF
##define HAVE_LIBPRELUDE
#_ACEOF
#
# fi
#fi

You may also need to change the definition of LIBDNET and DNETFLAGS in the configuration file. The configure file will have these variables set by calling dumbnet-config without any path. If your system does not know the location of dumbnet-config, this will cause problems. Instead, figure out the values and replace the calls to dumbnet-config with those values. To find the values that should be used, issue the command:

 /usr/local/src/snort-2.8.4.1 root# /usr/local/bin/dnet-config --libs
-L/usr/local/lib -ldnet
 /usr/local/src/snort-2.8.4.1 root# /usr/local/bin/dnet-config --cflags
-I/usr/local/include

In the above case, you should change all occurrences in the configure file from having:

 LIBDNET="`dumbnet-config --libs`"
 DNETFLAGS="`dumbnet-config --cflags`"

to (depending on the results of the above dnet-config results):

 LIBDNET="-L/usr/local/lib -ldnet"
 DNETFLAGS="-I/usr/local/include"

The post by Martin Roesch, suggest a replacement for the function IpListEval. You may want to replace that in file spp_iplist.c.

 /usr/local/src/snort-2.8.4.1 root# vi ./src/preprocessors/spp_iplist.c

We are going to add in support to place alerts into a MySQL database. If MYSQL is installed on the system, you can use the “–with-mysql” configuration option to specify where. In a previous post, “Introduction to MySQL,” we went through the installation of MySQL into the /usr/local/mysql directory. For such an installation, the –with-mysql-includes=/usr/local/mysql/include and –with-mysql-libraries=/usr/local/mysql/lib command options must be used. In order to use the dynamic plugin libraries, Snort needs to be able to find libmysqlclient.so. On some operating systems, you may have problems. Adding LDFLAGS=”-L/usr/local/mysql/lib/mysql” should work.

You may want to consider configuring Snort to allow decoder and preprocessor rule eventing. This allows you to enable and disable decoder and preprocessor events on a rule by rule bases. It also allow you to specify the rule type or action of a decoder or preprocessor event on a rule by rule basis. Enable this configuration option with the configuration option using –enable-iplist.

We will also be adding in large file support. To allow enable blacklist files, we must include the configuration option –enable-iplist. If you had to install libdnet in a special location, you will need to specify that location with the “–with-dnet-includes=” and “–with-dnet-libraries=” configuration options.

We will configure Snort with the following command:

 /usr/local/src/snort-2.8.4.1 root# CFLAGS="-D_LARGEFILE_SOURCE \
-D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64" \
./configure --prefix=/usr/local/snort --with-libpcap-includes=/usr/local/snort/include \
--with-libpcap-libraries=/usr/local/snort/lib \
--with-libpcre-includes=/usr/local/pcre/include \
--with-libpcre-libraries=/usr/local/pcre/lib \
--with-mysql-includes=/usr/local/mysql/include\
--with-mysql-libraries=/usr/local/mysql/lib \
--enable-iplist \
--enable-decoder-preprocessor-rules

After you configure Snort, you continue to make and install it.

 /usr/local/src/snort-2.8.4.1 root# make
 /usr/local/src/snort-2.8.4.1 root# make check
 /usr/local/src/snort-2.8.4.1 root# make install
 /usr/local/src/snort-2.8.4.1 root# mkdir -p /usr/local/snort/etc
 /usr/local/src/snort-2.8.4.1 root# cp etc/* /usr/local/snort/etc
 /usr/local/src/snort-2.8.4.1 root# mkdir -p /usr/local/snort/preproc_rules
 /usr/local/src/snort-2.8.4.1 root# cp preproc_rules/*.rules /usr/local/snort/preproc_rules
 /usr/local/src/snort-2.8.4.1 root# /usr/local/snort/bin/snort -V

 ,,_ -*> Snort! <*-
 o" )~ Version 2.8.4.1 (Build 38)
 '''' By Martin Roesch & The Snort Team: http://www.snort.org/team.html
 Copyright (C) 1998-2009 Sourcefire, Inc., et al.
 Using PCRE version: 7.8 2008-09-05

You may need to move the lib_sfdynamic_example_rule.so, if you receive errors. Just issue the command:

 /usr/local/src/snort-2.8.4.1 root# mv /usr/local/snort/lib/snort_dynamicrules/\
lib_sfdynamic_example_rule.so \
/usr/local/snort/lib/snort_dynamicrules/lib_sfdynamic_example_rule.so.broke

Rules

Now we need some rules. For this example we will get the rules from the Snort and the Emerging Threats site. You will need to register for the rules at the Snort site. Do consider subscribing for the latest up-to-date rules. Registered users can only access rules 30 days after their release.

 root# cd /usr/local/snort/rules
 /usr/local/snort/rules root# wget http://www.emergingthreats.net/rules/emerging-all.rules
 /usr/local/snort/rules root# cd /usr/local/src
 /usr/local/src root# wget \
http://www.snort.org/pub-bin/downloads.cgi/\
Download/vrt_os/snortrules-snapshot-CURRENT.tar.gz
 /usr/local/src root# md5sum snortrules-snapshot-CURRENT.tar.gz
184aed405da3f1043b82d81c98122237 snortrules-snapshot-CURRENT.tar.gz
 /usr/local/src root# mv snortrules-snapshot-CURRENT.tar.gz /usr/local/snort/
 /usr/local/src root# cd /usr/local/snort/
 /usr/local/snort root# tar xzf snortrules-snapshot-CURRENT.tar.gz
 /usr/local/snort root# rm snortrules-snapshot-CURRENT.tar.gz
 /usr/local/snort root# vi /usr/local/snort/etc/snort.conf

Modify /usr/local/snort/etc/snort.conf to your environment. Make sure the RULE_PATH is set to /usr/local/snort/rules. If you configured Snort to enable decoder and preprocessor rules, you will need to add a line specifying the location of those files. Define PREPROC_RULE_PATH with the line:

 var PREPROC_RULE_PATH ../preproc_rules

Later in the snort.conf file include the lines (before other rule lists are included):

 include $PREPROC_RULE_PATH/preprocessor.rules
 include $PREPROC_RULE_PATH/decoder.rules

If you wish to use the emerging threat rules, add:

 include $RULE_PATH/emerging-all.rules

in the /usr/local/snort/etc/snort.conf file. Do not forget to adjust dynamicpreprocessor file and dynamicengine path. Mac OS X users will need to use the dynamic libraries. Uncomment the Mac OS X lines in the Snort configuration file.

Dumbpig

Leon Ward has released a Perl program, Dumbpig, which will check Snort rules for badly formatted entires and incorrect usage. He has even added blacklist support (see posting “ET RBN Blacklists with Snort and DumbPig“). To pull down dumbpig.pl, the required Perl modules, and run it against the Emerging Threats rule set:

 root# cd /home/snort/perl
 /home/snort/perl root# wget rm-rf.co.uk/downloads/dumbpig.pl
 /home/snort/perl root# chmod u+x ./dumbpig.pl
 /home/snort/perl root# cpan -e "Parse::snort"
 /home/snort/perl root# cpan -e "LWP::Simple"
 /home/snort/perl root# ./dumbpig.pl -r /usr/local/snort/rules/emerging-all.rules
DumbPig version 0.5 - leon.ward@sourcefire.com
Because I hate looking for the same dumb problems with snort rule-sets

          __,,    ( Dumb-pig says     )
        ~(  oo ---( "ur rulz r not so )
          ''''    ( gud akshuly" *    )

Config
----------------------
* Sensivity level - 3/3
* Blacklist outputi : Disabled
* Processing File - /home/snort/rules//emerging-all.rules
* Check commented out rules : Disabled
* Pause : Disbled
* ForceFail : Disabled
* Censor : Disabled
* Quite mode : Disabled
----------------------
Issue 1
1 Problem(s) found with rule on line 59 of /home/snort/rules//emerging-all.rules

alert tcp $HOME_NET any -> \
[75.126.138.202,72.249.118.38,208.78.69.70,204.13.249.70,208.78.68.70] $HTTP_PORTS  ( \
    msg:"ET CURRENT_EVENTS Possible Downadup/Conficker-A Infection Checking Geographical Location"; \
    flow:to_server; \
    classtype:trojan-activity; \
    reference:url,www.microsoft.com/security/portal/Entry.aspx?Name=Worm%3aWin32%2fConficker.A; \
    reference:url,www.f-secure.com/v-descs/worm_w32_downadup_a.shtml; \
    threshold:type both, count 5, seconds 60, track by_src; \
    reference:url,doc.emergingthreats.net/bin/view/Main/2008803; \
    reference:url,www.emergingthreats.net/cgi-bin/cvsweb.cgi/sigs/CURRENT_EVENTS/CURRENT_Conficker; \
    sid:2008803; \
    rev:3; \
)
- TCP/UDP rule with no deep packet checks? This rule looks more suited to a firewall or blacklist
alert tcp $HOME_NET any ->
[75.126.138.202,72.249.118.38,208.78.69.70,204.13.249.70,208.78.68.70] $HTTP_PORTS (msg:"ET CURRENT_
EVENTS Possible Downadup/Conficker-A Infection Checking Geographical
Location"; flow:to_server; classtype:trojan-activity;
reference:url,www.microsoft.com/security/portal/Entry.aspx?Name=Worm%3aWin32%2fConficker.A;
reference:url,www.f-secure.com/v-descs/worm_w32_downadup_a.shtml;
threshold:type both, count 5, seconds 60, track by_src;
reference:url,doc.emergingthreats.net/bin/view/Main/2008803;
reference:url,www.emergingthreats.net/cgi-bin/cvsweb.cgi/sigs/CURRENT_EVENTS/CURRENT_Conficker;
sid:2008803; rev:3;)

There are many more issues reported. I included the first issue, because it brings up the point that rules can be rewritten to use blacklists. In Leon’s post, he found rewriting the Russian Business Network rules dropped processing time by over 40 percent. The Dumbpig program, along with the blacklist patch, can help tremendously with performance.

Configuration File

In the /usr/local/snort/etc/snort.conf file, make sure to add iplist preprocessor command. For example, if we are going to combine the three separate source for blacklists into one blacklist file, we might want to do:

 preprocessor iplist: blacklist watchlist /usr/local/snort/conf/combo.blacklist \
 whitelist /usr/local/snort/conf/default.whitelist

Testing Snort Using Attack Data

Whenever installing a new version of Snort, it is a good idea to test it. Leon Ward has made available a pcap file containing attacks that occurred back in 2001 against a honeypot. If you have other data that will produce interesting results, please feel free to use that.

 root# mkdir -p /data/ids/tcpdump
 root# cd /data/ids/tcpdump
 /data/ids/tcpdump root# wget http://rm-rf.co.uk/downloads/Honeynet-RFP-iis.tgz
 /data/ids/tcpdump root# tar xzf Honeynet-RFP-iis.tgz
 /data/ids/tcpdump root# /usr/local/snort/bin/snort -c /usr/local/snort/etc/snort.conf \
-A fast -l /data/ids/tcpdump -r ./Honeynet-RFP-iis.pcap
 /data/ids/tcpdump root# ls /data/ids/tcpdump/alert /data/ids/tcpdump/snort.log.*

Two results files should get created. The file /data/ids/tcpdump/alert will contain the alerts and /data/ids/tcpdump/snort.log.<date>, which contains the pcaps of the detected events.

Blacklists

We will pull down upates from DNS-BH Malware Domain Blocklist, the Global Watchlist, and Ninja Chimp Strike Force Blacklist. The file format for the DNS-BH blocklist is:

# domain type original_reference-why_it_was_listed dateadded seqnum note--pound sign=comment 1
# notice notice duplication is not permitted 2
 00.devoid.us malware www.cyber-ta.org/releases/malware-analysis/public/SOURCES/DNS.Cumulative.Summary 20090321 3
 032439.com malware www.malwaredomainlist.com 20080822 9
 0503.pass.as malware www.threatexpert.com/report.aspx?md5=21d60d5e9b8c9353b1b55994dfa1b11e 20080503 11

The Global Watchlist uses the file format:

# watchlist.security.org.my, contact mel@hackinthebox.org
# ip/net, source, comment, name, last update (GMT+8)
10.50.50.50, http://www.dshield.org/ipsascii.html, Dshield: Top IPs, dshield-top-ips, 2009/07/20 00:00:11
61.184.255.175, http://www.dshield.org/ipsascii.html, Dshield: Top IPs, dshield-top-ips, 2009/07/20 00:00:11
222.82.249.235, http://www.dshield.org/ipsascii.html, Dshield: Top IPs, dshield-top-ips, 2009/07/20 00:00:11

Ninja Chimp uses the file format:

# This is a compiled list of dirty hosts associated
# with bruteforcing attempts, spam, botnets, etc.
# The list is comprised of data from Arbor Networks,
# Project Honeypot, Shadowserver, and about 24+ hosts
# I maintain. It is sorted on an hourly basis to keep
# information current and is consistently changing

Sun Jul 19 12:59:03 CDT 2009

99.254.50.139
99.248.26.177

In order to track the data and be able to generate various reports, we will take the data and place it into a databases. The database schema can be fairly simple. The “ip” feild represents the ip or network address. We will add an “end_ip” to reduce calculation time and allow use to search if an IP appears within ip/cidr.

mysql> describe watchlist;
+------------+------------------+------+-----+---------------------+-------+
| Field | Type | Null | Key | Default | Extra |
+------------+------------------+------+-----+---------------------+-------+
| ip | int(10) unsigned | NO | PRI | 0 | |
| cidr | int(2) unsigned | NO | | 0 | |
| source | varchar(100) | YES | | NULL | |
| comment | varchar(50) | YES | | NULL | |
| name | varchar(50) | YES | | NULL | |
| lastupdate | datetime | YES | | 0000-00-00 00:00:00 | |
| domain | varchar(50) | YES | | NULL | |
| reported | datetime | YES | | 0000-00-00 00:00:00 | |
| active | enum('yes','no') | YES | | no | |
| end_ip | int(10) unsigned | YES | MUL | 0 | |
+------------+------------------+------+-----+---------------------+-------+

We will create two Perl programs. The first program to pull the data down from the websites and the second program parses the files and load the information into the tables. The different blacklist sites are pulling information from the same sources. Overlap is expected. Each IP and network will get only one entry into the database and be listed once in the Snort blacklist file. The first program to pull data down from the sites might look like:

#!/usr/local/bin/perl -w

use LWP::UserAgent;

sub readpage {
 my($url) = @_;

 my $ua = new LWP::UserAgent;

 # Go out and retrieve page
 my $req = new HTTP::Request('GET', $url);
 my $res = $ua->request($req);

 # Check if the requested webpage is there and return results
 if ($res->is_success) { # Request successful
 return(1,$res->content)
 }
 else {
 return(0,"");
 }
}

sub pulldate {
 my($url,$inputfn) = @_;

 my ($pjstatus,$page) = &readpage($url);
 if ($pjstatus) {
 open(OUTFILE, ">$inputfn") || die "ERROR: Can't open $inputfn: $!\n";
 print OUTFILE $page;
 close(OUTFILE);
 }
 return($pjstatus);
}

# Main

my $base_dir = "/home/snort/projects/blacklists";
my $datadir = $base_dir . "/data/";

chdir $datadir || die "ERROR: Data directory $datadir does not exist: $!\n";
&pulldate("http://watchlist.security.org.my/all.txt",$datadir . 'watchlist.dat');
&pulldate("http://www.malwaredomains.com/files/domains.txt",$datadir . 'domains.dat');
&pulldate("http://www.infiltrated.net/blacklisted",$datadir . 'blacklisted.dat');

The second Perl program to parse the blacklists, is a bit longer. Initially I planned in including it, but this post is already quite long. I will leave the program to the reader’s creative mind. You know the format of the data sources and you know the database. It is not difficult to write a program to parse the files and place the data into the database. Just make sure to check if the IP/network is already in the watchlist table. If there is already an entry, only update the lastupdate feild. You will use that to track IPs and networks that may be dropped from the blacklist files.

You should note that the various blacklist will report the IPs and domains differently. The DNS-BH blocklist provides the domainname. You may want to convert that into IP/CIDR notation. Ninja Chimp blacklist contains only a list of IPs. The Global watchlist will provide IPs or networks. You may choose to use the Net::DNS and/or the Geo::IP Perl modules, depending on what information you want to store. If you use these modules, you should also be aware that such operations may end up warning the hostel domains that your site is trying to lookup information on those IPs. There is always the option to tunnel the request to an an IP that will not resolve back to your organization.

Since I like to gather information on the IPs and networks for use in evaluating the potential threat level, I keep the following information for each IP/network:

mysql> describe watchips;
+--------------+------------------+------+-----+---------------------+-------+
| Field | Type | Null | Key | Default | Extra |
+--------------+------------------+------+-----+---------------------+-------+
| ip | int(10) unsigned | NO | PRI | 0 | |
| hostname | varchar(50) | YES | | NULL | |
| entered | datetime | YES | | 0000-00-00 00:00:00 | |
| lastseen | datetime | YES | | 0000-00-00 00:00:00 | |
| asn | int(5) unsigned | NO | MUL | 0 | |
| asn_comp | varchar(50) | YES | | NULL | |
| asn_desc | varchar(75) | YES | | NULL | |
| network | int(10) unsigned | NO | | 0 | |
| cidr | int(2) unsigned | YES | | NULL | |
| ccode | varchar(2) | NO | MUL | NULL | |
| rir | varchar(10) | YES | | NULL | |
| rir_moddate | date | YES | | 0000-00-00 | |
| ccode3 | varchar(2) | YES | | NULL | |
| country_name | varchar(50) | YES | | NULL | |
| region | varchar(25) | YES | | NULL | |
| region_name | varchar(50) | YES | | NULL | |
| city | varchar(25) | YES | | NULL | |
| postal_code | varchar(25) | YES | | NULL | |
| latitude | decimal(7,4) | NO | | 0.0000 | |
| longitude | decimal(7,4) | NO | | 0.0000 | |
| time_zone | varchar(25) | YES | | NULL | |
| area_code | varchar(5) | YES | | NULL | |
| continent | varchar(2) | YES | | NULL | |
| metro_code | int(3) unsigned | NO | | 0 | |
+--------------+------------------+------+-----+---------------------+-------+

Creating Blacklist Files for Snort

The files Snort uses for blacklists are specified in the snort.conf file (mentioned above). The format for these files are:

# This is a blacklist file, there are many like it but this one is mine
# Comments are supported
#10.1.1.0/24 192.168.0.0/16 # I can do inline comments too and put
 # multiple CIDR blocks on one line
10.50.50.50/32

A Perl program to pull the data from the tables and generate the blacklist files in the format Snort expects can be as simple as:

#!/usr/local/bin/perl -w

use DBI;

my $base_dir = "/usr/local/snort/conf";
my $snortfile = $base_dir . "/combo.blacklist";
my $db = "badips";
my $mysql_user = "secretpig";
my $mysql_passwd = "secretpigpassword";
my $db_host = 'localhost';
my $results = "";

local($dbh) = DBI->connect("DBI:mysql:$db:$db_host",
$mysql_user, $mysql_passwd) || die "ERROR: Connecting: $DBI::errstr\n";

my $start_time = `/bin/date -d '7 day ago' +"%Y-%m-%d %H:%M:%S"`;
chomp $start_time;
my $sql = qq{ SELECT inet_ntoa(ip), cidr, comment, source
 FROM watchlist WHERE lastupdate >= ? AND active=? };
$sth = $dbh->prepare( $sql );
$rc = $sth->execute($start_time,"YES");
if ($rc) {
 while (my($ip,$cidr,$ocomment,$source) = $sth->fetchrow_array()) {
 my $comment = "$source";
 if ($ocomment ne "") {
 $comment = $ocomment . " ($source)";
 }
 $results .= sprintf "%-20s # $comment\n","$ip/$cidr";
 }
}
open(OUTFILE,">$snortfile");
print OUTFILE $results;
close(OUTFILE);
exit;

Final Thoughts

The above program will generate a large number of blacklisted IPs and domains. It might be helpful to add a process that helps evaluates the threat a little further. This was the motivation behind the work done by Jian Zhang (SRI), Phillip Porras (SRI), and Johannes Ullrich (SANS Institute). They attempted to develop a highly predictive blacklist (HPB). The technology is similar to Google’s PageRank algorithm using a multiphase approach which uses a relevance ranking and severity assessment to produce blacklists potentially unique for each organization or organization type participating in the HPB process.

While I do believe in tapping into the collaborative intelligence of the security community, one needs to verify the integrity of the information before it can be trusted. It is ill advised to automatically block IPs or networks listed in any blacklist unless your organization can accept the consequences. This holds true for commercial products as well. For example, image your organization HQ’s IP, through an accident or hackers, ending up on the blacklist. That is a large part of the motivation behind the work I have been doing with the Threat Observation, Tracking, and Evaluation Model (TOTEM). While I encourage organizations to take advantage of such options as sharing their organization’s information with DShield and receiving tailored HPBs, always verify before taking actions. Develop some form of an additional assessment method within your organization. Good security solutions involve layers. Blacklists can be a very useful tool in an organization’s security arsenal.

The original slides made heavy use of the Microsoft PowerPoint animation feature. Unfortunately, SlideShare does not currently support animation. You can download the presentation and the animations will work, but I ended up modifying the slides so they are more viewable online. SlideBoom will keep the animation, but it does it by creating a video of the presentation. I decided to stick with SlideShare and spare you the resulting nine minute video. While I should add audio and make a SlideCast, this post might never be completed if I wait until I have time to create a really nice web presentation.

Merriam-Webster defines a totem as any supposed entity that watches over or assists a group of people, such as a family, clan, or tribe. In this presentation I focused on how TOTEM assists in watching over and evaluating the threat an IP represents. The idea behind TOTEM is simple: compare threat information from sources such as watchlists (DShield, Emerging Threats, SenderBase, etc.) to activities with the organization (IDS/IPS, flow logs, etc.) and other locations (SANS ISC, DOE federated model, etc.). As new threat information and activity sources are added, a better evaluation can be rendered.

The purpose of this presentation has been to share the basic ideas behind TOTEM with the hope that others may provide helpful insight. So far I have not disappointed. I wanted to thank everyone for I have received some very intriguing ideas.

The Three Wise Men Speak

As readers of this blog know, I have been looking at the Bro IDS. I hope to shortly release a post on setting up Bro 1.4, which was released this Friday. It will be interesting to watch these two IDS/IPS progress in the upcoming year. With two such development efforts, it was somewhat surprising to read the news release from the Open Information Security Foundation (OIS) that “OISF Receives Grant Funding for Open Source Next Generation IDS/IPS.” Good to see that innovation is occurring in an area of security previously given up for dead (see post, “IDS/IPS: The Mark Twain of the Security World“).

For cutting edge knowledge on Snort 3 development, Martin Roesch, Leon Ward, and Richard Bejtlich are three wise men who can provide greater insight. Fortunately, they have all written posts on Snort 3:

• Snort 3 Architecture Series Part 1: Overview by Martin Roesch
• Snort 3.0 Architecture Series Part 2: Changes and Betas by Martin Roesch
• Snort 3.0 Architecture Series Part 3: The command shell by Martin Roesch
• Snort 3 Beta on Ubuntu / Debian Installation by Leon Ward
• Using SnortSP and Snort 2.8.2 by Richard Bejtlich
• The power of Snort 3.0 by Richard Bejtlich

Overview

In this post, we will be installing Snort 2.8.3.1, the Snort Security Platform (SnortSP), and the Snort 3 analytical engine. Please see Martin Roesch’s, Leon Ward’s, and Richard Bejtlich’s posts for more in-depth discussion. I am going to discuss a few basic concepts of the Snort 3 architecture, go through installation, and discuss some configuration and operation of the software. I plan on following this post with a another concerning the setup and installation of Bro 1.4. I will follow that post with an analysis of results from the two systems. At some point, we will discuss integration of results into RTIR.

The Snort 3 architecture consists of the software framework, called the Snort Security Platform (SnortSP) and engines. Sometimes SnortSP will be referred to as “the framework” or just “framework.” SnortSP is designed to perform basically as an “operating system” for packet-based network security applications, providing common functionality that all programs need. For example, it operates on the data source performing such functions as acquiring the data (DAQ), decoding, flow normalization, IP defragmentation, and stream reassembly. SnortSP is composed of the action system, attribute management system (AMS), and the dispatcher. It allows you the ability to interact with the system through the command shell and snortd. From a developer’s point of view, SnortSP is what gathers data and handles any evasive techniques or other conditions that occur in suspicious and malicious traffic. SnortSP normalizes the data and then provides this cleaned up data to the engines for inspection. The engines are analysis modules that plug into SnortSP. Sometimes the engines will be referred to as “engine modules,” “analytics modules,” or simply the “analytics.” Multiple engines can run simultaneously. Here is a reference diagram.

Some of the major features of Snort 3 include:

• Shell-based user interface with embedded scripting language
• Native IPv6, MPLS and GRE support
• Native support for inline operation
• More subsystem plugin types such as data acquisition modules, decoders and traffic analyzers
• Multithreaded execution model – multiple analysis engines may operate simultaneously on the same traffic
• Performance increases

SnortSP comes with a Snort 2.8.2 detection engine implemented as a SnortSP engine module. Annother engine that is being developed is an network contextually aware engine that will allow Snort to understand what it is defending. Marty describes network context as, “essentially data about the environment that is being defended by Snort, the composition of the hosts in the network as well as the local network composition.” The idea is that no longer will you have to manually teach Snort rules. Marty reports “You teach Snort what the network looks like so it can defend itself accordingly. It tunes itself. My end goal is to have a self-tuning protection engine.” Richard provides insight into another upcoming engine, “SnortSP is expected to support a new engine called Policy Enforcement Point (PEP), a stateless (yes, not stateful) firewall that integrates with Sourcefire’s other products. Any engine running on the SnortSP will be able to call PEP to enforce access control decisions, assuming the sensor/IPS is in a place to take such actions.” It will be very interesting to see what engines are developed. This is the power of the design, giving Snort 3 the ability to grow to meet the needs of the security community.

Installation and Configuration

Generally, it is smart to start with what you know and determine expected results. We will begin with setting up and installing the latest stable version of Snort (Snort 2.8.3.1). Once it is configure and running, we will use some test data consisting of actual attacks that occurred in 2001. This allows us to develop a baseline of expected results. After that, we will move on to installing SnortSP, which comes with a implemented version of Snort 2.8.2 detection engine. First, you will need to have libpcre version 6.0 or greater installed.

PCRE (pcre-7.8)

The PCRE library is a set of functions that implement regular expression pattern matching using the same syntax and semantics as Perl 5. If you can install PCRE via a port specific to your operating system, that is the best way to install in order to avoid having to keep the software up-to-date. Below are the instructions for installing the software from source.

 root# cd /usr/local/src
 /usr/local/src root# wget \

http://downloads.sourceforge.net/pcre/pcre-7.8.tar.gz?modtime=1220617433&big_mirror=0

 /usr/local/src root# tar xzf pcre-7.8.tar.gz
 /usr/local/src root# cd pcre-7.8
 /usr/local/src/pcre-7.8 root# ./configure --prefix=/usr/local/pcre
 /usr/local/src/pcre-7.8 root# make
 /usr/local/src/pcre-7.8 root# make test
 /usr/local/src/pcre-7.8 root# make install

Libpcap and Large Files Support

Under some OSs, you need to compile libpcap and Snort to support large files (files large than 2G). Since the source code of libpcap will be needed for this configuration, we are going to install the resulting libpcap under /usr/local/snort. If the libpcap installed on your system does not produce an error, skip this step. You will want to follow these steps only if you get the following error when running Snort with a large file:

 root# ls -lh /data/ids/full2.pcap
-rw-r--r-- 1 root root 12G Oct 26 10:01 /data/ids/full2.pcap
 root# /usr/local/snort/bin/snort -o -A none -c /usr/local/snort/conf/snort.conf -l /logs/snort/logs \
-r /data/ids/full2.pcap
Error getting stat on pcap file: /data/ids/full2.pcap: Value too large for defined data type
RROR: Error getting pcaps
Fatal Error, Quitting..

First, we need to compile large file support into libpcap. As mentioned above, we will install the libraries under /usr/local/snort.

 root# cd /usr/local/src
 /usr/local/src root# wget http://www.tcpdump.org/release/libpcap-0.9.8.tar.gz
 /usr/local/src root# wget http://www.tcpdump.org/release/libpcap-0.9.8.tar.gz.sig
 /usr/local/src root# wget http://www.tcpdump.org/tcpdump-workers.asc
 /usr/local/src root# gpg --import tcpdump-workers.asc
gpg: key 89E917F3: "tcpdump.org (SIGNING KEY) " not changed
gpg: Total number processed: 1
gpg:              unchanged: 1
 /usr/local/src root# gpg --verify libpcap-0.9.8.tar.gz.sig libpcap-0.9.8.tar.gz
gpg: Signature made Tue 25 Sep 2007 10:11:56 PM EDT using DSA key ID 89E917F3
gpg: Good signature from "tcpdump.org (SIGNING KEY) "
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 0227 54EB 4C30 9185 FD31  33A3 464D 3CEB 89E9 17F3
 /usr/local/src root# tar xzf libpcap-0.9.8.tar.gz
 /usr/local/src root# cd libpcap-0.9.8
 /usr/local/src/libpcap-0.9.8 root# ./configure --prefix=/usr/local/snort \
CFLAGS="-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE \
-D_FILE_OFFSET_BITS=64"
 /usr/local/src/libpcap-0.9.8 root# make
 /usr/local/src/libpcap-0.9.8 root# make shared
 /usr/local/src/libpcap-0.9.8 root# make install
 /usr/local/src/libpcap-0.9.8 root# make install-shared

In the next section, we will discuss how to install Snort with and without large file support.

Snort (snort-2.8.3.1)

The latest release of Snort, as of this writing, is Snort 2.8.3.1. Below we will get the software, verify, configure, and install the software under the /usr/local/snort area. Please adjust this to your environment. Reminder to Mac OS X and FreeBSD users, use the md5 command instead of md5sum.

 root# cd /usr/local/src
 /usr/local/src root# wget http://www.snort.org/dl/snort-2.8.3.1.tar.gz
 /usr/local/src root# wget http://www.snort.org/dl/snort-2.8.3.1.tar.gz.md5
 /usr/local/src root# cat snort-2.8.3.1.tar.gz.md5
f7c03b322dae31dafc45756630f2946c  snort-2.8.3.1.tar.gz
 /usr/local/src root# md5sum snort-2.8.3.1.tar.gz
f7c03b322dae31dafc45756630f2946c  snort-2.8.3.1.tar.gz
 /usr/local/src root# tar xzf snort-2.8.3.1.tar.gz
 /usr/local/src root# cd  snort-2.8.3.1

We are going to add in support to place alerts into a MySQL database. If MYSQL installed by the system, you can use the “–with-mysql” configuration option. In a previous post, “Introduction to MySQL,” we went through the installation of MySQL into the /usr/local/mysql directory. For such an installation, the –with-mysql-includes=/usr/local/mysql/include and –with-mysql-libraries=/usr/local/mysql/lib command options must be used. In order to use the dynamic plugin libraries, Snort needs to be able to find libmysqlclient.so. On some operating systems, you may have problems. Adding LDFLAGS=”-L/usr/local/mysql/lib/mysql” should work. If you do not need to compile in large file support, you can do the compilation simply with the command:

 /usr/local/src/snort-2.8.3.1 root# ./configure --prefix=/usr/local/snort \
--with-libpcre-includes=/usr/local/pcre/include \
--with-libpcre-libraries=/usr/local/pcre/lib \
--with-mysql-includes=/usr/local/mysql/include\
--with-mysql-libraries=/usr/local/mysql/lib

For adding in large file support, and having Snort use the libpcap installed under /usr/local/snort, do that with the following command:

 /usr/local/src/snort-2.8.3.1 root# CFLAGS="-D_LARGEFILE_SOURCE \
-D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64"  \
./configure --prefix=/usr/local/snort --with-libpcap-includes=/usr/localsnort/include \
--with-libpcap-libraries=/usr/local/snort/lib \
--with-libpcre-includes=/usr/local/pcre/include \
--with-libpcre-libraries=/usr/local/pcre/lib \
--with-mysql-includes=/usr/local/mysql/include\
--with-mysql-libraries=/usr/local/mysql/lib

After you configure Snort (with or without large file support), you continue to make and install it.

 /usr/local/src/snort-2.8.3.1 root# make
 /usr/local/src/snort-2.8.3.1 root# make check
 /usr/local/src/snort-2.8.3.1 root# make install
 /usr/local/src/snort-2.8.3.1 root# mkdir -p /usr/local/snort/etc
 /usr/local/src/snort-2.8.3.1 root# cp etc/* /usr/local/snort/etc
 /usr/local/src/snort-2.8.3.1 root# /usr/local/snort/bin/snort -V

   ,,_     -*> Snort! < *-
  o"  )~   Version 2.8.3.1 (Build 17)
   ''''    By Martin Roesch & The Snort Team: http://www.snort.org/team.html
           (C) Copyright 1998-2008 Sourcefire Inc., et al.
           Using PCRE version: 7.8 2008-09-05

Rules

Now we need some rules. For this example we will get the rules from the Snort and the Emerging Threats site. You will need to register for the rules at the Snort site. Do consider subscribing for the latest up-to-date rules. Registered users can only access rules 30 days after their release.

 root# cd /usr/local/src
 /usr/local/src root# wget http://www.emergingthreats.net/rules/emerging.rules.tar.gz
 /usr/local/src root# tar xzf emerging.rules.tar.gz

 /usr/local/src root# mkdir -p /usr/local/snort/rules
 /usr/local/src root# mv rules/* /usr/local/snort/rules
 /usr/local/src root# rmdir rules
 /usr/local/src root# wget \

http://www.snort.org/pub-bin/downloads.cgi/\

Download/vrt_os/snortrules-snapshot-CURRENT.tar.gz
 /usr/local/src root# md5sum snortrules-snapshot-CURRENT.tar.gz
184aed405da3f1043b82d81c98122237  snortrules-snapshot-CURRENT.tar.gz
 /usr/local/src root# mv  snortrules-snapshot-CURRENT.tar.gz /usr/local/snort/
 /usr/local/src root# cd /usr/local/snort/
 /usr/local/snort root# tar xzf snortrules-snapshot-CURRENT.tar.gz
 /usr/local/snort root# rm snortrules-snapshot-CURRENT.tar.gz
 /usr/local/snort root# vi /usr/local/snort/etc/snort.conf

Modify /usr/local/snort/etc/snort.conf to your environment. Make sure the RULE_PATH is set to /usr/local/snort/rules. If you wish to use the emerging threat rules, add:

include $RULE_PATH/emerging.conf

in the /usr/local/snort/etc/snort.conf file. Do not forget to adjust dynamicpreprocessor file and dynamicengine path. Mac OS X users will need to use the dynamic libraries. Uncomment the Mac OS X lines in the Snort configuration file.

Keeping Rules Current

If you want to setup a process to keep your rules as up-to-date as possible from the snort site, you will want to use the program Oinkmaster.

 root# cd /usr/local/src
 /usr/local/src root# wget http://prdownloads.sourceforge.net/oinkmaster/oinkmaster-2.0.tar.gz?download
 /usr/local/src root# md5sum oinkmaster-2.0.tar.gz
d2a1b56f51cf40e919c63206ca4ec8f8  oinkmaster-2.0.tar.gz
 /usr/local/src root# tar xzf oinkmaster-2.0.tar.gz
 /usr/local/src root# cd oinkmaster-2.0
 /usr/local/src/oinkmaster-2.0 root# cp oinkmaster.pl /usr/local/snort/bin
 /usr/local/src/oinkmaster-2.0 root# cp oinkmaster.conf /usr/local/snort/etc

Modify /usr/local/snort/etc/oinkmaster.conf changing “url=” to the location where you want rules to be archived. The URL will also require a Oink Code, which is available once you register with Snort and log into your account. Multiple URLs can be specified for multiple files. You can have Oinkmaster pull down rules from Snort and Emerging Threats. I use oinkmaster to pull down the latest rules, place them in new rules directory (ex: /usr/local/snort/new-rules), backup the previously rules, and send email about what is new. Some rules may have negative impact on Snort’s operation. It is unwise to automatically update the rules without review. Below, the backup/archive area is the directory /usr/local/snort/archive. Modify /usr/local/snort/bin/oinkmaster.pl to know where oinkmaster.conf is located. At that point you are ready to use oinkmaster to update your rules.

 /usr/local/src/oinkmaster-2.0 root# mkdir /usr/local/snort/archive
 /usr/local/src/oinkmaster-2.0 root# vi /usr/local/snort/bin/oinkmaster.pl
 /usr/local/src/oinkmaster-2.0 root# vi /usr/local/snort/etc/oinkmaster.conf
 /usr/local/src/oinkmaster-2.0 root# /usr/local/snort/bin/oinkmaster.pl \
-o /usr/local/snortr/new-rules -b /usr/local/snort/archive
 /usr/local/src/oinkmaster-2.0 root# crontab -l
10 6 * * * /usr/local/snort/bin/oinkmaster.pl -o /usr/local/snort/new-rules -b /usr/local/snort/archive \
| Mail -s "Snort Rule Updates" abbot@securitymonks.com 2>&1

For automatic updates, set up a cron job that runs the above command and email the results.

Testing Snort Using Attack Data

Leon Ward has made available a pcap file containing attacks that occurred back in 2001 against a honeypot. If you have other data that will produce interesting results, please feel free to use that.

 root# mkdir -p /data/ids/tcpdump
 root# cd /data/ids/tcpdump
 /data/ids/tcpdump root# wget rm-rf.co.uk/downloads/Honeynet-RFP-iis.tgz
 /data/ids/tcpdump root# tar xzf Honeynet-RFP-iis.tgz
 /data/ids/tcpdump root# /usr/local/snort/bin/snort -c /usr/local/snort/etc/snort.conf \
-A fast -l /data/ids/tcpdump -r ./Honeynet-RFP-iis.pcap
 /data/ids/tcpdump root# ls /data/ids/tcpdump/alert /data/ids/tcpdump/snort.log.*

Two results files should get created. The file /data/ids/tcpdump/alert will contain the alerts and /data/ids/tcpdump/snort.log.<date>, which contains the pcaps of the detected events.

SnortSP

At this point we are ready to install SnortSP. The first step is to make sure the system has the required software installed, which includes:

• Libdnet 1.10 or higher
• A recent Libpcap
• Lua 5.1.1 or better
• Depending o the OS, a UUID library

The second step involves getting the software, verifying, configuring, and installing the security platform. The engine will be built in the next section.

 root# cd /usr/local/src
 /usr/local/src root# wget http://www.snort.org/dl/prerelease/3.0.0-b2/\
snortsp-3.0.0b2.tar.gz
 /usr/local/src root# wget http://www.snort.org/dl/prerelease/3.0.0-b2/\
snortsp-3.0.0b2.tar.gz.md5
 /usr/local/src root# wget http://www.snort.org/dl/prerelease/3.0.0-b2/\
snortsp-3.0.0b2.tar.gz.sig
 /usr/local/src root# cat snortsp-3.0.0b2.tar.gz.md5
4b2259c08ebe66cf63d91359996c93d9  snortsp-3.0.0b2.tar.gz
 /usr/local/src root# md5sum snortsp-3.0.0b2.tar.gz
4b2259c08ebe66cf63d91359996c93d9  snortsp-3.0.0b2.tar.gz
 /usr/local/src root# gpg --verify snortsp-3.0.0b2.tar.gz.sig snortsp-3.0.0b2.tar.gz
gpg: Signature made Fri 18 Jul 2008 11:09:25 AM EDT using DSA key ID B10683B0
 /usr/local/src root# gpg --keyserver pgpkeys.mit.edu --recv-key B10683B0
gpg: requesting key B10683B0 from hkp server pgpkeys.mit.edu
gpg: key B10683B0: "Snort Release Team (Snort Release Team signing key) " not changed
gpg: Total number processed: 1
gpg:              unchanged: 1
 /usr/local/src root# gpg --verify snortsp-3.0.0b2.tar.gz.sig snortsp-3.0.0b2.tar.gz
gpg: Signature made Fri 18 Jul 2008 11:09:25 AM EDT using DSA key ID B10683B0
gpg: Good signature from "Snort Release Team (Snort Release Team signing key) "
gpg: Note: This key has expired!

Primary key fingerprint: 0A0A BD0C 8FAA BDDC E7A2  A8C3 E6FA 8BA1 B106 83B0

The next step is to configure, install, and test SnortSP installation. We will be using the /usr/local/snort area as the installation directory. Please adjust to your environment.

 /usr/local/src root# tar xzf snortsp-3.0.0b2.tar.gz
 /usr/local/src root# cd  snortsp-3.0.0b2
 /usr/local/src/snortsp-3.0.0b2 root# ./configure --prefix=/usr/local/snort
 /usr/local/src/snortsp-3.0.0b2 root# make
 /usr/local/src/snortsp-3.0.0b2 root# make check
 /usr/local/src/snortsp-3.0.0b2 root# make install
 /usr/local/src/snortsp-3.0.0b2 root# mkdir -p /usr/local/snort/etc/SnortSP
 /usr/local/src/snortsp-3.0.0b2 root# cp etc/* /usr/local/snort/etc/SnortSP
 /usr/local/src/snortsp-3.0.0b2 root# /usr/local/snort/bin/snortsp -V
SnortSP Version 3.0.0b2

Snort 2.8.2 Detection Engine

The next step is to install the Snort 2.8.2 detection engine. We will immediately shut it down with the ssp.shutdown() command. Please see previous sections “Libpcap and Large Files Support” and “PCRE (pcre-7.8).” I am going to include the configuration options for libpcap, pcre, and MySQL. Please adjust the configuration if necessary in the same way snort-2.8.3.1 was modified.

 /usr/local/src/snortsp-3.0.0b2 root# cd src/analysis/snort
 /usr/local/src/snortsp-3.0.0b2/src/analysis/snort root# ./configure --prefix=/usr/local/snort \
 --with-platform-libraries=/usr/local/snort/lib/snortsp \
 --with-platform-includes=/usr/local/snort/include \
 --with-libpcap-includes=/usr/localsnort/include \
 --with-libpcap-libraries=/usr/local/snort/lib \
 --with-libpcre-includes=/usr/local/pcre/include \
 --with-libpcre-libraries=/usr/local/pcre/lib \
 --with-mysql-includes=/usr/local/mysql/include\
 --with-mysql-libraries=/usr/local/mysql/lib
 /usr/local/src/snortsp-3.0.0b2/src/analysis/snort root# make
 /usr/local/src/snortsp-3.0.0b2/src/analysis/snort root# make check
 /usr/local/src/snortsp-3.0.0b2/src/analysis/snort root# make install
 /usr/local/src/snortsp-3.0.0b2 root# /usr/local/snort/bin/snortsp \
-L /usr/local/snort/etc/SnortSP/snort.lua
   ,,_     -*> SnortSP! < *-
  o"  )~   Version 3.0.0b2 (Build 9) [BETA]
   ''''    By Martin Roesch & The Snort Team: http://www.snort.org/team.html
           (C) Copyright 2008 Sourcefire Inc.
> Control thread running - 3086609296 (9501)
> ssp.shutdown()
 /usr/local/src/snortsp-3.0.0b2 root#

The command “snortsp -L /usr/local/snort/etc/SnortSP/snort.lua” is telling snortsp to use the file “snort.lua“. Examine the file for definitions of functions. Below are a few example functions calls that may be of interest:

 root# /usr/local/snort/bin/snortsp -L /usr/local/snort/etc/SnortSP/snort.lua
snort> ssp.help()
[*] SnortSP Commands:
    help()
    set_log_level( [debug|info|notice|warn|error|critical] )
    shutdown()
  Available subsystems within SnortSP have their own help() methods:
    dsrc        - Data Source
    eng         - Dispatcher/Engine
    analyzer    - Analytics Modules
    output      - Output Modules
  For example: dsrc.help() will call the Data Source help function
snort> dsrc.help()
[*]  Data Source Config Structure
 = {name        = ,
                  type        = ,
                  intf        = ,
                  flags       = [1|2|8],
                  command     = ,
                  snaplen     = ,
                  maxflows    = ,
                  maxidle     = ,
                  flow_memcap = ,
                  filename    = ,
                  max_count   = ,
                  display     = ,
                  gtp_support = [enable]
                  defrag      = {
                                 policy = <>,
                                 max_trackers = <>,
                                 timeout = <>,
                                 memcap = <>
                                }
}

    name: User string used to refernce the instantiated data
          source object.  If no name is provided a UUID is
          generated.
    type: Name of the DAQ module to use.  The list of available
          DAQ modules can be obtained using the
          dsrc.list_daq_modules() command.
    intf: Name of the interface to use for traffic acquisition.
    flags: Bitwise OR of available mode flags for a given DAQ.
           Some modes may not be available on all DAQs.
           Generally you will want to use either mode 1, 2
           or 8.  Default is 0.  To enable inline mode use a
           value of 10.
           Available modes are:
                1 = FILE READ MODE
                2 = PROMISC MODE
                8 = INLINE MODE
    command: BPF filter or other filter command for DAQs that
             support them.  Default is no filtering.
    snaplen: Max bytes to capture per packet.  Default is 0.
    maxflows: Max number of flows to keep in the flow manager
              at one time.  If this number is exceeded the least
              recently used flow is purged.  Default is 8192.
    maxidle: Number of seconds a flow can be idle before being
             timed-out and removed from the flow table.  Default is
             60.
    flow_memcap: Max number of bytes to allow to be used by the
                 flow table.  Default is 16MB.
    filename: In file playback mode, specifies the name of the
              file to run through the SnortSP instance.
    max_count: Number of packets to process before stopping this
               thread.  Default is 0, which is unlimited.
    display: Name of the display mode for printing packets as they
             are processed.  Available modes are:
                none    = Don't print packets at runtime.
                basic   = Display basic packet info on one line.
                classic = Display packets using the classic tcpdump
                          format.
                plus    = Classic mode + MAC header printout.
                max     = Full printout of every packet header
                          field, one field per line.
             Default is none.
    gtp_support: Activate support for the GTP protocol.  Default is
                 disabled.
    defrag: Specifies the IP/IPv6 defragmentation paramters.
                policy: Reassmebly policy for overlapping fragments.
                    Avaliable modes are:
                        
                    Default is bsd.
                max_trackers: The maximum number of unique packets
                              in the process of being reassembled at a
                              at any given time.  Default is 8192.
                timeout: The maximum time in seconds for a fragmented
                         packet to receive all fragments.  Default is 60.
                memcap: Maximum number of bytes to allow to be used for
                        fragment reassembly.  Default is 32MB.
[*] Data Source Commands
    new(  ) - Instantiate a new data source object
    delete(  ) - Delete a data source object
    list() - List all of the instantiated data source objects
    list_daq_modules() - List all of the available DAQ modules
    show( source_name ) - Show the config of the named
                          object
    help() - Print this message

  For example, the following commands will create a data source
  using the afpacket DAQ in inline mode bridging the eth2 and
  eth3 interfaces allowing 262144 flows to occupy 10 MBs of
  memory while timing out flows after 5 minutes of idle time:
      dsrc1 = {name="src",
               type="afpacket",
               intf="eth2:eth3",
               flags=10,
               snaplen=0,
               maxflows=262144,
               maxidle=300,
               flow_memcap=10000000}
      dsrc.new(dsrc1)

snort> dsrc.list()
[*] 0 data sources configured
snort> sniff("eth0")
Creating new data source
Flow manager created with 16384 flow capacity
Engine "e1" created
Linking engine "e1" to data source "src1"
Calling engine_start()

init_pcap: Initializing network interface eth0
[*] Data Source Config:
        Name: src1
        Type: pcap
        Interface: eth0
        Filename:
        Snaplen: 1514
        Flags: 0x00000002
        Display: None (0)
        Filter command:
        DAQ: 0x8078b00
        User Context: 0x9378170
        Max flows: 16384
        Max idle: 60
        Memcap: 10000000
[*] Flow Manager Config:
        Max flows: 16384
        Max idle: 60
        Memcap: 10000000
[*] DAQ config:
   Interface: eth0
   Snaplen: 1514
   Datalink: 1
   Count: 0
   Packet Count: 0
   Promisc flag: 1
   File flag: 0
   pcap ptr: 0x9396488
   analysis context ptr: 0xb71ce008
[*] Spawning engine thread!
snort> e1 thread running - 3072121744 (24685)

snort> dsrc.list()
[*] 1 data sources configured
Name: src1    DAQ: pcap    interface: eth0    Running
snort> dsrc.show("src1")
[*] Data Source Config:
        Name: src1
        Type: pcap
        Interface: eth0
        Filename:
        Snaplen: 1514
        Flags: 0x00000002
        Display: None (0)
        Filter command:
        DAQ: 0x8078b00
        User Context: 0x9378170
        Max flows: 16384
        Max idle: 60
        Memcap: 10000000
[*] Flow Manager Config:
        Max flows: 16384
        Max idle: 60
        Memcap: 10000000
[*] DAQ config:
   Interface: eth0
   Snaplen: 1514
   Datalink: 1
   Count: 0
   Packet Count: 117
   Promisc flag: 1
   File flag: 0
   pcap ptr: 0x9396488
   analysis context ptr: 0xb71ce008
snort> fsniff ("eth0", "port 21")
snort> dsrc.list()
[*] 2 data sources configured
Name: src2    DAQ: pcap    interface: eth0    Running
Name: src1    DAQ: pcap    interface: eth0    Running
snort> ssp.shutdown()
 root#

Lua

Lua is a lightweight scripting language embedded in SnortSP. The Snort Detection Engine is configured via a conf file, but SnortSP is configured by a Lua file. The Snort conf file location is specified in the Lua file. In this section, we are going to be running the shell script sspiffy.sh. The program comes with the SnortSP package and it will convert Snort 2.8.x files into a Lua scripts and a new configuration file for SnortSP.

Previously, we ran snort 2.8.3.1 against the Honeynet-RFP-iis.pcap data with the command:

 root# cd /data/ids/tcpdump
 /data/ids/tcpdump root# /usr/local/snort/bin/snort -c /usr/local/snort/etc/snort.conf \
-A fast -l /data/ids/tcpdump -r ./Honeynet-RFP-iis.pcap

We are going to create and use /data/ids/logs where result log files will be kept. First, backup the snort configuration file and result files previously created with snort 2.3.8.1.

 root# cd /usr/local/snort/etc
 /usr/local/snort/etc root# cp snort.conf snort.conf.orig
 /usr/local/snort/etc root# mkdir -p /data/ids/logs
 /usr/local/snort/etc root# mv /data/ids/tcpdump/alert /data/ids/logs/alert.orig
 /usr/local/snort/etc root# mv /data/ids/tcpdump/snort.log.* /data/ids/logs/

We defined dynamicpreprocessor and dynamicengine in snort configuration file (/usr/local/snort/etc/snort.conf) file, so we do not need to specify their location via command line with –dynamic-preprocessor-lib-dir and and –dynamic-engine-lib. We will use the /usr/local/snort/etc as our configuration directory. The script sspiffy.sh will create a new snort.conf file. To generate a Lua file to run SnortSP with snort, we would use sspiffy.sh as follows:

 root# cd /usr/local/snort/etc
 /usr/local/snort/etc root#  /usr/local/src/snortsp-3.0.0b2/sspiffy.sh \
/usr/local/snort -c /usr/local/snort/etc/snort.conf.orig \
-r /data/ids/tcpdump/Honeynet-RFP-iis.pcap -l /data/ids/logs -A fast 

Checking command line arguments ...

snort.conf is a copy of /usr/local/snort/etc/snort.conf.orig.2 with the following changes:
#SSP - set in lua: dynamicpreprocessor directory /usr/local/snort/lib/snort_dynamicpreprocessor/
#SSP - set in lua: dynamicengine /usr/local/snort/lib/snort_dynamicengine/libsf_engine.so
#SSP - deleted: preprocessor frag3_global: max_frags 65536
#SSP - changed: preprocessor frag3_engine: policy first detect_anomalies

snort.lua is the script that configures and executes SSP.

To run SSP as configured:
/usr/local/snort/bin/snortsp -L snort.lua -P
 -S  

 /usr/local/snort/etc root#  ls snort.lua snort.conf
 /usr/local/snort/etc root#  vi snort.lua snort.conf

The two files that were created in your current directory are snort.lua and snort.conf. Take a look at the files and modify them where appropriate. Make sure the full path to snort.conf is specified in snort.lua (conf=”/usr/local/snort/etc/snort.conf”). To run snortsp with the generated files, use the command:

 /usr/local/snort/etc root#  /usr/local/snort/bin/snortsp -L /usr/local/snort/etc/snort.lua

Now go to the /data/ids/logs area and compare the alert file just generated with the alert.orig file generated by snort 2.8.3.1. The results do match. The differences between the files comes down to SnortSP alert file not including log entries for http_inspect.

Final Thoughts

The new Snort architecture is a very exciting design change that will help future versions of Snort respond quickly to an every changing security battleground. Besides the potential speed performance, along with the ability to take advantage of the multicore systems being developed, what I find particularly interesting is that the new architecture provides a framework where developers can build their applications. There is real potential here for intriguing development and flexibility. Different environments have different requirements and needs. Soon organizations will have the ability to choose from different engines resulting in custom configurations that best address each organization’s risks.

Everyone is looking for a turn key solution. People need to accept the fact that security is hard. By the time you find a solution, the problem has changed. With an ever changing attack vector, there are no turn key solutions that can do anything more than provide help. This is an important concept, because once accepting this fact, it changes a person’s view on viable solutions. That is what I like about the Snort 3 architecture. It is designed to be capable of being adjusted rapidly, which is a really cool concept. All security tools should be designed in such a manner. Our adversaries adjust quickly. We need to be able to also. On the eve of Snort turning ten, it is good to see that the Sourcefire team has been building such a promising IDS that will serve us well far into the future.

Samuel Langhorne Clemens, better known as Mark Twain, once wrote, “Rumors of my death have been greatly exaggerated!” It seems worldwide IDS/IPS revenue grew 19% in 2006. In 2007, the IDS/IPS marked was a $932 million industry. Frost & Sullivan expects the market to grow to $2.1 billion in the next five years, citing “complex attacks, ongoing vulnerability discoveries, and the need for companies to comply with new legislation” as major contributing drivers. Of course these numbers are focused on the commercial side of IDS/IPS.

“Computers are like Old Testament gods; lots of rules and no mercy,” wrote Joseph Campbell, an American mythology professor, writer, and lecturer. I often think of this quote when it comes to the use of signature based solutions in security. As I previously posted, IDS/IPS technology is moving aware from being solely signature based, to a blend of signature based, anomaly detection, and activity based methodologies. The Race to Zero contest is being held right now (August 8-10th) during Defcon 16. It is a hacking competition where known viruses will be tweaked in an attempt to foil signature-based blacklists of several major antivirus engine. The point is to demonstrate how easy it is to get around signature based solutions.

Liam Tung, over on ZDnet has written a very good article titled, “Signature-based antivirus is dead: Get over it“. In the article, Simon Clausen, founder & CEO at PC Tools, reports that the security industry has been looking beyond blacklists. “I would very much disagree that AV is dead. Really, traditional signature-based AV is going to be dead in a few years, but what every antivirus company is evolving towards, like us, is behavioural AV technology, so AV will be alive.”

Martin Roesch, CTO and Founder of Sourcefire, has posted concerning the upcoming Snort 3 architecture that has as a key component a contextually aware engine. This will add to Snort the ability to understand what it is defending. Snort 3 architecture is built around the concept of network context, which is essentially data about the environment and the composition of the hosts in the network as well as the local network composition. The software framework is called SnortSP (the Snort Security Platform) and the initial beta has been released. By leveraging network context, Roesch hopes to reduce/simplify/eliminate tuning as much as possible, be able to generate event priorities, and address network and transport layer evasion.

Peter Judge, in his ZDnet article “Sourcefire: Don’t Snort at open-source security,” quotes Roesch talking on the future of the security market:

Threat management has three phases. Before the threat, the firewall and patch management should prevent threats; during the attack, the IPS should block them; and, afterwards, network-behaviour analysis [NBA] should reveal damage and remedy it. These tend to be stove-piped technologies, where nobody talks to anyone. There are not enough people. You have got to get people out of the equation. You have to automate.

The US government, through the Department of Homeland Security, has issued a Request for Information (RFI) which highlights the analytical skills that DHS is seeking from a staffing perspective. Through this RFI, one can determine the technological focus DHS has for the administration’s requested $294 million fiscal cybersecurity budget. The 2009 cybersecurity budget is a significant increase over the 2008 enacted budget of $210 million. Ben Bain, in his article “DHS seeks cybersecurity capability info” quotes Alan Paller, research director at the SANS Institute, as saying that most DHS department employees don’t know how to do complex intrusion detections, log analysis or reverse engineering malware. DHS is looking for folks with experience with EINSTEIN data analysis, tools, techniques and network flow analysis capabilities, the TIC deployment environment, and compliance metrics. Ben Bain reports on the highly classified Comprehensive National Cybersecurity Initiative (CNCI) that DHS is “planning on implementing a new version of the intrusion detection and alert system — EINSTEIN 2 — designed monitor agencies’ Internet access points for malicious activity and capture intrusion data along with data transmitted in proximity to an alert.” It sounds like the government believes there is still life in some form of intrusion detection technology.

While I have mentioned Snort, it is not the only IDS/IPS. Keeping to the world of open source, most of my IDS/IPS time is split between Snort and Bro. I am very interested in Roesch’s work on the contextually aware engine. It will make Snort a more powerful tool. Now Bro has advance features giving it the ability to discern network anomalies that are caused by hostile activity. Bro also has some ability to detect violations of expected traffic rules to defend against previously-unknown attack techniques. For additional information on Bro, there are three blogs particularly helpful. Seth Hall of The Ohio State University has started the “A Bro Blog.” The ICSI Networking Group has a blog with contributions from the big names in Bro: Mark Allman, Sally Floyd, Christian Kreibich, Vern Paxson, Robin Sommer, and Nicholas Weaver. C.S.Lee (geek00L) on his site “When {Puffy} Meets ^RedDevil^” will frequently do postings involving Bro.

I started this post asking if Gartner was correct and are IDS/IPS a dying technology? By looking at both the IDS/IPS market and the government’s cyber security focus, we see the technology is very much alive. Not so much much because Gartner was wrong about IDS/IPS producing false positives and negatives, that monitoring puts a burden on the organization, that incident response can be a taxing process, nor that being able to monitor an increasingly higher bandwidth is challenging. Those problems still exist. Because of that, there is much work being done to help the IDS/IPS industry handle data more flexibly, efficiently, or accurately. Gartner was wrong with the assumption that the technology would remain stagnant and if something is difficult, it will not be implemented. IDS/IPS exist because organizations need the detection and prevention capabilities. Companies face an ever increasing reliance on information technology for competitive advantage while dealing with increasingly complex attacks, ongoing vulnerability discoveries, and the need for companies to comply with new legislation. IDS/IPS solutions continue to be part of organizations’ security programs because the technology continues to evolve while integrating new solutions. In the world of evolution, adaptability will trump better design if the better design is incomplete and inflexible in an ever changing environment.

While one should not rush to be an early adapter, it is good to keep in mind what is coming down the pike. If you are doing work with an network intrusion detection system, such as Bro, FreeBSD 7 looks to have some key performance improvements that makes it a solid choice. The post titled, “FreeBSD 7 will be revolutionary” made this observation:

Also of importance are the improvements to the networking stack. With gigabit (or faster) network cards being the norm these days, FreeBSD’s support for TCP/IP Segmentation Offload (TSO) and Large Receive Offload (LRO) will no doubt prove to be very useful. Along with the new sendfile() implementation, and the improved sosend() functionality, we will likely see some large networking performance boosts.

The latest developments in FreeBSD can be found at the “What’s cooking for FreeBSD 7?” page. Several performance improvements are outlined by Kris Kennaway of the FreeBSD Project in his presentation titled, “Introducing FreeBSD 7.0.’ The presentation compares performance increases shown by PostgreSQL and MySQL. These database packages utilize some complex operations that demonstrate the improvements under FreeBSD 7.

How does that help Bro? That requires some examination and explanation. From the “Hardware and Software Requirements” section of the Bro Wiki, it states:

Operating System Recommended: FreeBSD. Bro works with many Unix systems, including Linux and Solaris, but has been primarily tuned for FreeBSD. We currently recommend using FreeBSD version 4.10 for Bro. If your site has a large number of packets or connections per second you shouldlook at the section on Hardware and OS Tuning. FreeBSD 5.x should work, but is not quite as fast as 4.10.

Before you start looking for version of FreeBSD 4.10 on ebay, there have been significant improvements made in later versions of FreeBSD. Note that there is no date associated with this Wiki entry. The lack of mention of FreeBSD 6.x indicates the entry was made prior to 6.x being released. This appears to be an outdated post. To provide a better sense of performance between the various FreeBSD versions, Chris Buechler wrote the blog entry titled “Network Performance Update.” Chris describe, in relation to the pfSense firewall/server platform:

m0n0wall 1.2 still makes us look silly (1.5 times as fast), but that’s to be expected with its FreeBSD 4.x base. FreeBSD 6.2 has closed that gap considerably from the disaster that was FreeBSD 5.x, and FreeBSD 7 looks to draw nearer to 4.x performance. Note that I’m strictly talking about single processor machines, SMP systems are a much different story, but I won’t comment on those until I get a chance to do some testing.

Holding the comparisons to a single processor machine is an important point. There have been discussions on the Bro mailing list concerning whether multiple processors are helpful. How FreeBSD performs with multiple processors depend on what version is being used. We will examine that a little later. Robin Sommer, one of the Bro developers stated, “All of the main analysis is done in a single process and not able to make use of multiple CPUs.” It was reported that top-of-the-line dual Xeon CPUs (>$4,000 of CPU) performed ~5% better than a single PentiumD at under $500.

What version and hardware setup are the Bro developers using and what recommendation would they make on tuning the operating system? The article titled, “Operational Experiences with High Volume Network Intrusion Detection” by Holger Dreger, Anja Feldmann, Vern Paxson, and Robin Sommer stated that the system they were using for high volume network intrusion detection was “the primary NIDS monitor is a Dual Athlon MP 1800+ with 2 GB Memory, currently running FreeBSD 5.2.1. It is connected via a Gigabit Ethernet.” They tested it against, “The others are separate Athlon XP 2600+ based systems with 1 GB of RAM running Linux 2.4.” Their goal was to compare tuning parameters. In Robbin’s thesis, he states in order to get the performance they desired they, “patched the kernel to increase the NIC driver’s internal receive buffers. Moreover, we patched the packet-capture sub-system to increase its buffers by three orders of magnitude.”

Kris Kennaway did a very interesting presentation comparing FreeBSD 4.x, 5.x, and 6.x titled “Filesystem Performance on FreeBSD.” Kris tested on a 4 CPU AMD64 system, so he could only test 5.4 vs 6.0 (the latest release at that time). The results will depend on what is being tested. For full details, please view the presentation. An interesting result was that FreeBSD 6.0 performed 30% faster than 4.11 for concurrent writes. 6.0 was 15% faster than 5.4 for concurrent reads. Kris has also done some performance test involving BIND and found FreeBSD 7 had a 60% higher peak performance over version 6.1. The point is, it would appear that later version of FreeBSD made significant performance improvements over FreeBSD 5.2.1 (the version used by the Bro development team in their paper). FreeBSD 5.x and FreeBSD 6.x where to some degree transitional operating systems moving 4.x to 7.x.

To help make all these performance reports make sense, let’s begin with an overview/history of the versions pulling liberally from Kris’s presentation, “Introducing FreeBSD 7.0 “:

FreeBSD 4.x is a single-threaded kernel with limited multiprocessor support.

• Able to run user code on multiple processors
• Only one process at a time can execute in the kernel (“Giant lock” around entire kernel)
• Device interrupts may be processed in parallel, subject to some constraints

The historical BSD kernel architecture worked very well for single-processor systems. It fundamentally does not scale to multi-processor systems, which are now becoming universal.

In the Bro discussion list, performance is often discussed. The FreeBSD group started working on multiprocessor support with the SMPng project. Bascially, we see the development in FreeBSD 5.x and 6.x:

FreeBSD 5.0-5.2.1 (2003-01-17 – 2004-02-22)

Debut of the new architectural model for symmetric multiprocessor support in FreeBSD.

FreeBSD 5.3 (2004-11-06), 5.4 (2005-05-09)

• The fundamental architectural changes were largely in place
• Some initial progress with kernel parallelism by 5.3 and 5.4 (network stack, virtual memory, …)”

SMPng was improved in 6.x:

FreeBSD 6.0 (2005-11-01), 6.1 (2006-05-08), 6.2 (2007-01-15)

• Stabilized the work of the 5.x branch
• Performance benefits from subsequent development work
  e.g. Virtual File System (VFS) and Unix File System (UFS) now allow parallel access
• Large parts of the kernel may now operate in parallel, with significant performance gains on many common workloads

With FreeBSD 7.x, the kernel will be a fully parallel system. The “Giant lock” is no longer present on almost all possible workloads. Major shift of focus from correctness to optimization. The above mentioned document will demonstrate impressive results. The document lists the following improvements:

New filesystems

• ZFS
• Sun’s amazing new filesystem moves the goalposts. Stay tuned for more in the presentation from Pawel.
• unionfs: overlay multiple filesystem hierarchies into one. Broken for many years but now usable again.
• XFS support (read-only)
• CODA distributed filesystem support fixed
• UFS quotas are now parallelized
• NFS client and server parallelized
• Performance improvements for NFS client
• SCSI layer (CAM) is now parallelized, including many drivers. Performance benefits for SCSI device access.
• iSCSI initiator (in base system) and target (in ports), allowing remote exporting and local mounting of SCSI devices over TCP/IP

New GEOM (pluggable storage layer) modules

• gjournal; block level journalling provider (can be used with UFS for journalling support)
• gvirstor; virtualized storage provider (create a huge disk image sparsely populated with disks, add more later)
• gcache; read cache for storage layers with small request sizes
• gmultipath; support for multiple paths to the same storage provider (fiber channel, etc)
• gpart; virtualized partitioning support (GPT, APM, …)

Network Stack Changes

• Complete elimination of giant lock from network stack
• On-going cleanup and development work
• Socket buffer automatic sizing; dynamically responds to network conditions for improved throughput
• SCTP (Stream Control Transmission Protocol)
• Migration from KAME IPSec to Fast IPSec
  • Improved performance
  • Hardware acceleration with cryptographic accelerators
  • Both IPv4 and IPv6
• Direct dispatch of inbound network traffic
  • Avoids context switching, improves CPU cache locality, allows concurrency
  • Significant performance benefits on many workloads
• Optional in-kernel Just-In-Time compiler for Berkeley Packet Filter (BPF) programs (tcpdump, etc)
• In-kernel Network Address Translation (NAT) modules for natd(8)
• Link aggregation (create virtual interfaces for fault tolerance and higher capacity)
• Rapid spanning tree protocol support

Network Drivers

• Support for commonly encountered 10 gigabit ethernet drivers: Chelsio (cxgb), Intel (ixgbe), Myricom (mxge), Neterion (nxge)
• Transmit Segmentation Off-load (TSO)/Large Receive Off-load (LRO); off-load send/receive into the ethernet driver
• New devices supported

Wireless

• Wireless 802.11 layer is stable
  • high power ath cards (Senao, Ubiquiti, Wistron)
  • 900MHz ath cards (Ubiquiti, Zcomax)
  • ath (Atheros), iwi, ral (Ralink), ural (RT2500USB) drivers are high quality
• New drivers
  • rum (Ralink RT2500USB, RT2601USB)
  • Intel wireless drivers: ipw (Intel PRO/Wireless 2100), iwi (2200BG/2225BG/2915ABG) works out of the box
  • ZyDAS ZD1211/ZD1211B
• WPA (Wifi Protected Access) support stable
• New scanning support (background scanning, roaming)
• Atheros protocol extensions 802.11n support (forthcoming standard)
  • higher performance: up to 135 Mb/sec, channel bonding, improved range, etc
  • drivers not yet committed
• Preparation for future changes (virtual access points, etc)

New CPU Architectures

• Improved support for ARM architecture
  • Improved AT91RM9200 (Atmel) support
  • support for Avila Gateworks Xscale boards was added, including a rewrite of the Intel code
  • permission from Intel to bundle u-code
  • Boot loader can load from Secure Digital (SD) flash cards
  • FreeBSD/ARM used as the basis for growing number of embedded devices
• Sun Ultrasparc T1 (preliminary)
  • 8 cores, 4 threads per core = 32 logical CPUs per package
  • A very interesting new CPU architecture, and one to watch in the future
  • T2: 8 threads * 8 cores = 64 logical CPUs per package!
• X-box!

Security Subsystems: Audit subsystem

• Fine-grained, configurable logging of security-relevant events: System calls, application and user space activities
• Now available by default in GENERIC kernel
• Originally developed for Mac OS X, ported and enhanced. A nice example of code-sharing in the Apple ! FreeBSD direction
• Builds on the other advanced security features developed by the TrustedBSD project for FreeBSD priv(9) API
• common interface for kernel privilege checking
• Privilege model can be modified by Mandatory Access Control (MAC) modules

User-level Changes

• Many updates to system applications and utilities
• cached: caches queries to nsswitch (\name service switch”: user/group/host lookups) for improved performance
• Ports collection
  • Currently contains 17692 ported third-party applications (1774 more than 6.2)
  • Major changes since 6.2:
    • X.org 7.3 (many improvements, e.g. working composite support for improved visual effects)
    • KDE 3.5.7
    • GNOME 2.18.3
    • More than 24000 other changes and updates

Performance

• Performance optimizations throughout the system
• The ULE scheduler is now recommended instead of the historical 4BSD scheduler
  • Better interactive performance on desktop systems
  • Significantly better performance on SMP systems
  • 4BSD will remain the default scheduler in 7.0 to be conservative, but likely to switch for 7.1
• If you find a workload that FreeBSD 7.0 performs poorly on, we want to hear about it!

Other Kernel Changes

• Partial linux 2.6.16 emulation support (not enabled by
  default)
• Support for Message Signaled Interrupts (MSI) and Extended Message Signaled Interrupts (MSI-X)
• IPMI (Intelligent Platform Management Interface); monitoring system hardware.
• Improved support for legacy-free hardware (e.g. MacBook pro)
• FireWire support for the boot loader
• Asynchronous I/O (AIO) support is parallelized: used by e.g. qemu
• New pseudo-tty system, allocates on demand without built in limits, and without requiring root privilege

Development Tools Internal

• GCC 4.2.1
• Improvements to hwpmc (CPU performance counters)
• symbol versioning added to several libraries
• New scalable malloc(3) (jemalloc)
• Optimized kernel locking primitives (sx, rwlocks)
• POSIX message queues
• contigmalloc(9) with buddy allocator
• kernel malloc(9) red zone debugging support
• Improved kernel lock profiling infrastructure
• mini-dumps

It looks like FreeBSD has some nice changes that should be released shorty. The changes will position FreeBSD well to deal with the high network demand that a Bro box may encounter.

Last week I spent Monday driving through a few states. It was an eight hour drive. When possible, I prefer driving over flying. While it may take longer, I use the time to listen to podcasts. Since I had taken the SANS System Forensics, Investigation & Response course (SEC 508), I had access to their lectures in MP3 format. The lecture on Computer Investigative Law for Forensic Analysts was prepared and taught by Richard P. Salgado. I had taken the course at a Community SANS event, close to where my brother lives. Yes, I was trying to keep my expenses down, and my brother and his family were kind enough to put me up for the week. While the course was well taught, knowledge of the legal issues of forensics was not the instructors strong point. This was reflected by the fact that the students hated that day. If only they had Richard P. Salgado. He did an amazing job.

Why am I mentioning this on a blog posting on intrusion detection systems (IDS)? The law has an ever increasing role in IT. This is especially true in the area of forensics, incident response, and intrusion detection/prevention. Before you setup any IDS system, make sure you are authorized and legally clear to do so.

With that disclaimer out of the way, I spent the weekend beginning to develop a network monitoring system. Sure, for years I have worked with Snort, but I am doing something different. For those unfamiliar with Snort, to quote their site:

Snort® is an open source network intrusion prevention and detection system utilizing a rule-driven language, which combines the benefits of signature, protocol and anomaly based inspection methods. With millions of downloads to date, Snort is the most widely deployed intrusion detection and prevention technology worldwide and has become the de facto standard for the industry.

It is a great product. Along with Snort, I have used the Basic Analysis and Security Engine (BASE), which is based on the Analysis Console for Intrusion Databases (ACID) project. BASE provides a web front-end to query and analyze the alerts coming from a SNORT IDS system. If you are pulling down software, I also suggest checking out Sguil.

Richard Bejtlich recently posted on his blog, TaoSecurity, an entry titled “DHS Einstein Demonstrates Value of Session Data.” Richard makes the statement, in relation to collecting session data:

This is just the sort of project I’d like to roll out at my new job, possibly combining Argus with ArgusEye, or maybe just Sguil without Snort.

An intriguing project. This weekend was about setting up an IDS system using Bro. To understand the importance of Bro, you need to first review the different styles of intrusion detection.

• Signature Based – looks for specific, known attacks.
  • Pros: good attack libraries, easy to understand results.
  • Cons: unable to detect new attacks or even just variants.
• Anomaly Detection – build/infer a profile of “normal use” and flag deviations.
  • Pros: potentially detects wide rand of attacks, including previously unknown types of attacks.
  • Cons: can be “trained” to accept attacks as normal, and potentially misses a wide rand of attacks including known attacks.
• Activity Based – inspect traffic and construct “events,” look for patterns of activity that deviate from a site policy.
  • Pros: potentially detects wide range of attacks (including novel), framework can accommodate signatures and anomalies.
  • Cons: policies/specification require significant development and maintenance and harder to construct attack libraries

Snort is a signature based IDS. Bro is an activity based IDS, though it does include a signature engine for matching specific patterns in packet streams. Bro is compatible with Snort. somewhat. With Bro analysis, signature matches generate events which are amenable to high level policy script processing rather than direct alerts. Other difference include that Snort is user friendly and Bro is a beast to learn. Worse still, there are no good guides for Bro. Sure, you can subscribe to the mailing list and there is a Bro Wiki. Geek00l has done some very good postings:

• Regex – Magic for NetSe[x|c]Anal(yst)?
• Bro-IDS: Enable Full Content Data Logging
• Time Machine – Payload Centric
• Bro-IDS v1.2
• Bro-IDS – Signature Matching
• FreeBSD – IDS Sensor Tweaking
• Bro-IDS – The learning process
• Multipurposes post :]
• Bro-IDS – Be Loved
• Bro-IDS – Installation Experience

Geek00l convinced Richard Bejtlich take a second look at Bro, and Richard posted:

• Bro Basics Follow-Up
• Bro Basics

That will get you started.

My interest in Bro comes from the fact that a design goal of Bro was to handle high speed, large volume monitoring. Snort, on a security appliance, can handle such traffic. Force10 released such a box, the P10, which can handle up to 1000 signatures. I have worked with the open source version of Snort on high volume networks, and it has not been pleasant. While the P10 might work well, I am interested in different capabilities.

Bro offers an interesting solution to handling monitoring on 10G traffic. If you are working with FreeBSD, there are ways to tune the kernel. While I have previously run into problems with Bro, my past problems were more likely due to trying to work under the Apple environment. Supported 10G Ethernet cards drivers had not yet been developed. Fortunately, that appears to have changed. I’ll post more as I make progress.