Confused over the different functionality and information IDS/IPS engines provide? Are you looking for an open source solution? There are options, each with different features and strengths. To help determine what might work best in your environment, we will walk through the setup of three popular open source intrusion detection packages: Bro, Snort, and Suricata. Keep in mind, IDS/IPS engines are not simple setup and leave security solutions. The systems need to be configured and tuned to the unique environment they are operating in. While configuration and tuning are essential tasks, it will be considered outside the scope of this post. In later posts, we will tune the systems and examine the results as we test out sample malicious traffic captures. After trying the different open source IDS/IPS tools, the reader will be in a better position to judge how the software might meet the particular requirements of their organizations.
There has been a bit of a verbal flare up between the folks behind Snort and Suricata. Matt Jonkman, founder of Emerging Threats and OISF’s president, recent statement that “[Intrusion detection technology] has been stagnant for the last five years”  did not sit well with Martin Roesch, Snort’s creator. Roesch questioned Suricata’s stated benefits when he responded, “OISF has wrapped Suricata in some cool computer science concepts, but they have not delivered on their vision. [Suricata] offers a sub-set of Snort’s functionality at a fraction of its performance.”  Roesch went on to say, “They’ve produced a clone of Snort that performs worse at taxpayer’s expense.”
Matt Olney, a senior research engineer for the Sourcefire Vulnerability Research Team (VRT) addressed the multi-threaded benefit when he wrote, “Trust me, if multi-threading were the answer, the industry would have moved there in short order.”  Olney went on to quote results of an internal test pitting Snort against Suricata, “With rules loaded, Suricata runs up to about 200MB per second. Snort, with rules, hits 894MB per second with no drops.”
Jonkman questioned those finding when he wrote, “Those stats are ridiculous, and they refuse to publish details of the equipment and configuration used.” Jonkman goes on to explain, “We know that we’re not, right now, cycle for cycle, faster than Snort … but we’re getting six times the performance as Snort on the same hardware, with version 1.0.” Victor Julien, lead developer of Suricata, explain , “Is Suricata faster than Snort on a single core cycle for cycle, tick for tick? No. Itâ€™s pretty clear we arenâ€™t, I didnâ€™t expect us to be either. But we scale. Weâ€™ve had reports of running on a 32 core box and scaling to use all cores.” Russ McRee, a senior security analyst / researcher and founder of holisticinfosec.org, adds, “Consider that an unnamed military body has tested Suricata versus Snort on a large scale platform (24 processors and 128GB of RAM) and saw a very clear 6-fold speed increase over a tuned Snort implementation on the same platform.”
Russ McRee article on Suricata in August’s ISSA Journal  contained a table comparing features, which we will add Bro information to:
|Complete IPv6 Support||Yes||Some||Complete|
|IP Reputation||Somewhat||No||Yes (soon)|
|Automated Protocol Detection||Yes||No||Yes|
|GPU Acceleration||No||No||Yes (soon)|
|Global Variables/Flowbits||Yes||No||Yes (soon)|
|Inline Windows Support||No||No||Yes|
|GeoIP Lookups||Yes||No||Yes (soon)|
|Advanced HTTP Parsing||Yes||No||Yes|
|HTTP Access Logging||Yes||No||Yes|
|SMB Access Logging||Planned||No||Yes (soon)|
|HTTP Blocklist Lookups||Yes||No||Yes (soon)|
Seth Hall, Information Security Detection-Response Architect at GE (and one of the top Bro developers), addressed the above table and pointed out some of the strong features of Bro. Seth writes:
- Multithreaded processing: Work is ongoing on this, but nothing releasable yet. Bro does have a fully functional cluster deployment model which helps users to scale support on a single box and/or across multiple boxes.
- IPv6 Support: Due to a bug, which hopefully will be addressed by the next release, IPv6 support is unusable in large scale production.
- IP reputation: You could say that Bro has IP reputation, it’s easy to utilize lists of addresses at least. I’m going to be working heavily on an intelligence sources framework for Bro soon too which will be able to consume a wide range of intelligence sources including IP addresses.
- Automated protocol detection: There’s even an academic paper about it  if you’d like to find out exactly how it works.
- Global variables/flowbits: Bro support for this sort of thing is far beyond what anything else has inherently because Bro has a complete programming language.
- GeoIP lookups: I added that myself several years ago. Bro supports IPv6 geoip lookups in addition to IPv4 and ASN lookups using another database for libGeoIP.
- Advanced HTTP Parsing: Bro has had it for years.
- HTTP Access Logging: Definitely. My script  will be included in the next release too.
- SMB Access Logging: This is something that I’m planning on tackling soon. I don’t know what the level of support for SMB is currently, but there is a parser already.
- HTTP Blocklist lookups: Yes, I consider this similar to the IP reputation and it’s going to be included in the intelligence sources framework. Some usage of URL lists is already included in a script that I distribute separately  but which will be in the next release of Bro.
- Free: Bro is under the BSD license, so in my opinion it’s actually more free than Snort or Suricata which are both under the GPL and much more difficult to share code with.
While Snort and Suricata have been a bit in the public spotlight recently, the developers of Bro have stayed clear of the recent verbal debate. Bro is the third open source IDS/IPS engine we will be working with. It is primarily funded by the National Science Foundation’s Strategic Technologies for the Internet program. Robin Sommer this week announced  that the International Computer Science Institute (ICSI) and the National Center for Supercomputing Applications (NCSA) have been awarded a grant of almost $3M for extensive Bro development. To quote Robin:
The funded project aims specifically at addressing much of the feedback that we have received from Bro users over the years. It will enable us to refine many of the rough edges that the system has accumulated over time[*], improve Bro’s performance significantly, and also make it much easier for the community to contribute to the project.
Expect some interesting work from the Bro camp. Some of Bro’s current stated features and benefits  include:
- Network Based: Bro is a network-based IDS. It collects, filters, and analyzes traffic that passes through a specific network location. A single Bro monitor, strategically placed at a key network junction, can be used to monitor all incoming and outgoing traffic for the entire site. Bro does not use or require installation of client software on each individual, networked computer.
- Rich Application-Layer Analysis: A primary feature of Bro is that it includes detailed, parser-driven analysis of many popular application protocols. The output of these analyzers is a stream of events that describe observed activity in semantically rich, high-level terms. These events themselves do not constitute security alerts, but rather provide the input for further, stateful processing using Bro’s custom scripting language.
- Custom Scripting Language: Bro policy scripts are programs written in the Bro language. They contain the “rules” that describe what sorts of activities are deemed troublesome. They analyze the network activity and initiate actions based on the analysis. Although the Bro language takes some time and effort to learn, once mastered, the Bro user can write or modify Bro policies to detect and alert on virtually any type of network activity.
- Pre-written Policy Scripts: Bro comes with a rich set of policy scripts designed to detect the most common Internet attacks while limiting the number of false positives, i.e., alerts that confuse uninteresting activity with the important attack activity. These supplied policy scripts will run “out of the box” and do not require knowledge of the Bro language or policy script mechanics.
- Powerful Signature Matching Facility: Bro policies incorporate a signature matching facility that looks for specific traffic content. For Bro, these signatures are expressed as regular expressions, rather than fixed strings. Bro adds a great deal of power to its signature-matching capability because of its rich language. This allows Bro to not only examine the network content, but to understand the context of the signature, greatly reducing the number of false positives. Bro comes with a set of high value signatures policies, selected for their high detection and low false positive characteristics.
- Network Traffic Analysis: Bro not only looks for signatures, but can also analyze network protocols, connections, transactions, data amounts, and many other network characteristics. It has powerful facilities for storing information about past activity and incorporating it into analyses of new activity.
- Detection Followed by Action: Bro policy scripts can generate output files recording the activity seen on the network (including normal, non-attack activity). They can also generate problem alerts to event logs, including the operating system syslog facility. In addition, scripts can execute programs, which can, in turn, send e-mail messages, page the on-call staff, automatically terminate existing connections, or, with appropriate additional software, insert access control blocks into a router’s access control list. With Bro’s ability to execute programs at the operating system level, the actions that Bro can initiate are only limited by the computer and network capabilities that support Bro.
In my previous post, “Snort 3: The Next Generation” , Marty provided a roadmap of where Snort is heading. While changes have since been made as Snort develops, the philosophy remains the same. Sourcefire is moving forward on a solid security framework. See the recent work on the Sourcefire’s Razorback framework  and . Olney described Razorback in this way , “It isn’t Snort, it isn’t ClamAV, and it isn’t Suricata. It’s a new approach to the detection problem, and was built from the ground up in close collaboration with groups that are facing APT-level threats. It may not be perfect, it may not even be the right answer (but we think it is), but it is truly innovative.”
More immediate, examine the the improved features Snort 2.9.0 Beta :
- Updates to HTTP Inspect to extract and log IP addresses from X-Forward-For and True-Client-IP header fields when Snort generates events on HTTP traffic.
- Updates to SMTP preprocessor to support MIME attachment decoding across multiple packets.
- Updates to the Snort packet decoders for IPv6 for improvements to anomaly detection.
The new features include:
- Feature rich IPS mode including improvements to Stream for inline deployments. Additionally a common active response API is used for all packet responses, including those from Stream, Respond, or React. A new response module, respond3, supports the syntax of both resp & resp2, including strafing for passive deployments. When Snort is deployed inline, a new preprocessor
has been added to handle packet normalization to allow Snort to interpret a packet the same way as the receiving host.
- Use of a Data Acquisition API (DAQ) that supports many different packet access methods including libpcap, netfilterq, IPFW, and afpacket. For libpcap, version 1.0 or higher is now required. The DAQ library can be updated independently from Snort and is a separate module that Snort links to.
- A new rule option ‘byte_extract’ that allows extracted values to be used in subsequent rule options for isdataat, byte_test, byte_jump, and content distance/within/depth/offset.
- Two new rule options to support base64 decoding of certain pieces of data and inspection of the base64 data via subsequent rule options.
- Added a new pattern matcher that supports Intel’s Quick Assist Technology for improved performance on supported hardware platforms. Visit http://www.intel.com to find out more about Intel Quick Assist.
Last, but not least, several characteristics OSIF report Suricata has to handle today’s threat are :
For additional background information, I have written several past posts on IDS/IPS (see “Suricata: A Next Generation IDS/IPS Engine” , “Installing Bro IDS 1.4″ , “Snort 3: The Next Generation” , â€œBlacklisting with Snortâ€, , “IDS/IPS: The Mark Twain of the Security World” , and “IDS” ). I will not repeat that information in this post.
We will be setting these software packages up to be used on a development machine. The exchange between the Snort and Suricata was focused on timing and features. For that reason, I am interested in maximizing features verses stability. We will be using the latest software, which may mean beta or even CVS versions.
The three IDS/IPS engines share most of the supporting software requirements, depending on configuration options. Below are a few required libraries and software packages:
|Autotools||The “autotools” consist of autoconf, automake, and libtool. These will likely be installed on your system. You need the autotools if you will be using source from the Bro’s Subversion repository. You will need to run autogen.sh after you check out the code. We will go through the steps below.||Bro, Suricata|
|BIND8 headers and libraries||Most OSs will have BIND installed by default. BIND (Berkeley Internet Name Domain) is an implementation of the Domain Name System (DNS) protocols.||Bro|
|Bison or byacc||Most OSs will have bison installed by default. Bison is a general-purpose parser generator that converts an annotated context-free grammar into an LALR(1) or GLR parser for that grammar.||Bro, Suricata|
|Flex||Most OSs will have flex installed by default. Flex is a tool for generating scanners. A scanner, sometimes called a tokenizer, is a program which recognizes lexical patterns in text.||Bro, Suricata|
|Libdnet||Libdnet provides a simplified, portable interface to several low-level networking routines.||Snort|
|Libpcap||Most OSs will have libpcap installed by default. It is the packet capture library. You may need to install it with support large files (files large than 2G). If you have a Linux kernnel, you will want to configure libpcap for PF_RING support.||Bro, Snort, Suricata|
|LibYAML||LibYAML is a YAML parser and emitter written in C that is used to parse the configuration file.||Suricata|
|PCRE||The PCRE library is a set of functions that implement regular expression pattern matching using the same syntax and semantics as Perl 5.||Snort|
Below are a few libraries and software packages that are not required, but you should consider installing. The packages, except GeoIP and Google Perftools, should have binaries available for your OS. Use these ports to install the packages and save yourself the trouble of having to keep the software updated. We will go through through the installation of GeoIP and Google Perftools from source code.
|GnuPG||Free implementation of the OpenPGP standard.||Bro, Suricata|
|libcap-ng||The libcap-ng library is intended to make programming with posix capabilities much easier than the traditional libcap library.||Suricata|
|LibGeoIP||ability to determine the location of IP addresses.||Bro, Suricata (Future)|
|Libmagic||Add ability to determine file types, as with the ftp analyzer.||Bro|
|libnet||Libnet is a generic networking API that provides access to several protocols.||Suricata|
|libnetfilter_queue||libnetfilter_queue is a userspace library providing an API to packets that have been queued by the kernel packet filter.||Suricata|
|libnfnetlink||libnfnetlink is the low-level library for netfilter related kernel/userspace communication. It provides a generic messaging infrastructure for in-kernel netfilter subsystems (such as nfnetlink_log, nfnetlink_queue, nfnetlink_conntrack) and their respective users and/or management tools in userspace.||Suricata|
|OpenSSL||Tough to image a system not having OpenSSL installed. It is needed to analyze ssh certificates by the HTTP analyzer and for encrypted Bro to Bro communication.||Bro|
|PF_RING||PF_RING is a new type of network socket that dramatically improves the packet capture speed.||Bro, Snort, Suricata|
|zLib||Libz is a compression library. It is used for decompressing HTTP bodies by the HTTP analyzer, and for compressed Bro-to-Bro communication.||Bro|
|XML Analyzer||The XML analyzer is highly-experimental code written by Tobias Kiesling. Installation of Xerces-C++ and XQilla are required to use the XML analyzer. The code allows you to be able to easily adjust analysis capabilities to specific XML data formats. Xerces-C++ is a validating XML parser written in a portable subset of C++. XQilla is an XQuery and XPath 2 library and command line utility written in C++.||Bro, Suricata (Future)|
Installing Supporting Software
Which libraries and supporting software you install will be dependent on which options you use in configuring your IDS/IPS engines. If you can install packages (and not source), carefully consider this option. It will make maintenance easier. We will walk through the source installation for demonstration purposes.
For Linux users, the libcap-ng will be required for dropping privileges.
root# cd /usr/local/src /usr/local/src root# wget http://people.redhat.com/sgrubb/libcap-ng/libcap-ng-0.6.4.tar.gz /usr/local/src root# tar libcap-ng-0.6.4.tar.gz /usr/local/src root# cd libcap-ng-0.6.4 /usr/local/src/libcap-ng-0.6.4 root# ./configure /usr/local/src/libcap-ng-0.6.4 root# make /usr/local/src/libcap-ng-0.6.4 root# make install
Make sure Libdnet is in your library path:
root# /sbin/ldconfig -p | grep -i libdnet libdnet32.so.1 (libc6) => /usr/lib/libdnet32.so.1 libdnet32.so (libc6) => /usr/lib/libdnet32.so
If you do not get a path returned, you will need to install libdnet (use –prefix if it needs to be installed in a special location). We will pull down it down the CVS version, because we will need the sctp.h file to be installed.
root# cd /usr/local/src /usr/local/src root# wget svn checkout http://libdnet.googlecode.com/svn/trunk/ libdnet-cvs /usr/local/src root# cd libdnet-cvs /usr/local/src/libdnet-cvs root# ./configure /usr/local/src/libdnet-cvs root# make /usr/local/src/libdnet-cvs root# make install /usr/local/src/libdnet-cvs root# cp include/dnet/sctp.h /usr/local/include/dnet
If you have installed libdnet in a special location, make sure to include its path in /etc/ld.so.conf.
The library libnet will be required for packet-injecting.
root# cd /usr/local/src /usr/local/src root# wget http://github.com/sam-github/libnet/tarball/libnet-1.1.4 \ -O libnet-1.1.4.tgz /usr/local/src root# tar xzf libnet-1.1.4.tgz /usr/local/src root# cd sam-github-libnet-d2bedb5 /usr/local/src/sam-github-libnet-d2bedb5 root# ./autogen.sh /usr/local/src/sam-github-libnet-d2bedb5 root# ./configure /usr/local/src/sam-github-libnet-d2bedb5 root# make /usr/local/src/sam-github-libnet-d2bedb5 root# make install
Libnfnetlink and Libnetfilter
If you plan on using the IPS capabilities (inline support), you will need to install libnfnetlink and libnfnetlink-queue.
root# cd /usr/local/src /usr/local/src root# wget \ ftp://ftp.netfilter.org/pub/libnfnetlink/snapshot/libnfnetlink-20100823.tar.bz2 /usr/local/src root# bunzip2 libnfnetlink-20100823.tar.bz2 /usr/local/src root# tar xf libnfnetlink-20100823.tar /usr/local/src root# cd libnfnetlink-20100823 /usr/local/src/libnfnetlink-20100823 root# ./autogen.sh /usr/local/src/libnfnetlink-20100823 root# ./configure /usr/local/src/libnfnetlink-20100823 root# make /usr/local/src/libnfnetlink-20100823 root# make check /usr/local/src/libnfnetlink-20100823 root# make install /usr/local/src/libnfnetlink-20100823 root# cd /usr/local/src /usr/local/src root# wget \ ftp://ftp.netfilter.org/pub/libnetfilter_queue/snapshot/libnetfilter_queue-20100824.tar.bz2 /usr/local/src root# md5sum libnetfilter_queue-20100824.tar.bz2 69ce1eb24632bfed050cd936e0fe660c libnetfilter_queue-20100824.tar.bz2 /usr/local/src root# bunzip2 libnetfilter_queue-20100824.tar.bz2 /usr/local/src root# tar xf libnetfilter_queue-20100824.tar /usr/local/src root# cd libnetfilter_queue-20100824 /usr/local/src/libnetfilter_queue-20100824 root# ./autogen.sh /usr/local/src/libnetfilter_queue-20100824 root# PKG_CONFIG_PATH=/usr/local/lib/pkgconfig/ ./configure /usr/local/src/libnetfilter_queue-20100824 root# make /usr/local/src/libnetfilter_queue-20100824 root# make check /usr/local/src/libnetfilter_queue-20100824 root# make install
Libpcap and PF_RING
If you are running on a system with a Linux kernels 2.6.x or greater, you will want to install PF_RING on your system. PF_RING is a network socket that can greatly improve packet capture speed. PF_RING polls packets from NICs by means of Linux NAPI. NAPI (“New API”) is a modification to the device driver packet processing framework, which is designed to improve the performance of high-speed networking. NAPI copies packets from the NIC to PF_RING circular buffer. The application then reads packets from the ring. PF_RING can distribute incoming packets to multiple rings (hence multiple applications) simultaneously. Please see “Exploiting Commodity Multicore Systems for Network Traffic Analysis”  for additional information.
We are going to walk through a specific example with certain ethernet card drivers and a particular linux kernel. Please make sure not to copy the commands blindly. Adjust to your system. The below is for demonstration purposes. These steps are based on Gunjan Bansal blog . Please see Gunjanâ€™s blog for additional explanation.
First step, provide some basic information on the system.
root# /sbin/ifconfig -a eth0 Link encap:Ethernet HWaddr 00:00:00:00:00:00 root# /sbin/ethtool -i eth0 driver: e1000e version: 1.0.2-k3.1 firmware-version: 1.3-1 bus-info: 0000:00:19.0 root# /bin/uname -r 2.6.18-194.8.1.el5PAE
In this example I will be working off the ethernet interface eth0 and the ethernet driver e1000e. The kernel release is 2.6.18-194.8.1.el5PAE.
Second step is to download the PF_RING software from ntop through the SVN repository, configure, compile, and install. The â€œ/sbinâ€ directory will need to be in your PATH or you will get a complaint about â€œldconfig: Command not found.â€
root# cd /usr/local/src root# PATH=$PATH:/sbin /usr/local/src root# mkdir pf_ring && cd pf_ring /usr/local/src/pf_ring root# svn co https://svn.ntop.org/svn/ntop/trunk/PF_RING/ /usr/local/src/pf_ring root# cd PF_RING/kernel /usr/local/src/pf_ring/PF_RING/kernel root# make /usr/local/src/pf_ring/PF_RING/kernel root# make install /usr/local/src/pf_ring/PF_RING/kernel root# cd ../userland/lib /usr/local/src/pf_ring/PF_RING/userland/lib root# make /usr/local/src/pf_ring/PF_RING/userland/lib root# make install
Under some OSs, you need to compile libpcap to support large files (files large than 2G). We are going to install the resulting libpcap under /usr/local. Large file support is required if the following kind of error is produced:
root# ls -lh /data/ids/full2.pcap -rw-r--r-- 1 root root 12G Oct 26 10:01 /data/ids/full2.pcap root# /usr/local/snort/bin/snort -o -A none -c \ /usr/local/snort/conf/snort.conf -l /logs/snort/logs \ -r /data/ids/full2.pcap Error getting stat on pcap file: /data/ids/full2.pcap: Value too large for defined data type ERROR: Error getting pcaps Fatal Error, Quitting..
To compile large file support into libpcap:
/usr/local/src/pf_ring/PF_RING/userland/lib root# cd ../libpcap-1.0.0-ring/ /usr/local/src/pf_ring/userland/libpcap-1.0.0-ring root# ./configure --prefix=/usr/local \ CFLAGS="-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE \ -D_FILE_OFFSET_BITS=64" /usr/local/src/pf_ring/userland/libpcap-1.0.0-ring root# make /usr/local/src/pf_ring/userland/libpcap-1.0.0-ring root# make shared /usr/local/src/pf_ring/userland/libpcap-1.0.0-ring root# make install /usr/local/src/pf_ring/userland/libpcap-1.0.0-ring root# make install-shared
By configuring tcpdump with support for PF_RING, all applications (tcpdump and our IDS/IPS engines) will be able to access simultaneously the PF_RING circular buffer.
/usr/local/src/pf_ring/userland/libpcap-1.0.0-ring root# cd ../tcpdump-4.0.0 /usr/local/src/pf_ring/userland/tcpdump-4.0.0 root# CFLAGS="-D_LARGEFILE_SOURCE \ -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64" LDFLAGS="-lpfring -lpcap" LD_LIBRARY_PATH="/usr/lib:/usr/local/lib" ./configure /usr/local/src/pf_ring/userland/tcpdump-4.0.0 root# make /usr/local/src/pf_ring/userland/tcpdump-4.0.0 root# make install
Replace the ethernet driver.
/usr/local/src/pf_ring/userland/tcpdump-4.0.0 root# cd ../../drivers/intel/e1000e-1.0.15/src /usr/local/src/pf_ring/PF_RING/drivers/intel/e1000e-1.0.15/src root# make /usr/local/src/pf_ring/PF_RING/drivers/intel/e1000e-1.0.15/src root# make install
The third step is to activate PF_RING if its not already activated. Use lsmod to check if pf_ring is started or not.
/usr/local/src/pf_ring/PF_RING/drivers/intel/e1000e-1.0.15/src root# /sbin/lsmod | grep pf_ring pf_ring 46680 0 /usr/local/src/pf_ring/PF_RING/drivers/intel/e1000e-1.0.15/src root# cd \ /lib/modules/2.6.18-194.8.1.el5PAE/kernel/net/pf_ring /lib/modules/2.6.18-194.8.1.el5PAE/kernel/net/pf_ring root# /sbin/insmod \ pf_ring.ko transparent_mode=1 /lib/modules/2.6.18-194.8.1.el5PAE/kernel/net/pf_ring root# cd \ ../../drivers/net/e1000e
Step four, you will unload the ethernet card driver (e1000e) and load the new driver. Keep in mind, unloading the driver means ethernet access will be lost. It is wise not to issue this command remotely.
/lib/modules/2.6.18-194.8.1.el5PAE/kernel/drivers/net/e1000e root# /sbin/rmmod \ e1000e ; /sbin/insmod e1000e.ko
You now are PF_RING enabled.
The yaml library will be required for parsing Suricata configuration file.
root# cd /usr/local/src /usr/local/src root# wget http://pyyaml.org/download/libyaml/yaml-0.1.3.tar.gz /usr/local/src root# tar xzf yaml-0.1.3.tar.gz /usr/local/src root# cd yaml-0.1.3 /usr/local/src/yaml-0.1.3 root# ./configure /usr/local/src/yaml-0.1.3 root# make /usr/local/src/yaml-0.1.3 root# make check /usr/local/src/yaml-0.1.3 root# make install
GeoIP Installation and Configuration
MaxMind GeoIP is a collection of APIs for looking up the location of an IP address. There is a collection of free GeoLite databases, which are not as accurate as the GeoIP databases, but will do for starting out and testing with Bro. To setup GeoIP for use with Bro, please follow the commands below.
root# cd /usr/local/src /usr/local/src root# wget http://www.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz /usr/local/src root# gunzip GeoLiteCity.dat.gz /usr/local/src root# mkdir -p /usr/local/share/GeoIP /usr/local/src root# mv GeoLiteCity.dat /usr/local/share/GeoIP/GeoIPCity.dat /usr/local/src root# wget http://www.maxmind.com/download/geoip/api/c/GeoIP.tar.gz /usr/local/src root# tar xzf GeoIP.tar.gz /usr/local/src root# cd GeoIP-1.4.6 /usr/local/src/GeoIP-1.4.6 root# ./configure /usr/local/src/GeoIP-1.4.6 root# make /usr/local/src/GeoIP-1.4.6 root# make check /usr/local/src/GeoIP-1.4.6 root# make install
Make sure /usr/local/lib is placed into your library path.
Google Perftools Installation and Configuration
Google’s perftools is a collection of a high-performance multi-threaded malloc() implementation and some performance analysis tools. Google’s perftools have replaced mpatrol for leak-checking and heap-profiling. We will compile Bro with –enable-perftools. By default, perftools will install under /usr/local directory. With perftools compiled into Bro, there are two command-line options made available:
|Option||What the option controls|
|-m||turns on leak checking of the main packet loop, with some uninteresting leaks are suppressed. Currently, with one exception (the RPC analyzer; problem not yet found), it reports no leaks when running the test suite.|
|-M||turns on heap profiling: Bro will take a snapshot of the heap before starting the main packet loop and another one when finished.|
To help with the installation of Google’s perftool, the ICSI Networking Group has written a post “Making Sure Your Bro Code Does Not Leak.” The post will provide additional information. The basic steps to install perftools are:
root# cd /usr/local/src /usr/local/src root# wget http://google-perftools.googlecode.com/files/google-perftools-1.6.tar.gz /usr/local/src root# tar xzf google-perftools-1.6.tar.gz /usr/local/src root# cd google-perftools-1.6 /usr/local/src/google-perftools-1.6 root# ./configure /usr/local/src/google-perftools-1.6 root# make /usr/local/src/google-perftools-1.6 root# make check /usr/local/src/google-perftools-1.6 root# make install
The PCRE library is a set of functions that implement regular expression pattern matching using the same syntax and semantics as Perl 5. If you can install PCRE via a binary specific to your operating system, that is the best way to install in order to avoid having to keep the software up-to-date. Below are the instructions for installing the software from source.
root# cd /usr/local/src /usr/local/src root# wget \ http://downloads.sourceforge.net/project/pcre/pcre/8.10/pcre-8.10.tar.gz /usr/local/src root# wget \ http://sourceforge.net/projects/pcre/files/pcre/8.10/pcre-8.10.tar.gz.sig/download /usr/local/src root# gpg --verify pcre-8.10.tar.gz.sig pcre-8.10.tar.gz /usr/local/src root# tar xzf pcre-8.10.tar.gz /usr/local/src root# cd pcre-8.10 /usr/local/src/pcre-8.10 root# ./configure --prefix=/usr/local/pcre /usr/local/src/pcre-8.10 root# make /usr/local/src/pcre-8.10 root# make test /usr/local/src/pcre-8.10 root# make install
The XML analyzer is highly-experimental code written by Tobias Kiesling. Installation of Xerces-C++ and XQilla is required to use the XML analyzer. The code allows you to be able to easily adjust analysis capabilities to specific XML data formats. Xerces-C++ is a validating XML parser written in a portable subset of C++. XQilla is an XQuery and XPath 2 library and command line utility written in C++.
root# cd /usr/local/src /usr/local/src root# wget http://downloads.sourceforge.net/xqilla/XQilla-2.2.4.tar.gz /usr/local/src root# wget http://mirror.its.uidaho.edu/pub/apache/xerces/c/3/sources/xerces-c-3.1.1.tar.gz /usr/local/src root# md5sum xerces-c-3.1.1.tar.gz 6a8ec45d83c8cfb1584c5a5345cb51ae xerces-c-3.1.1.tar.gz /usr/local/src root# tar xzf xerces-c-3.1.1.tar.gz /usr/local/src root# tar xzf XQilla-2.2.4.tar.gz /usr/local/src root# ln -s XQilla-2.2.4 xqilla /usr/local/src root# cd xerces-c-3.1.1 /usr/local/src/xerces-c-3.1.1 root# ./configure /usr/local/src/xerces-c-3.1.1 root# make /usr/local/src/xerces-c-3.1.1 root# make check /usr/local/src/xerces-c-3.1.1 root# make install
With Xerces-C++, configure and install XQilla.
root# cd /usr/local/src/xqilla/ /usr/local/src/xqilla root# ./configure --with-xerces=/usr/local/src/xerces-c-3.1.1/ /usr/local/src/xqilla root# make /usr/local/src/xqilla root# make install
We will be working off the instructions previously posted in “Installing Bro IDS 1.4″ , just updating the material to reflect the requirements of the current Bro software. There a few options when installing Bro. Bro was not developed for the PHB. Advance security software provides the power to the user, with all the options to adapt it to your environment. To quote the Bro site, “Bro has been developed primarily as a research platform for intrusion detection and traffic analysis. It is not intended for someone seeking an ‘out of the box’ solution. Bro is designed for use by Unix experts who place a premium on the ability to extend an intrusion detection system with new functionality as needed, which can greatly aid with tracking evolving attacker techniques as well as inevitable changes to a site’s environment and security policy requirements.” With the Unix experts in mind, we will go through the steps involved to install both the stable and the development versions of Bro.
Current Stable Version
The current version should be the most stable. To install, follow these commands:
root# cd /usr/local/src /usr/local/src root# wget ftp://bro-ids.org/bro-1.5-release.tar.gz /usr/local/src root# tar xzf bro-1.5-release.tar.gz /usr/local/src root# cd bro-1.5.1
The configuration and installations appears below.
Reading the posts on the Bro mailing list, reveals that modifications have already been made to the current release. Fixes are being made continuously. These changes, while fixing problems, might introduce new problems. You do have the option of getting the most up-to-date code possible through the subversion repository. The Bro development team has made available two subparts of the repository: the trunk and development branches. The trunk is the main development head from which releases are made on a regular basis. It should be fairly stable with changes passing a regression suite to ensure the code do not break existing functionality. It is still considered experimental and not suitable for critical deployment. Below is how to download code from the trunk.
root# cd /usr/local/src /usr/local/src root# mkdir bro-cvs /usr/local/src/bro-cvs root# cd bro-cvs /usr/local/src/bro-cvs root# svn checkout http://svn.icir.org/bro/trunk/bro /usr/local/src/bro-cvs root# mv bro bro-1.5.1.cvs /usr/local/src/bro-cvs root# cd bro-1.5.1.cvs /usr/local/src/bro-cvs/bro-1.5.1.cvs root# ./autogen.sh
Robin’s Development Branch
The developers merge their work into the the Bro subversion trunk. Robin Sommer has a separate branch which contains experimental code for:
- the Bro Cluster framework
- NetFlow support (by Bernhard Ager)
- a BitTorrent analyzer (by Nadi Sarrar and Bernhard Ager)
- an XML analyzer (by Tobias Kiesling)
- Python bindings for Broccoli
- restructured logic for taking drop decisions via Bro’s notice framework (by Brian Tierney and Robin Sommer)
- a test-suite for Bro’s communication & serialization subsystems
- various tweaks and bugfixes
If you want the latest work done by Robin and others mentioned above, you can get access to the code with the following commands.
root# cd /usr/local/src /usr/local/src root# mkdir bro-cvs /usr/local/src root# cd bro-cvs /usr/local/src/bro-cvs root# svn checkout http://svn.icir.org/bro/branches/robin/work /usr/local/src/bro-cvs root# mv work bro-1.5.1.robin /usr/local/src/bro-cvs root# cd bro-1.5.1.robin /usr/local/src/bro-cvs/bro-1.5.1.robin root# ./autogen.sh
Configure and Install
Because of the various bug fixes and the additional features which add interesting options, we are going to step through installation of Robin’s branch. Please use the version of Bro appropriate for your operation.
root# cd /usr/local/src/bro-cvs/bro-1.5.1.robin /usr/local/src/bro-cvs/bro-1.5.1.robin root# CFLAGS="-D_LARGEFILE_SOURCE \ -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64" LDFLAGS="-lpfring -lpcap" \ LD_LIBRARY_PATH="/usr/lib:/usr/local/lib" ./configure --prefix=/usr/local/bro \ --enable-debug --enable-perftools /usr/local/src/bro-cvs/bro-1.5.1.robin root# make /usr/local/src/bro-cvs/bro-1.5.1.robin root# make check /usr/local/src/bro-cvs/bro-1.5.1.robin root# make install
If you run into any problems, go to back to the stable version of Bro and see if you can get it to compile. Then you may want to try the subversion trunk code.
We will be following the direction posted previously in â€œBlacklisting with Snortâ€³ . Below we will get the software, verify, configure, and install the software under the /usr/local/snort area. Please adjust this to your environment. Reminder to Mac OS X and FreeBSD users, use the md5 command instead of md5sum.
root# cd /usr/local/src /usr/local/src root# wget wget http://www.snort.org/downloads/116 -O snort-184.108.40.206.tar.gz /usr/local/src root# wget http://www.snort.org/downloads/116/show_md5 /usr/local/src root# cat show_md5 â€œb1119396a32e9df0d80404e4b6c49166â€ /usr/local/src root# md5sum snort-220.127.116.11.tar.gz b1119396a32e9df0d80404e4b6c49166 snort-18.104.22.168.tar.gz /usr/local/src root# tar xzf snort-22.214.171.124.tar.gz /usr/local/src root# cd snort-126.96.36.199
We are going to add in support to place alerts into a MySQL database. If MYSQL is installed on the system, you can use the “–with-mysql” configuration option to specify where. In a previous post, â€œIntroduction to MySQL,â€ we went through the installation of MySQL into the /usr/local/mysql directory. For such an installation, the –with-mysql-includes=/usr/local/mysql/include and –with-mysql-libraries=/usr/local/mysql/lib command options must be used. In order to use the dynamic plugin libraries, Snort needs to be able to find libmysqlclient.so. On some operating systems, you may have problems. Adding LDFLAGS=”-L/usr/local/mysql/lib/mysql” should work.
You may want to consider configuring Snort to allow decoder and preprocessor rule eventing. This allows you to enable and disable decoder and preprocessor events on a rule by rule bases. It also allow you to specify the rule type or action of a decoder or preprocessor event on a rule by rule basis. Enable this configuration option with the configuration option using –enable-decoder-preprocessor-rules.
We will also be adding in large file support. If you had to install libdnet in a special location, you will need to specify that location with the “–with-dnet-includes=” and “–with-dnet-libraries=” configuration options.
We will configure Snort with the following command:
/usr/local/src/snort-188.8.131.52 root# CFLAGS="-D_LARGEFILE_SOURCE \ -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64" LDFLAGS="-lpfring -lpcap" \ LD_LIBRARY_PATH="/usr/lib:/usr/local/lib" \ ./configure --prefix=/usr/local/snort --with-libpcap-includes=/usr/local/include \ --with-libpcap-libraries=/usr/local/lib \ --with-libpcre-includes=/usr/local/pcre/include \ --with-libpcre-libraries=/usr/local/pcre/lib \ --with-mysql-includes=/usr/local/mysql/include \ --with-mysql-libraries=/usr/local/mysql/lib \ --enable-decoder-preprocessor-rules --enable-zlib
Check config.log if you had any problems or just want to make sure Snort configured everything correctly. After you configure Snort, you continue to make and install it.
/usr/local/src/snort-184.108.40.206 root# make /usr/local/src/snort-220.127.116.11 root# make check /usr/local/src/snort-18.104.22.168 root# make install /usr/local/src/snort-22.214.171.124 root# mkdir -p /usr/local/snort/etc /usr/local/src/snort-126.96.36.199 root# cp etc/* /usr/local/snort/etc /usr/local/src/snort-188.8.131.52 root# mkdir -p /usr/local/snort/preproc_rules /usr/local/src/snort-184.108.40.206 root# cp preproc_rules/*.rules /usr/local/snort/preproc_rules /usr/local/src/snort-220.127.116.11 root# /usr/local/snort/bin/snort -V ,,_ -*> Snort! <*- o" )~ Version 18.104.22.168 (Build 39) '''' By Martin Roesch & The Snort Team: http://www.snort.org/team.html Copyright (C) 1998-2010 Sourcefire, Inc., et al. Using PCRE version: 8.10 2010-06-25
Now we need some rules. For this example we will get the rules from the Snort and the Emerging Threats site. You will need to register for the rules at the Snort site. Do consider subscribing for the latest up-to-date rules. Registered users can only access rules 30 days after their release.
root# cd /usr/local/snort/rules /usr/local/snort/rules root# wget http://www.emergingthreats.net/rules/emerging-all.rules /usr/local/snort/rules root# cd /usr/local/src /usr/local/src root# wget \ https://www.snort.org/downloads/83 \ -O snortrules-snapshot-CURRENT.tar.gz /usr/local/src root# md5sum snortrules-snapshot-CURRENT.tar.gz /usr/local/src root# mv snortrules-snapshot-CURRENT.tar.gz /usr/local/snort/ /usr/local/src root# cd /usr/local/snort/ /usr/local/snort root# tar xzf snortrules-snapshot-CURRENT.tar.gz /usr/local/snort root# rm snortrules-snapshot-CURRENT.tar.gz /usr/local/snort root# vi /usr/local/snort/etc/snort.conf
Modify /usr/local/snort/etc/snort.conf to your environment. Make sure the RULE_PATH is set to /usr/local/snort/rules. If you configured Snort to enable decoder and preprocessor rules, you will need to add a line specifying the location of those files. Define PREPROC_RULE_PATH with the line:
var PREPROC_RULE_PATH ../preproc_rules
Later in the snort.conf file include the lines (before other rule lists are included):
include $PREPROC_RULE_PATH/preprocessor.rules include $PREPROC_RULE_PATH/decoder.rules
If you wish to use the emerging threat rules, add:
in the /usr/local/snort/etc/snort.conf file. Do not forget to adjust dynamicpreprocessor file and dynamicengine path. Mac OS X users will need to use the dynamic libraries. Uncomment the Mac OS X lines in the Snort configuration file.
Leon Ward has released a Perl program, Dumbpig, which will check Snort rules for badly formatted entries and incorrect usage. He has even added blacklist support (see posting “ET RBN Blacklists with Snort and DumbPig“). To pull down dumbpig.pl, the required Perl modules, and run it against the Emerging Threats rule set:
root# cd /home/snort/perl /home/snort/perl root# wget http://dumbpig.googlecode.com/files/dumbpig-0.9.tgz /home/snort/perl root# tar xzf dumbpig-0.9.tgz /home/snort/perl root# chmod u+x ./dumbpig.pl /home/snort/perl root# cpan -e "Parse::Snort" /home/snort/perl root# cpan -e "LWP::Simple" /home/snort/perl root# ./dumbpig.pl -r /usr/local/snort/rules/emerging-all.rules DumbPig version 0.9 - email@example.com Because I hate looking for the same dumb problems with snort rule-sets __,, ( Dumb-pig says ) ~( oo ---( "ur rulz r not so ) '''' ( gud akshuly" * ) Config ---------------------- * Sensivity level - 3/3 * Blacklist outputi : Disabled * Processing File - /home/snort/rules//emerging-all.rules * Check commented out rules : Disabled * Pause : Disbled * ForceFail : Disabled * Censor : Disabled * Quite mode : Disabled ---------------------- Issue 1 1 Problem(s) found with rule on line 59 of /home/snort/rules//emerging-all.rules alert tcp $HOME_NET any -> \ [22.214.171.124,126.96.36.199,188.8.131.52,184.108.40.206,220.127.116.11] $HTTP_PORTS ( \ msg:"ET CURRENT_EVENTS Possible Downadup/Conficker-A Infection Checking Geographical Location"; \ flow:to_server; \ classtype:trojan-activity; \ reference:url,www.microsoft.com/security/portal/Entry.aspx?Name=Worm%3aWin32%2fConficker.A; \ reference:url,www.f-secure.com/v-descs/worm_w32_downadup_a.shtml; \ threshold:type both, count 5, seconds 60, track by_src; \ reference:url,doc.emergingthreats.net/bin/view/Main/2008803; \ reference:url,www.emergingthreats.net/cgi-bin/cvsweb.cgi/sigs/CURRENT_EVENTS/CURRENT_Conficker; \ sid:2008803; \ rev:3; \ ) - TCP/UDP rule with no deep packet checks? This rule looks more suited to a firewall or blacklist alert tcp $HOME_NET any -> [18.104.22.168,22.214.171.124,126.96.36.199,188.8.131.52,184.108.40.206] $HTTP_PORTS (msg:"ET CURRENT_ EVENTS Possible Downadup/Conficker-A Infection Checking Geographical Location"; flow:to_server; classtype:trojan-activity; reference:url,www.microsoft.com/security/portal/Entry.aspx?Name=Worm%3aWin32%2fConficker.A; reference:url,www.f-secure.com/v-descs/worm_w32_downadup_a.shtml; threshold:type both, count 5, seconds 60, track by_src; reference:url,doc.emergingthreats.net/bin/view/Main/2008803; reference:url,www.emergingthreats.net/cgi-bin/cvsweb.cgi/sigs/CURRENT_EVENTS/CURRENT_Conficker; sid:2008803; rev:3;)
The Open Information Security Foundation ( OISF) developed Suricata to be on the leading edge in IDS/IPS software. It is very much still in development. Is that not what makes it so interesting? Before you begin, be aware of some known issues. Check out the development roadmap for upcoming scheduled releases. If you run into problems, see if it is a known issue and share your experiences with the community of developers. That is the best way to make sure Suricata improves.
With required and options software on the system, installing Suricata is pretty straight forward. Pull down the source, configure, compile, and install.
root# cd /usr/local/src /usr/local/src root# wget http://www.openinfosecfoundation.org/download/suricata-1.0.1.tar.gz /usr/local/src root# wget http://www.openinfosecfoundation.org/download/suricata-1.0.1.tar.gz.sig /usr/local/src root# wget http://www.openinfosecfoundation.org/download/OISF.asc /usr/local/src root# gpg --import OISF.asc /usr/local/src root# gpg --verify suricata-1.0.1.tar.gz.sig suricata-1.0.1.tar.gz gpg: Signature made Thu 29 Jul 2010 02:34:58 PM EDT using RSA key ID 051CC261 gpg: Good signature from "OISF
Modify suricata.yaml to reflect your environment. At this point, you can run Suricata with the command:
root# /usr/local/suricata/bin/suricata -c /usr/local/suricata/etc/suricata.yaml \ -s /usr/local/suricata/etc/classification.config -i eth0
To Be Continued…
Henry J. Kaiser, the father of modern American shipbuilding, once said, “Live daringly, boldly, fearlessly. Taste the relish to be found in competition – in having put forth the best within you”  Hopefully Jonkman, Julien, Roesch, and Olney Roesch will relish their competition and the community will enjoy the fruits of their efforts. One powerful benefit of open source is that it allows organizations the flexibility to pull down the source and setup the software in their own environment. One can easily try the packages out and become familiar with the benefits of the different IDS/IPS engines. The more you know, the better you will be at defending your organization.
Setting up the three IDS/IPS engines is only the first step. In later posts, we will continue by examining the configuration and output from Bro, Snort, and Suricata. This should help the reader understand the features each might offer an organization. Bro, Snort, and Suricata are just tools. While they will have different features, it is the person who yields the tool that determines its effectiveness. Determine for yourself what works best. Even if you have a single development box, you can setup and test against small subsets of your own network traffic. Visit Wiresharkâ€™s Sample capture page for links to pcap files and additional sources. There is also the OpenPacketâ€™s Capture Repository, which provides the security community the capability to comment and vote on submitted pcap files. Give the IDS/IPS engines a test ride and please feel free to share your experiences.
 Marc Ambinder, August 13th 2010, “Pentagon Wants to Secure Dot-Com Domains of Contractors,” http://www.theatlantic.com/politics/archive/2010/08/pentagon-wants-to-secure-dot-com-domains-of-contractors/61456/.
 John Gerber, January 5th 2010, “Suricata: A Next Generation IDS/IPS Engine,” http://blog.securitymonks.com/2010/01/05/suricata-a-next-generation-idsips-engine/.
 John Gerber, October 29th 2008, “Installing Bro IDS 1.4,” http://blog.securitymonks.com/2008/10/28/installing-bro-ids-14/.
 John Gerber, October 20th 2008, “Snort 3: The Next Generation,” http://blog.securitymonks.com/2008/10/20/snort-3-the-next-generation/.
 John Gerber, August 9th, 2008, “IDS/IPS: The Mark Twain of the Security World,” http://blog.securitymonks.com/2008/08/09/idsips-the-mark-twain-of-the-security-world/.
 John Gerber, June 17th, 2007, “IDS,” http://blog.securitymonks.com/2007/06/17/ids/.
 Jaikumar Vijayan, July 20th 2010, “DHS, vendors unveil open source intrusion detection engine,” http://www.computerworld.com/s/article/9179436/DHS_vendors_unveil_open_source_intrusion_detection_engine.
 Ellen Messmer, July 20th 2010, “Is open source Snort dead? Depends who you ask ,” http://www.networkworld.com/news/2010/072010-is-snort-dead.html?page=1.
 “Wikiquote: Henry J. Kaiser,” http://wapedia.mobi/enwikiquote/Henry_J._Kaiser.
 Matt Olney, July 20th 2010, “Innovation — You Keep Using That Word…,” http://vrt-sourcefire.blogspot.com/2010/07/innovation-you-keep-using-that-word.html.
 “Razorback”, http://sourceforge.net/projects/razorbacktm/files/.
 “Next Generation Open-Source IDS to Address Issues Facing Network Security Industry,” July 19th 2010, http://www.businesswire.com/portal/site/home/permalink/?ndmViewId=news_view&newsId=20100719005970&newsLang=en.
 Russ McRee, August 3rd 2010, “Suricata in toolsmith: meet the meerkat,” http://holisticinfosec.blogspot.com/2010/08/suricata-in-toolsmith-meet-meerkat.html.
 Victor Julien, July 22nd 2010, “On Suricata performance,” http://www.inliniac.net/blog/2010/07/22/on-suricata-performance.html.
 “Bro Features and Benefits,” June 28th 2010, http://www.bro-ids.org/Features.html.
 “Snort 2.9.0 Beta,” June 6th 2010, https://s3.amazonaws.com/snort.org/snort-beta/20100727/release_notes_290_beta.txt?AWSAccessKeyId=AKIAJJSHU7YNPLE5MKOQ&Expires=1282246315&Signature=jchbdD6d3G0ncaqOkrQAps6ZV5M%3D.
 John Gerber, July 19th 2009, â€œBlacklisting with Snort,â€ http://blog.securitymonks.com/2009/07/19/blacklisting-with-snort/.
 Russ McRee, August 2010, “Suricata: An Introduction,” http://holisticinfosec.org/toolsmith/pdf/august2010.pdf
 “Dynamic Application-Layer Protocol Analysis for Network Intrusion Detection,” Proceedings of the 15th conference on USENIX Security Symposium – Volume 15, 2006, http://www.icir.org/robin/papers/usenix06.pdf.
 Seth Hall, August 6th 2010, Github Social Coding, “http://github.com/sethhall/bro_scripts/blob/master/http-ext.bro.
 Seth Hall, August 6th 2010, Github Social Coding,http://github.com/sethhall/bro_scripts/blob/master/http-ext.bro#L64.
 Gunjan Bansal, June 16th 2010, â€œInstallation Guide for PF_RING,â€ http://gunjan-bansal.blogspot.com/2010/06/installation-guide-for-pfring.html.
 Luca Deri and Francesco Fusco, January 30th 2010, â€œExploiting Commodity Multi-core Systems for Network Traffic Analysis,â€ http://luca.ntop.org/MulticorePacketCapture.pdf.
 Mathew Olney and Matthew Watchinski, “Implementing Resource Intensive Detection Techniques With the Razorback Framework,” http://iweb.dl.sourceforge.net/project/razorbacktm/Whitepapers/razorback-whitepaper-0.1.pdf.
 Patrick Mullen and Ryan Pentney, Defcon Razorback Presentation, “https://www.defcon.org/images/defcon-18/dc-18-presentations/Mullen-Pentney/DEFCON-18-Mullen-Pentney-Razorback.pdf.
 Robin Sommer, August 24th 2010, The ICSI Networking Group Blog, “Major NSF Funding for Bro Development,” http://blog.icir.org/2010/08/major-nsf-funding-for-bro-development.html.