First, a little about some of the work. I have had to evaluate IPs for an indication of their security threat. One method of evaluation is to compare the IPs to know bad actors. In this post, we will discuss a few data sources that are freely available, a few software packages that might prove useful, and finish up pointing to some sources for further evaluation.

Data Source

You can use various data feeds. Misbehaving IPs that are identified by your IDSP/IPS, honeypots, firewall logs, router logs, syslog servers, etc. will be of particular interest, being specific to your organization. For the sake of discussion, I wanted to point out some freely available sources of IPs that are blacklisted by the Internet community.

• The Harimau Watchlist – Mel Mudin (spoonfork) provides this valuable source of information. Please read his post, “The Harimau Watchlist” for additional information. The information is updated daily.
• Malware Domain Blocklist – this information is maintained as part of the DNS-BH project and represents a list of domains that are known to be used to propagate malware and spyware.

The sources for the Harimau Watchlist include:

• Dshield Top IPs
• Dshield Top Blocks
• ShadowServer’s Know Russian Business Network
• ShadowServer’s Known Bot Command & Control IPs/Blocks
• EmergingThreats Known Compromised IPs/Blocks
• Spamhaus Top IPs
• Atlas (Arbor Networks) Top Threat Source
• TrustedSource.org Top Email Senders
• TrustedSource.org Most Active Storm Web Proxies
• TrustedSource.org Most Newly Activated Storm Web Proxies
• TrustedSource.org Most Recently Seen Storm Web Proxies
• Projecthoneypot.org’s Most Recent Email Harvesters
• Projecthoneypot.org’s Most Recent Spam Servers
• Projecthoneypot.org’s Most Recent Comment Spammers
• Projecthoneypot.org’s Most Recent Dictionary Attackers
• Senderbase.org Top 100 Spammers
• Senderbase.org Top 100 Virus Senders

The Malware Domain Blocklist sources include ddanchev.blogspot.com, www.matchent.com, siteadvisor, threatexpert, and many more. For more details, see the site.

Programming

I will not go into details now, but it is easy enough to setup a cron job to pull the information down and add the IPs to a database. If you decide to do this in Perl, a few modules that will come in handy:

• LWP::UserAgent – can be used to dispatch web requests.
• DBI – Perl database interface.
• Net::Abuse::Utils – provides functions to lookup information about an IP or ASN. Information includes country code for an IP or ASN, ASN announcing an IP via BGP, CIDR network an IP is announced in, contact email addresses based on IP whois info, contact email addresses for a domain based on abuse.net data, contact email address from the SOA record for the rDNS zone for an IP, and listing information for an IP in a specific DNSBL.
• Geo::IP – provides a simple file-based database. The GeoIP database simply contains IP blocks as keys, and countries as values. The data contains all public IP addresses and should be more complete and accurate than reverse DNS lookups.
• Net::DNS – allows the programmer to perform nearly any type of DNS query.

A few other software packages you will likely use:

• MySQL – is a multi-threaded and multi-user SQL (Structured Query Language) database server.
• GeoLite Country – is similar to the GeoIP Country database, but is slightly less accurate. Please review Instructions on how to use our CSV databases with a SQL database.
• GeoLite City – is similar to the GeoIP City database, but is less accurate.
• Geo/IPfree – Perl module for looking up country of IP Address.

A Few Interesting Possibilities

One thing that can be done with the IPs is to map them using Google Earth. This will require you to create KML files, which are not difficult once you have the IPs along with their DNS and GeoIP data. Two scripts that help generate KML files from security data are:

• Cosight – the security log file visualization tool used by the Colorado ISOC. Cosight parses logfiles looking for connections to or from internet addresses. It then uses the geolocation database from Maxmind to convert those addresses to coordinates for output as a KML overlay file.
• KisGearth – a small perl script to convert kismet xml and gps logfiles to google earth kml files.

A few months ago, I did a post “Unclear and Present Danger.” The post outlined some of the electronic dangers facing an organization on the Internet. Thanks to the fantastic work done by the Shadowserver Foundation, we have a nice collection of some very interesting statistics mapped by country. Those examples can be very useful when mapping misbehaving IPs. Rather than repeat what has previously been posted, I’ll leave it to the reader to visit that entry.

While searching for an interesting way to represent and drill down from continents, to countries etc., I came across GeoTree, a hierarchical toponym browser for GeoNames. GeoNames is part of the Linked Data project, which brings together data from public sources and builds a web of open and free data where data sets are interlinked with each other. The Linked Data project represents a great wealth of information. Below is a mapping done by Richard Cyganiak of the projects involved in the Linked Data projects:

Walter Rafelsberger provides two interesting examples, that can be adapted for security representation and interpretation. Both examples make use of the Processing language. Processing is a data visualization programming language. Read more about Processing on Ben Fry’s or Casey Reas‘ blog.

• Geosketch of world cities with a population of more than 1000, labeling those cities with more than 5 million:
  
• The second example visualizes conversations of about 1500 users from Twitter. The arcs link positions of people who talk to each other:
  

Nathan Yau, from Flowing Data posted about “40 Essential Tools and Resources to Visualize Data.” The post contains valuable information with additional resource links. I came across Nathan’s post, while checking out FlowingData’s graphic post “Watching the Growth of Walmart Across America.” I was not able to embed the object. You will need to click on the image to view the growth of Walmart.

What is really nice is that you can downloaded the code, including the Actionscripts with the openings data from FlowingData’s site . With that code other types of growth can be illustrated in a similar manner. That is really nice. Modest Maps, a BSD-licensed display and interaction library for tile-based maps in Flash (ActionScript 2.0 and ActionScript 3.0) and Python was used to map the data. This reminds me of code_swarm:

code_swarm – Eclipse (short ver.) from Michael Ogawa on Vimeo.

If you have never watched the code_swarm video, you have to check it out. It was done by Michael Ogawa. The example above shows the commit history of the Eclipse open source project. To quote Michael:

code_swarm, shows the history of commits in a software project. A commit happens when a developer makes changes to the code or documents and transfers them into the central project repository. Both developers and files are represented as moving elements. When a developer commits a file, it lights up and flies towards that developer. Files are colored according to their purpose, such as whether they are source code or a document. If files or developers have not been active for a while, they will fade away. A histogram at the bottom keeps a reminder of what has come before.

It is a great example of visualizing something we traditionally would not think of outside of your run of the mill reports and numbers.

Take a look at Jamie Wilkinson’s post “Obama Wikipedia page edits,” which is a visualization of people who have contributed to the Barack Obama page on Wikipedia between October 2005 – November 2008. Users who edit a lot drift toward the center. Visualized using code_swarm (Processing) and Jamie’s Wikipedia page history parser Wikiswarm (Ruby). Code and instructions on how Jamie created this visualization can be found in his post “Wikiswarm: visualize Wikipedia page histories.”

Obama Wikipedia page edits from Jamie Dubs on Vimeo.

Most important, the code_swarm source if freely available.

Final Words

Today we explored a few interesting paths for representing data. Three excellent books to help guide us further on the visualization paths are:

• Security Data Visualization by Greg Conti.
• Applied Security Visualization by Raffael Marty.
• Processing: A Programming Handbook for Visual Designers and Artists by Casey Reas and Ben Fry (forward by John Maeda) .

We have all heard the proverb, “A picture is worth a thousand words.” Another famous quote states, “The devil is in the details.” Or, if you prefer, “God is in the details.” If life was a Star Trek episode, Kirk could have used those two quotes to cause a computer to explode. Both statements are true and false, depending on the circumstances.

It is wise to remember the words of Siddhartha Gautama: “These blind men, every one honest in his contentions and certain of having the truth, formed schools and sects and factions.” Geocoding and data visualization simply provide tools to help interpret information. Interpretations are not absolute. If you are looking for a silver bullet that will help the blind see, and the ignorant smart, I am afraid your search must continue. The author A. L. Linall, Jr. once wrote, “Visualization and belief in a pattern of reality, activates the creative power of realization.” The best solutions will come from using a combination of tools to help explore the possibilities, discover insights, view the results from different views which helps with realization, and provide a way to effectively communicate results.

Joseph Campbell, an American mythology professor, writer, and lecturer, wrote “Computers are like Old Testament gods; lots of rules and no mercy.” In the security world, signatures would be the rules that computers follow. While signatures can be very useful, they also are very limiting. In a previous post, titled simply “IDS“, we discussed how Intrusion Detection System/Intrusion Prevention System (IDS/IPS) technology is moving away from being solely signature based to a blend of signature based, anomaly detection, and activity based methodologies.

This is not surprising when we examine other areas of security. Liam Tung, writer for ZDnet, has written an article titled, “Signature-based antivirus is dead: Get over it“. In the article, Simon Clausen, founder & CEO at PC Tools, reports that the security industry has been looking beyond blacklists. “I would very much disagree that AV is dead. Really, traditional signature-based AV is going to be dead in a few years, but what every antivirus company is evolving towards, like us, is behavioural AV technology, so AV will be alive.”

Bro is not signature based. Instead, it was developed to be activity based with some support for anomaly detection. From the beginning, Bro has always been focused on connections instead of just packets. The Bro development team continues to advance the software, as demonstrated with the extensive changes in version 1.4. Below are a few of the new features:

• Can import Netflow version 5 data.
• A BitTorrent analyzer is now available.
• The “Bro Lite” configuration is now deprecated and will not in
  general be supported.
• Substantial updates to Broccoli, a Bro client library.
• Extensive changes to allow Bro to process packets captured in the past intermingled with those captured in real-time.
• scan.bro has been heavily modified to better support distributed scan analysis.
• The new policy script targeted-scan.bro looks for repeated access from the same source to the same server, to detect things like SSH password-guessing attacks.
• GeoIP information now includes latitude and longitude.
• ssh.bro now supports the variable skip_processing_after_handshake which directs the event engine to omit any further processing of an SSH connection after its initial handshake.
• Google’s perftools have replaced mpatrol for leak-checking and heap-profiling.

Additional Information

For additional information on Bro, below is a list of a few good site.

• Bro Quick Start Guide – contains info on installing, configuring, and running Bro.
• Bro Wiki – intended for users and developers of Bro.
• The Bro Archives – mailing list archive.
• Emerging Bro – Bro signatures repository.
• The ICSI Networking Group Blog – the Blog for the Network Research at the International Computer Science Institute in Berkeley, CA. These are the folks that develop Bro.
• A Bro Blog – A blog by Seth Hall, a Bro master and contributor.
• C.S.Lee (geek00L) blog When {Puffy} Meets ^RedDevil^ .
• TaoSecurity – Richard Bejtlich has some good posts on Bro.
• Last Bro Workshop Material.

Supporting Software

Bro offers many configuration options, depending on how you will use the software. Below are a few libraries and software packages required by Bro:

Below are a few libraries and software packages that are not required, but you should consider installing. The packages, except GeoIP and Google Perftools, should have binaries available for your OS. Use these ports to install the packages and save yourself the trouble of having to keep the software updated. We will go through through the installation of GeoIP and Google Perftools from source code.

GeoIP Installation and Configuration

MaxMind GeoIP is a collection of APIs for looking up the location of an IP address. There is a collection of free GeoLite databases, which are not as accurate as the GeoIP databases, but will do for starting out and testing with Bro. To setup GeoIP for use with Bro, please follow the commands below.

root# cd /usr/local/src /usr/local/src root# wget \ http://www.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz /usr/local/src root# gunzip GeoLiteCity.dat.gz /usr/local/src root# mkdir -p /usr/local/share/GeoIP /usr/local/src root# mv GeoLiteCity.dat /usr/local/share/GeoIP/GeoIPCity.dat /usr/local/src root# wget \ http://www.maxmind.com/download/geoip/api/c/GeoIP.tar.gz /usr/local/src root# tar xzf GeoIP.tar.gz /usr/local/src root# cd GeoIP-1.4.5 /usr/local/src/GeoIP-1.4.5 root# ./configure /usr/local/src/GeoIP-1.4.5 root# make /usr/local/src/GeoIP-1.4.5 root# make check /usr/local/src/GeoIP-1.4.5 root# make install

Make sure /usr/local/lib is placed into your library path.

Google Perftools Installation and Configuration

Google’s perftools is a collection of a high-performance multi-threaded malloc() implementation and some performance analysis tools. Google’s perftools have replaced mpatrol for leak-checking and heap-profiling. We will compile Bro with –enable-perftools. By default, perftools will install under /usr/local directory. With perftools compiled into Bro, there are two command-line options made available:

To help with the installation of Google’s perftool, the ICSI Networking Group has written a post “Making Sure Your Bro Code Does Not Leak.” The post will provide additional information. The basic steps to install perftools are:

root# cd /usr/local/src /usr/local/src root# wget \ http://google-perftools.googlecode.com/files/google-perftools-0.99.2.tar.gz /usr/local/src root# tar xzf google-perftools-0.99.2.tar.gz /usr/local/src root# cd google-perftools-0.99.2 /usr/local/src/google-perftools-0.99.2 root# ./configure /usr/local/src/google-perftools-0.99.2 root# make /usr/local/src/google-perftools-0.99.2 root# make check /usr/local/src/google-perftools-0.99.2 root# make install /usr/local/src/google-perftools-0.99.2 root# export LDFLAGS=-L/usr/local/lib /usr/local/src/google-perftools-0.99.2 root# export CFLAGS=-I/usr/local/include /usr/local/src/google-perftools-0.99.2 root# export CPPFLAGS=-I/usr/local/include /usr/local/src/google-perftools-0.99.2 root# export LD_LIBRARY_PATH=/usr/local/lib

XML Analyzer

The XML analyzer is highly-experimental code written by Tobias Kiesling. Installation of Xerces-C++ and XQilla is required to use the XML analyzer. The code allows you to be able to easily adjust analysis capabilities to specific XML data formats. Xerces-C++ is a validating XML parser written in a portable subset of C++. XQilla is an XQuery and XPath 2 library and command line utility written in C++.

root# cd /usr/local/src /usr/local/src root# wget \ http://downloads.sourceforge.net/xqilla/XQilla-2.1.3.tar.gz /usr/local/src root# wget \ http://mirror.its.uidaho.edu/pub/apache/xerces/c/2/sources/xerces-c-src_2_8_0.tar.gz /usr/local/src root# md5sum xerces-c-src_2_8_0.tar.gz 5daf514b73f3e0de9e3fce704387c0d2 xerces-c-src_2_8_0.tar.gz /usr/local/src root# tar xzf xerces-c-src_2_8_0.tar.gz /usr/local/src root# tar xzf XQilla-2.1.3.tar.gz /usr/local/src root# ln -s XQilla-2.1.3 xqilla /usr/local/src root# cd xerces-c-src_2_8_0 /usr/local/src/xerces-c-src_2_8_0 root# patch -p1 < ../xqilla/src/xercesc_content_type.patch /usr/local/src/xerces-c-src_2_8_0 root# patch -p1 <../xqilla/src/xercesc_regex.patch /usr/local/src/xerces-c-src_2_8_0 root# export XERCESCROOT=`pwd` /usr/local/src/xerces-c-src_2_8_0 root# cd src/xercesc

FreeBSD 7 users will encounter a problem when trying the run the runConfigure command. The error “C compiler cannot create executables ” will be produced. The problem is on line 358 of runConfigure. The libc_r library cannot be found since it has been deprecated on FreeBSD since version 5.X and removed from version 7.0. Edit runConfigure to not include “-lc_r” in the list of threading libraries. Then issue the command:

/usr/local/src/xerces-c-src_2_8_0/src/xercesc root# runConfigure \ -pfreebsd -cgcc -xg++ -minmem -nsocket -tIconvFBSD -rpthread -s /usr/local/src/xerces-c-src_2_8_0/src/xercesc root# gmake /usr/local/src/xerces-c-src_2_8_0/src/xercesc root# gmake install

An easier option is to install xerces and xqilla through the FreeBSD port command:

root # cd /usr/ports/textproc/xqilla /usr/ports/textproc/xqilla root # make install clean

Other operating systems, such as Linux, do not require any special steps. You just need to run the commands:

/usr/local/src/xerces-c-src_2_8_0/src/xercesc root# runConfigure -plinux /usr/local/src/xerces-c-src_2_8_0/src/xercesc root# make /usr/local/src/xerces-c-src_2_8_0/src/xercesc root# make install

With Xerces-C++, configure and install XQilla.

/usr/local/src/xerces-c-src_2_8_0/src/xercesc root# cd /usr/local/src/xqilla/ /usr/local/src/xqilla root# ./configure --with-xerces=`pwd`/../xerces-c-src_2_8_0/ /usr/local/src/xqilla root# make /usr/local/src/xqilla root# make install

Bro Installation and Configuration

There a few options when installing Bro. Bro was not developed for the PHB. Advance security software provides the power to the user, with all the options to adapt it to your environment. To quote the Bro site, “Bro has been developed primarily as a research platform for intrusion detection and traffic analysis. It is not intended for someone seeking an ‘out of the box’ solution. Bro is designed for use by Unix experts who place a premium on the ability to extend an intrusion detection system with new functionality as needed, which can greatly aid with tracking evolving attacker techniques as well as inevitable changes to a site’s environment and security policy requirements.” With the Unix experts in mind, we will go through the steps involved to install both the stable and the development versions of Bro.

Current Stable Version

The current version should be the most stable. To install, follow these commands:

root# cd /usr/local/src /usr/local/src root# wget ftp://bro-ids.org/bro-1.4-release.tar.gz /usr/local/src root# tar xzf bro-1.4-release.tar.gz /usr/local/src root# cd bro-1.4

The configuration and installations appears below.

Subversion Trunk

Reading the posts on the Bro mailing list, reveals that modifications have already been made to the current release. Fixes are being made continuously. These changes, while fixing problems, might introduce new problems. You do have the option of getting the most up-to-date code possible through the subversion repository. The Bro development team has made available two subparts of the repository: the trunk and development branches. The trunk is the main development head from which releases are made on a regular basis. It should be fairly stable with changes passing a regression suite to ensure the code do not break existing functionality. It is still considered experimental and not suitable for critical deployment. Below is how to download code from the trunk.

root# cd /usr/local/src /usr/local/src root# mkdir bro-cvs /usr/local/src/bro-cvs root# cd bro-cvs /usr/local/src/bro-cvs root# svn checkout http://svn.icir.org/bro/trunk/bro /usr/local/src/bro-cvs root# mv bro bro-1.4.cvs /usr/local/src/bro-cvs root# cd bro-1.4.cvs /usr/local/src/bro-cvs/bro-1.4.cvs root# ./autogen.sh

Robin’s Development Branch

The developers merge their work into the the Bro subversion trunk. Robin Sommer has a separate branch which contains experimental code for:

• the Bro Cluster framework
• NetFlow support (by Bernhard Ager)
• a BitTorrent analyzer (by Nadi Sarrar and Bernhard Ager)
• an XML analyzer (by Tobias Kiesling)
• Python bindings for Broccoli
• restructured logic for taking drop decisions via Bro’s notice framework (by Brian Tierney and Robin Sommer)
• a test-suite for Bro’s communication & serialization subsystems
• various tweaks and bugfixes

If you want the latest work done by Robin and others mentioned above, you can get access to the code with the following commands.

root# cd /usr/local/src /usr/local/src root# mkdir bro-cvs /usr/local/src root# cd bro-cvs /usr/local/src/bro-cvs root# svn checkout http://svn.icir.org/bro/branches/robin/work /usr/local/src/bro-cvs root# mv work bro-1.4.robin /usr/local/src/bro-cvs root# cd bro-1.4.robin /usr/local/src/bro-cvs/bro-1.4.robin root# ./autogen.sh

Configure and Install

Because of the various bug fixes and the additional features which add interesting options, we are going to step through installation of Robin’s branch. Please use the version of Bro appropriate for your operation.

root# cd /usr/local/src/bro-cvs/bro-1.4.robin /usr/local/src/bro-cvs/bro-1.4.robin root# ./configure --enable-debug \ --enable-perftools --prefix=/usr/local/bro --with-xqilla /usr/local/src/bro-cvs/bro-1.4.robin root# make /usr/local/src/bro-cvs/bro-1.4.robin root# make check /usr/local/src/bro-cvs/bro-1.4.robin root# make install

If you run into any problems, go to back to the stable version of Bro and see if you can get it to compile. Then you may want to try the subversion trunk code.

Final Words

We have taken the first step and now have Bro installed. Installation is only the beginning. In an upcoming post, we will walk through configuring Bro and examining a simple policy. We will send some attacks against Bro, and see what kind of results are produced. Having completed the first step of installing Bro, we can move on and have some fun. As Humphrey Bogart put it in his famous last line from Casablanca, “Louis, I think this is the beginning of a beautiful friendship.”