Security Advancements at the Monastery » Open Source

Google Visualization: An Example Graphing NVD CVE Data

John Gerber — Fri, 16 Apr 2010 15:54:42 +0000

Google visualization offers graphing abilities to any number of projects. Why should security professionals care? If you are going to have to collect and present security metrics, it is best to showcase them in the very best manner possible. Andrew Jaquith in his article, “Creating meaningful information security metrics” states, “For 2010, Forrester Research expects that overall security budgets will rise less than 5 percent over 2009 –higher than in the previous year, but not by much.” Andrew goes on to point out, “smart security managers, sensing sudden vulnerability in their budgets, seek better ways to measure and prove the value of what they do every day.”

In today’s work environment there is a need to show changes, potential risks, improved performance, etc. in all areas of the company’s operations. Security professionals need to be prepared to answer the basic question, “why should the CIO or CEO care about security?” CSO Online has a great quote from the post, “From the CIO: Why You Didn’t Get the CISO Job” that challenges us to consider our views when it comes to security. The post states, “laser focus on your speciality is great in middle management. It’s what we want. One of the really hard things about jumping from management to executive is a focus on the whole of the business. It’s a rare person who manages it quickly or easily.” That is basically the problem with metrics. It is a battle between generalization to the point of uselessness and details to the point of not being understandable or collectible. At the end of the day, something needs to be done because the security industry is currently leaving upper management in the position of not understanding what is going on within their business. That is a risk that not acceptable.

Andrew’s article discusses what kind of security metrics should be used. Additional sources of information on security metrics can be found in a previous post entitled “Security Metrics.” The post provides links to wonderful sources on security metric information. You might also want to take a look at the CIS Consensus Security Metrics v1.0.0 guide, NIST Special Publication (SP) 800-55 Rev 1 “Security Metrics Guide for Information Technology Systems”, NIST IR-7564 “Directions in Security Metrics Research”, “Twenty Most Important Controls and Metrics for Effective Cyber Defense and Continuous FISMA Compliance,” and “Metrics, measures & Myths.” Once you have start gathering metrics, you will want to present them in an easy to understand format. This is where Google Visualization can help.

Today’s post walks through an example using the data from the National Institute of Standards and Technology (NIST) National Vulnerability Database (NVD ) Common Vulnerabilities and Exposures (CVE) database. The purpose is to provide a working example from which you can learn and apply to the various metrics gathered at your organization.

Data Source

A previous post, “Standardization and Interoperability in Security,” discussed how the Security Content Automation Protocol (SCAP) is an attempt to help defenders by providing a collection of XML schemas/standards that allow technical security information to be exchanged between tools. SCAP components consists of:

Common Configuration Enumeration (CCE): provide unique identifiers to system configuration issues in order to facilitate fast and accurate correlation of configuration data across multiple information sources and tools.
Common Platform Enumeration (CPE): a structured naming scheme for information technology systems, platforms, and packages.
Common Vulnerability Enumeration (CVE): a dictionary of publicly known information security vulnerabilities and exposures.
Common Vulnerability Scoring System (CVSS): a vulnerability scoring system designed to provide an open and standardized method of rating IT vulnerabilities. NIST has even provided a calculator for creating CVSS vulnerability severity scores.
eXtensible Checklist Configuration Description Format (XCCDF): a specification language for writing security checklists, benchmarks, and related kinds of documents. NIST has released the NIST Interagency Report 7275 Revision 3 “Specification for Extensible Configuration Checklist Description Format (XCCDF) Version 1.1.4.”
Open Vulnerability Assessment Language (OVAL): an information security community standard to promote open and publicly available security content, and to standardize the transfer of this information across security tools and services.

We are going to make use of the data from NVD/CVE XML feed with the Common Vulnerability Scoring System (CVSS) mappings (version 2.0). NIST documentation states:

CVSS provides an open framework for communicating the characteristics and impacts of IT vulnerabilities. Its quantitative model ensures repeatable accurate measurement while enabling users to see the underlying vulnerability characteristics that were used to generate the scores. Thus, CVSS is well suited as a standard measurement system for industries, organizations, and governments that need accurate and consistent vulnerability impact scores. Two common uses of CVSS are prioritization of vulnerability remediation activities and in calculating the severity of vulnerabilities discovered on one’s systems.

NVD provides CVSS ‘base scores‘ representing the innate characteristics of each vulnerability. ‘Temporal scores,’ which change over time due to events external to the vulnerability, are not provided though NVD does provide a CVSS score calculator. This allows an organization to add temporal data and even factor in ‘environmental scores‘ customized to reflect the impact of the vulnerability on the organization. Please refer to the CVSS standards guide and the OWASP Risk Rating Methodology concerning factors involved in estimating the severity of risks to your business.

NVD CVE XML Schema

For our example, we will be using the data feeds nvdcve-2.0-2010.xml and nvdcve-2.0-2009.xml. Examining the CVE XML 2.0 Schema, we are particularly interested in certain vulnerability and CVSS scoring information. For example, for CVE-2010-1228, we will parse and pull the following kind of information:

 id="CVE-2010-1228">
  CVE-2010-1228
  2010-04-01T18:30:00.453-04:00
  
  2010-04-05T00:00:00.000-04:00
  
  
    
      10.0
      NETWORK
      LOW
      NONE
      COMPLETE
      COMPLETE
      COMPLETE
      http://nvd.nist.gov

Using Perl to Retrieve the CVE File

Initially we will read the nvdcve-2.0-2010.xml and nvdcve-2.0-2009.xml files. If we start retrieving the file regularly, we would want to change this to nvdcve-2.0-recent.xml. Of course, previous years can also be read in to provide a longer perspective on vulnerabilities. A simple example of a Perl subroutine to read the NVD CVE file and save it locally would be:

sub readpage {
   my($url,$nvd_file) = @_;
   my($proxy) = "http://your-proxy-server:proxy-port";
   my $ua = new LWP::UserAgent;
   $ua->proxy(http  => $proxy);
   $ua->proxy(ftp => $proxy);
   $ua->proxy(https => $proxy);
   # Go out and retrieve page
   my $req = new HTTP::Request('GET', $url);
   my $res = $ua->request($req);
   my $pjstatus = 1;
   # Check if the requested webpage is there and return results
   if ($res->is_success) { # Request successful
       open(OUTFILE,">$nvd_file") || ($pjstatus = 0);
       if ($pjstatus) {
          print OUTFILE $res->content;
       }
       close(OUTFILE);
   }
   else {
      $pjstatus = 0;
   }
   return($pjstatus);
}

Please substitute “http://your-proxy-server:proxy-port” with your site’s proxy server and port, if applicable.

Creating a MYSQL Table to Hold the Data

There is a great deal of information in the NVD CVE file. You will need to determine what information your organization will be interested in storing and graphing. For better or worse, folks have come to expect vulnerabilities to have a “Low,” “Medium,” or “High” score. NIST has stated concerning the NVD Vulnerability Severity Ratings:

NVD provides severity rankings of “Low,” “Medium,” and “High” in addition to the numeric CVSS scores but these qualitative rankings are simply mapped from the numeric CVSS scores:
1. Vulnerabilities are labeled “Low” severity if they have a CVSS base score of 0.0-3.9.
2. Vulnerabilities will be labeled “Medium” severity if they have a base CVSS score of 4.0-6.9.
3. Vulnerabilities will be labeled “High” severity if they have a CVSS base score of 7.0-10.0.

While preferring quantitative over qualitative values, for this example I would like to create a stacked column chart. We will add a severity column which is based on the CVSS score. An example table follows:

CREATE DATABASE vulnerabilities;
USE vulnerabilities;
DROP TABLE IF EXISTS `nvdcve`;
CREATE TABLE `nvdcve` (
  `cve_id` varchar(13) NOT NULL,
  `published` datetime default NULL,
  `modified` datetime default NULL,
  `score` DECIMAL(5,2) default '0.0',
  `severity` varchar(6) default 'LOW',
  `vector` varchar(25) default NULL,
  `complexity` varchar(25) default NULL,
  `authentication` varchar(25) default NULL,
  `confidentiality` varchar(25) default 'NONE',
  `integrity` varchar(25) default 'NONE',
  `availability` varchar(25) default 'NONE',
  `summary` varchar(512) default NULL,
  PRIMARY KEY  (`cve_id`),
  INDEX (score),
  INDEX (vector)
)

Using Perl Populating the Database

Populating the database table is simply a matter of reading the file and adding the entries to the table. An example Perl subroutine follows:

sub readxml {
   my($nvd_file, $dbh) = @_;
   my $parser = XML::LibXML-> new();
   my $doc    = $parser-> parse_file($nvd_file);
   my $xc     = XML::LibXML::XPathContext-> new( $doc->documentElement() );
   $xc-> registerNs(
      def  => 'http://scap.nist.gov/schema/feed/vulnerability/2.0' );
   $xc-> registerNs(
     vuln => 'http://scap.nist.gov/schema/vulnerability/0.4' );
   $xc-> registerNs( cvss => 'http://scap.nist.gov/schema/cvss-v2/0.2' );
   for my $entry ($xc-> findnodes("/def:nvd/def:entry")) {
      my $cve = $xc-> find('vuln:cve-id',$entry);
      my $published = $xc-> find('vuln:published-datetime', $entry);
      my $modified = $xc-> find('vuln:last-modified-datetime', $entry);
      my $summary = $xc-> find('vuln:summary', $entry);
      my $skip = 0;
      my ($metrics) = $xc-> findnodes('vuln:cvss/cvss:base_metrics', $entry) or ($skip = 1);
      if (! $skip) {
         my $score = $xc-> find('cvss:score', $metrics);
         my $vector = $xc-> find('cvss:access-vector', $metrics);
         my $complexity = $xc-> find('cvss:access-complexity', $metrics);
         my $authentication = $xc-> find('cvss:authentication', $metrics);
         my $confidentiality =
            $xc-> find('cvss:confidentiality-impact', $metrics);
         my $integrity = $xc-> find('cvss:integrity-impact', $metrics);
         my $availability = $xc-> find('cvss:availability-impact', $metrics);
         my $severity = "LOW";
         if (int($score) >= 7) {
            $severity = "HIGH";
         }
         elsif (int($score) >= 4) {
            $severity = "MEDIUM";
         }
         my $sql = qq{ SELECT count(*) FROM nvdcve WHERE cve_id=? };
         my $sth = $dbh->prepare( $sql );
         my $rc = $sth->execute($cve);
         if ( $rc) {
            my($exist) = $sth->fetchrow_array();
            if (! $exist) {
                $sql = qq{ INSERT INTO nvdcve SET cve_id=?,
published=?, modified=?, score=?, severity=?, vector=?, complexity=?,
authentication=?, confidentiality=?, integrity=?,availability=?, summary=? };
               $sth = $dbh->prepare( $sql );
               $rc = $sth->execute($cve,$published,$modified,$score,
$severity,$vector,$complexity,$authentication,
$confidentiality,$integrity,$availability,$summary);
            }
         }
      }
   }
}

The Perl Program to Pull It All Together

The above subroutines use the Perl modules LWP::UserAgent, XML::LibXML, XML::LibXML::XPathContext, and DBI. A sample Perl program that calls the above subroutines to pull down the NVD CVE data and load it into a MySQL table would be:

#!/usr/local/bin/perl -w
use LWP::UserAgent;
use XML::LibXML;
use XML::LibXML::XPathContext;
use DBI;
BEGIN{push @INC, "/home/jgerber/projects/nvd/perl"}
use nvdsubs qw($db_host $db $mysql_user $mysql_passwd $mysql.sock
readpage readxml );
# Main
my $datadir = "/home/johngerber/projects/nvd/data";
my @timeData = localtime(time);
my $year = 1900 + $timeData[5];
my $prev_year = 1900 + $timeData[5] - 1;
my $url = "http://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-" .
    $year . ".xml";
my $prev_url = "http://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-" .
    $prev_year . ".xml";
my $nvd_file = $datadir  . "/nvdcve-". $year . ".xml";
my $prev_nvd_file = $datadir  . "/nvdcve-". $prev_year . ".xml";
$db = "vulnerabilities";
local($dbh) = DBI->connect("DBI:mysql:mysql_socket=$mysql.sock;$db:$db_host",
$mysql_user, $mysql_passwd) || die "ERROR: Connecting: $DBI::errstr\n";
my ($pjstatus) = &readpage($prev_url,$prev_nvd_file);
if ($pjstatus) {
   &readxml($prev_nvd_file,$dbh);
}
$pjstatus = &readpage($url,$nvd_file);
if ($pjstatus) {
   &readxml($nvd_file,$dbh);
}
exit;

The nvdsubs.pm file will not be included in this post. The subroutines are defined and the only pieces missing are the MySQL database username and password. You don’t need mine. Add your own. At this point, we have everything we need to finally use Google Visualization to create a graph.

Google Visualization

We are going to create a Perl program that will read our MySQL nvdcve table and generate the JavaScript that will render our charts on the client’s browser. First, we want to define the JavaScript we want to produce. Just to alleviate some concerns, with Google Visualization your data is only shared between your server and the client connecting. This is unlike Google Charts where your data is sent to Google where it is made into a chart and the result is sent back. Google states concerning the logging of chart data (via Google Charts), “The chart data included in the HTTP request is saved in temporary logs for no longer than two weeks for internal testing and debugging purposes.” Every example in the Google Visualization Gallery will state the data policy. For Google Charts, stated at the bottom of the page for each gadget description the data policy:

While Google Visualization gadgets will have the following stated data policy:

Loading Google Libraries

The first thing the JavaScript needs to do is load the required libraries. This is accomplished with the lines:

Area Chart and Table

In this example we are going to create an column chart. In a later section, “Other Charting Options” (see below) we define different Google Visualization charting options.

JavaScript code for a sample column chart would be:

    <script type='text/javascript'>
      google.load('visualization', '1', {packages:['columnchart']});
      google.setOnLoadCallback(drawChart);
      function drawChart() {
        var data = new google.visualization.DataTable();
        data.addColumn('date', 'Date');
        data.addColumn('number', 'High');
        data.addColumn('number', 'Medium');
        data.addColumn('number', 'Low');
        data.addRows([
           [new Date(2009, 0, 30),92,97,3],
           [new Date(2009, 1, 27),168,142,25],
           [new Date(2009, 2, 31),141,165,9],
           [new Date(2009, 3, 30),132,203,12],
           [new Date(2009, 4, 29),158,153,8],
           [new Date(2009, 5, 30),200,199,22],
           [new Date(2009, 6, 31),190,195,11],
           [new Date(2009, 7, 31),127,139,14],
           [new Date(2009, 8, 30),233,208,14],
           [new Date(2009, 9, 30),163,167,18],
           [new Date(2009, 10, 30),129,172,8],
           [new Date(2009, 11, 31),200,211,19],
           [new Date(2010, 0, 29),157,139,14],
           [new Date(2010, 1, 26),137,143,12],
           [new Date(2010, 2, 31),252,242,18],
           [new Date(2010, 3, 13),92,118,17]
        ]);
        var chart = new google.visualization.ColumnChart(document.getElementById('s4graph'));
        chart.draw(data, {displayAnnotations:true, is3D: true, isStacked: true, min: 0,
          allowHtml: true, colors:[{color:'#E41B17', darker:'#C11B17'}, {color:'#FFA500', darker:'#E56717'}, {color:'#FFE87C', darker:'#C8B560'}]});
      }
    script>

The resulting image would be the following column chart:

Rendering the Table

When providing qualitative results, I like to back them up with more accurate numeric values. Let us include a table with links to the CVSS scores for each vulnerability.

    <script type='text/javascript'>
      google.load('visualization', '1', {packages:['table']});
      google.setOnLoadCallback(drawChart);
      function drawChart() {
        var data2 = new google.visualization.DataTable();
        data2.addColumn('date', 'Date');
        data2.addColumn('number', 'High');
        data2.addColumn('number', 'Medium');
        data2.addColumn('number', 'Low');
        data2.addRows([
           [{v:new Date(2009, 0, 30),
              f:'2009-01-30'}, 92,97,3],
           [{v:new Date(2009, 1, 27),
              f:'2009-02-27'}, 168,142,25],
           [{v:new Date(2009, 2, 31),
              f:'2009-03-31'}, 141,165,9],
           [{v:new Date(2009, 3, 30),
              f:'2009-04-30'}, 132,203,12],
           [{v:new Date(2009, 4, 29),
              f:'2009-05-29'}, 158,153,8],
           [{v:new Date(2009, 5, 30),
              f:'2009-06-30'}, 200,199,22],
           [{v:new Date(2009, 6, 31),
              f:'2009-07-31'}, 190,195,11],
           [{v:new Date(2009, 7, 31),
              f:'2009-08-31'}, 127,139,14],
           [{v:new Date(2009, 8, 30),
              f:'2009-09-30'}, 233,208,14],
           [{v:new Date(2009, 9, 30),
              f:'2009-10-30'}, 163,167,18],
           [{v:new Date(2009, 10, 30),
              f:'2009-11-30'}, 129,172,8],
           [{v:new Date(2009, 11, 31),
              f:'2009-12-31'}, 200,211,19],
           [{v:new Date(2010, 0, 29),
              f:'2010-01-29'}, 157,139,14],
           [{v:new Date(2010, 1, 26),
              f:'2010-02-26'}, 137,143,12],
           [{v:new Date(2010, 2, 31),
              f:'2010-03-31'}, 252,242,18],
           [{v:new Date(2010, 3, 13),
              f:'2010-04-13'}, 92,118,17],
        ]);
        var table = new google.visualization.Table(document.getElementById('s4graph_tab'));
        table.draw(data2, {showRowNumber: true, sortAscending: false, sortColumn: 0, allowHtml: true});
      }
    script>

The JavaScript code assumes there is a PHP program called cvealerts.php under the /nvd directory on your web server. Adjust to your environment. A sample PHP program that could be used for cvealerts.php is provided below. The resulting table chart would look like:

Handling Events: Interactions Between Graphs

We now have two different types of graphs representing the same data. We want to add interaction between the graphs so the viewer can see the relationship. With tables rows are selected when the user clicks, which correspond to the whole column of the stacked column chart. It is not a perfect fit, but it does demonstrate nicely use of adding interactions.

        // Set a 'select' event listener for the table.
        // When the table is selected,
        // we set the selection on the line graph.
        google.visualization.events.addListener(table, 'select', function() {
          chart.setSelection([{row: table.getSelection()[0].row, column: 1}]);
         });
        // Set a 'select' event listener for the graph.
        // When the graph is selected,
        // we set the selection on the table.
        google.visualization.events.addListener(chart, 'select', function() {
           table.setSelection([{row: chart.getSelection()[0].row}]);
        });

Providing Detailed Information

When the table chart link is clicked, we would like to provide some detailed information about the vulnerability. For this example, we will do this with a simple PHP program placed in the /nvd directory on the web server. The program is called cvealerts.php.


session_start();
function db_connect($table) {
   $result = mysql_pconnect(":", "", "");
   if (!$result) return false;
   if (!mysql_select_db($table)) return false;
   return $result;
}
function do_html_header($title,$checkuser,$logpage) {
?>
  <html> <head> <title>$title?>title>head>
  <body bgcolor="#FFFFFF">

}
function do_html_footer() {
?>
<table>
<tr><td ALIGN=CENTER NOWRAP WIDTH="590">font>
<font face="Verdana, Arial, Helvetica" size=-2>Notice to Users: Use
of this system constitutes consent to security monitoring and testing.
<br>All activity is logged with your host name and IP address.font>
td>tr>
table>
body>
 html>

}
// Main
$dates= array();
$stringlist = "";
if (isset($_GET['date'])) {
    $passdates = explode(",",$_GET['date']);
    for ($index=0; $index<count($passdates); $index++) {
       array_push($dates, $passdates[$index]);
       $stringlist .= $passdates[$index] . " ";
    }
}
else {
  print("Confusion over how you arrived at this page.\n");
  exit;
}
$stringlist = preg_replace("/ $/", "",$stringlist);
do_html_header("Review NVD CVE Announcements for Month Ending $stringlist",1,1);
$nvd_host = "http://web.nvd.nist.gov/view/vuln/detail?vulnId=";
$conn = db_connect("vulnerabilities");
if (!$conn)
   logit("Could not connect to database vulnerabilities - please try later.\n",1);
for ($index=0; $index<count($dates); $index++) {
   $rule = $dates[$index];
   $sql = "SELECT cve_id,score,published,vector,severity,complexity,left(summary,50)
    FROM vulnerabilities.nvdcve
      WHERE date_format(published,'%Y-%m')='$rule'
       ORDER BY (score+0)";
   $result = mysql_query($sql,$conn);
   if (!$result)
       logit("Problem with $sql\n",1);
   print("
\n");for($count=1;list($cve_id,$score,$date,$vector,$severity,$complexity,$shortsum)=mysql_fetch_array($result, MYSQL_NUM);++$count){?><tr><td CLASS="plfieldhdrleft" WIDTH="20%" BGCOLOR='#F0F5FF'>  print("$cve_id"); ?>
      td>
      <td CLASS="plfieldhdrleft" BGCOLOR='#F9FCFF'>
        print($score); ?>
      td>
      <td CLASS="plfieldhdrleft" BGCOLOR='#F0F5FF'>
        print($date); ?>
      td>
      <td CLASS="plfieldhdrleft" BGCOLOR='#F9FCFF'>
        print($vector); ?>
      td>
      <td CLASS="plfieldhdrleft" BGCOLOR='#F0F5FF'>
        print($severity); ?>
      td>
      <td CLASS="plfieldhdrleft" BGCOLOR='#F9FCFF'>
        print($complexity); ?>
      td>
      <td CLASS="plfieldhdrleft" BGCOLOR='#F0F5FF'>
        print($shortsum); ?>
      td>
      tr>

   }
}
print("
Bulletin 
Impact 
Date 
Vector 
Severity 
Complexity 
Short Summary
            
        

      
      
");
do_html_footer();

The PHP program would generate a HTML table displaying the NVD CVE alerts for that month. The table would look like:

When the CVE link is clicked on, the user is taken to the NIST NVD site where additional information is available.

Using Perl to Create the JavaScript

The Perl code is rather simple now that we have the MySQL tables defined and the JavaScript we want to generate. Much of the code consists of the JavaScript listed above.

#!/usr/local/bin/perl -w
use DBI;
use Time::Local;
use POSIX qw(strftime);
use LWP::UserAgent;
BEGIN{push @INC, "/home/jgerber/projects/nvd/perl"}
use ornl_feds qw($db_host $db $mysql_user $mysql_passwd );
sub slide_nvd_alerts {
  my($min_date,$graph_name,$web_link,$dbh) = @_;
  my $slide = "";
  my $slide_head = qq!
    
!;
   if ($min_date eq "") {
      my $sql2 = qq{ SELECT min(published) FROM vulnerabilities.nvdcve };
      my $sth2 = $dbh->prepare( $sql2 );
      my $rc2 = $sth2->execute();
      if ($rc2) {
         $min_date = $sth2->fetchrow_array();
      }
   }
   my $table_data = "";
   my $graph_data = "";
   my $sql2 = qq{ select date_format(published,'%Y-%m'),severity,count(severity)
      FROM vulnerabilities.nvdcve where published >= ? group by date_format(published,'%Y-%m'),severity };
   my $sth2 = $dbh->prepare( $sql2 );
   my $rc2 = $sth2->execute($min_date);
   if ($rc2) {
      my ($change,$virgin,$ht,$mt,$lt,$mmax_date) = ("",1,0,0,0,"");
      while (my($snapshot_date, $severity, $pcount) = $sth2->fetchrow_array()) {
         my $sql3 = qq{ SELECT max(published) FROM vulnerabilities.nvdcve where
date_format(published,'%Y-%m')=? };
         my $sth3 = $dbh->prepare( $sql3 );
         my $rc3 = $sth3->execute($snapshot_date);
         $max_date =  $sth3->fetchrow_array();
         $max_date =~ s/ \S+$//;
         if ($change ne $snapshot_date) {
            if (! $virgin) {
                my($year,$month,$day) = split("-",$mmax_date);
                my $mmonth = $month;
                $month--;
                $graph_data .= qq!           [new Date($year, $month, $day),$ht,$mt,$lt],
!;
                $table_data .= qq!           [{v:new Date($year, $month, $day),
              f:'$mmax_date'}, $ht,$mt,$lt],
!;
                ($ht,$mt,$lt) = (0,0,0);
             }
             $change = $snapshot_date;
          }
          if ($severity eq "HIGH") { $ht = $pcount; }
          elsif ($severity eq "MEDIUM") { $mt = $pcount; }
          elsif ($severity eq "LOW") { $lt = $pcount; }
          if ($mmax_date eq "") { $mmax_date = $max_date; }
          if ($mmax_date lt $max_date) { $mmax_date = $max_date; }
          $virgin = 0;
      }
      my($year,$month,$day) = split("-",$mmax_date);
      my $mmonth = $month;
      $month--;
      $graph_data .= qq!           [new Date($year, $month, $day),$ht,$mt,$lt]
!;
     $table_data .= qq!           [{v:new Date($year, $month, $day),
              f:'$mmax_date'}, $ht,$mt,$lt],
!;
   }
   $table_data .= "        ]);\n";
   $graph_data .= "        ]);\n";
   $slide = $slide_head .  $graph_data . $slide_head_table . $table_data . $slide_tail;
   return($slide);
}
sub slide_body {
  my($graph_name,$title,$style) = @_;
  my $table_name = $graph_name . "_tab";
  my $table_text = "div id=\"$table_name\"";
  if ($style ne "") {
     $table_text .= " style=\'$style\'";
  }
  my $slide2 = "$title
\n";
  my $itext = "div id=\"$graph_name\"";
  if ($style ne "") {
     $itext .= " style=\'$style\'";
  }
  $slide2 .= qq{
    
    <$itext>
    <$table_text>
       
    
    
  };
  return($slide2);
}
# Main
my $web_link = "/nvd";
my $results_dir = "/data/html" . $web_link;
my $result_file = $results_dir . "/nvdcve_stats.html";
my $debug = 1;
my $db = "vulnerabilities";
local($dbh) = DBI->connect("DBI:mysql:$db:$db_host", $mysql_user, $mysql_passwd) ||
   die "ERROR: Connecting: $DBI::errstr\n";
$slides_data .= &slide_body("s4graph","NVD CVE Alerts","width:700px; height:400px;");
$slides_head .= &slide_nvd_alerts("","s4graph",$web_link,$dbh);
open(OUTFILE,">$result_file");
print OUTFILE "\nNVD CVE Statistics\n";
print OUTFILE "\n";
print OUTFILE $slides_head;
print OUTFILE "\n\n";
print OUTFILE $slides_data;
print OUTFILE "\n";
close(OUTFILE);
exit;

Other Charting Options

Google, Google users, and other companies have shared some JavaScript visualizations built on the Google Visualization API to help you get started. Below are some example:

	Annotated Time Line (GWT Integrated) An animated time series chart. By: Google
	Area Chart (GWT Integrated) Interactive area chart. By: Google
	Bar Chart (GWT Integrated) Interactive bar chart. By: Google
	Bars of Stuff Fun bar charts using images of trains, chocolate, worms, and more. By: The visapi project
	Bio Heat Map Heatmaps are a useful way to visualize matricies of data. Scientists often use green-black-red heatmaps to visualize gene expression data from microarrays. This visualization supports both three color heatmaps (ex: green to black to red) and two color heatmaps (ex: white to yellow). By: Institute for Systems Biology
	Column Chart (GWT Integrated) Interactive column chart. By: Google
	Drastic Treemap A dynamic treemap in Flash. By: DrasticData
	Dygraphs The dygraphs JavaScript library produces interactive, zoomable charts of time series. By: Dan Vanderkam
	Filters A Visualization that acts as a control over other visualizations. It is rendered within the browser using HTML. This visualization offers the ability to select some criteria to filter the DataTable used by the controlled visualizations. By: Institute for Systems Biology
	Gauge (GWT Integrated) Each numeric value is displayed as a gauge. By: Google
	Geo Map (GWT Integrated) A map of a country, continent, or region map, with colors and values assigned to specific regions. Values are displayed as a color scale, and you can specify optional hovertext for regions. By: Google
	Intensity Map (GWT Integrated) An intensity map that highlights regions or countries based on relative values. By: Google
	Line Chart (GWT Integrated) Interactive line chart. By: Google
	Magic-Table The Magic Table is a JavaScript library that allows you to see more in your data by applying some simple visual techniques to transform a table. The table is displayed in the browser by the canvas element. Internet Explorer is not supported. By: Greg Ross
	Map (GWT Integrated) An interactive map that uses the Google Maps API. By: Google
	Motion Chart (GWT Integrated) Motion Chart: A dynamic flash based chart to explore several indicators over time. Required columns: bubble name, time and 2 columns of numeric values. Optional columns: Numeric values or categories. By: Google
	Organizational Chart A GWT Integrated simple organizational chart. By: Google
	Parallel Coordinates Chart Parallel Coordinates is a method of visualizing multivariate data. An n-dimensional space is represented as n parallel lines. Works for browsers based on Gecko or Presto (does not work in IE). This is written in Javascript, no Flash required. By: Sri Harsha Allamraju
	Pie Chart (GWT Integrated) Interactive pie chart. By: Google
	Piles of Money Column chart made of of money bills. By: The visapi project
	Scatter Chart (GWT Integrated) Interactive scatter chart. By: Google
	Table (GWT Integrated) A highly customizable table with sorting, paging and selection capabilities. By: Google
	TermCloud A list of terms, where the size and color of each word is determined by a specified frequency value (typically the number of times it appears in some text). By: The visapi project
	Thematic Mapping API Enables visualization of data in Google Earth or other geobrowsers through the use of the Google Visualization API and KML. By: Thematicmapping.org
	WordCloud Displays all words in text with size and color based on the number of time each word appears. By: The visapi project

Additional Information

Below is the talk that Itai Raz, the lead engineer for the Visualization API product at Google, gave at Google I/O 2009 titled “Using the Visualization API with GWT:”

Additional Possibilities

The work above is meant only to serve as a starting point. There is a great deal more information to expand upon. For example, we began this post pulling some information from the XML schema for CVE-2010-1228. One field we did not pull out from the XML file is:

The Common Weakness Enumeration (CWE) represents vulnerability types and NIST provides a CWE Cross Section Mapped into by NVD table. In the above example, we see an entry:

Name	CWE-ID	Description
Race Conditions	CWE-362	The state of a resource can change between the time the resource is checked to when it is accessed.

Clicking on the link will take us to the MITRE site that provides a great deal more information on CWE entries. It is easy enough to expand on the above program to harvest this information for a richer information database.

Another possibility is to expand the above program to pull additional information on the CVE entry. In additional to the data in the NVD CVE XML file, we could pull information from the NVD site. Using CVE-2010-1228 as an example, we could have the program pull down the page:

http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2010-1228

Notice the line:

CVSS v2 Base Score:10.0 (HIGH) (AV:N/AC:L/Au:N/C:C/I:C/A:C) (legend)

The (AV:N/AC:L/Au:N/C:C/I:C/A:C) provides values that were used in determining the base score. If you follow the link, you will see the values used in the calculations:

CVSS Base Score: 10
- Impact Subscore: 10
- Exploitability Subscore: 10
CVSS Temporal Score: Undefined
CVSS Environmental Score: Undefined
Overall CVSS Score: 10

NVD has made available the equations used in calculating the CVSS base score, temporal score, and environmental score.

Three other pieces of information that might provide interesting groupings are:

Access Complexity: Low **NOTE: Access Complexity scored Low due to insufficient information
Authentication: Not required to exploit
Impact Type: Allows unauthorized disclosure of information; Allows unauthorized modification; Allows disruption of service

What information is of interest and how it is used will be dependent on your organization. There is a great deal of information available and many directions you start examining.

Final Thoughts

I am often reminded of the old phrase, “Trust us, we are from the government.” No one really trusts anyone, especially when it comes to matters they do not understand. Just because you are from the security group at your organization, is that reason enough for the CEO to give you unlimited money and authority to do what you see fit? Of course not. While management might trust you, they may not believe that you are capable of seeing the big picture. That is after all their job.

Another great old saying is that “the devil is in the details.” Those details will likely fall in the security domain. In organization across the planet there is a tug of war between the details and the big picture with multiple groups adding in their opinions and views. You need to make the details understandable to your higher management to effectively argue your view. Finding effective metrics and finding clear representation is essential in today’s business. Google Visualization can be a useful tool in accomplishing this task.

OMB Says Bring on the Clouds: Frightening or Funny?

John Gerber — Mon, 18 Jan 2010 23:13:04 +0000

Jason Miller, Executive Editor for FederalNewsRadio, write in his article, “Agencies to justify not using cloud computing to OMB” that OMB “will require agencies to develop an alternative analysis discussing how they could use cloud computing for all major technology projects for the fiscal 2012 budget.” This is according to an internal budget documents obtained by FederalNewsRadio. The document details OMB’s plans for such high-profile initiatives such as data center consolidation and the use of cloud computing and cybersecurity spending.

Miller goes on to report that OMB will require “agencies launch a series of cloud computing pilots across the government in 2010 using the E-Government Fund.” In 2013, Miller reports, agencies must provide OMB “a complete alternatives analysis for mixed life cycle projects where agencies are spending new money-known as development, modernization and enhancement-and steady state or operations and maintenance funding for how they could move to cloud computing.”

Miller quotes a former government official as saying, “They are not saying use it, but are pushing us to look at it and do an analysis of alternatives and make a decision based on our business needs. They are pushing us to look at it, yet giving us the ability to decide whether it makes sense.”

How well does your organization understand cloud computing? How will security be handled? What can you do to prepare? During this time of tight budgets, maybe you do not have the funds and/or time to attend conferences and training events. Fortunately, presentations are being posted regularly to the web, allowing you to keep informed on technological challenges. For example, the ZISC Workshop on Security in Virtualized Environments and Cloud Computing, held September 10-11th in Zurich, recently posted all their presentations:

Welcome note	Bernhard Plattner and Diego Zamboni
Talk 1: Not Every Cloud has a Silver Lining	Gunter Ollmann, Damballa Inc., Atlanta GA, USA
Talk 2: Virtualization and Cloud Computing: Security’s Golden or Gilded Age	Kevin Skapinetz, IBM Internet Security Systems, Atlanta GA, USA
Talk 3: Using virtualization technology for fault and intrusion tolerance	Hans P. Reiser, University of Lisbon, Portugal
Talk 4: A survey of current security-related operating systems research	Timothy Roscoe, ETH Zurich, Switzerland
Talk 5: Of Cold Steam, Mist and Vapour: A View from the Inside of the Cloud	Dirk Kuhlmann, HP Labs Bristol, UK
Talk 6: New Cloud Computing challenges: the security impact in the “social” world.	Massimo Villari, University of Messina, Italy
Talk 7: Paradigms in virtualization based host security	Tal Garfinkel, VMware Inc., Palo Alto, CA, USA / Stanford University, Palo Alto CA, USA
Talk 8: Cloud Computing and Security: a Googley Perspective	Peter Dickman, Google Inc., Zurich, Switzerland
Talk 9: A NIST Perspective on Cloud Computing	Tim Grance, National Institute of Standards and Technology, USA
Talk 10: ENISA Risk Assessment of Cloud Computing – Preliminary Results	Giles Hogben, ENISA, EU
Talk 11: Attack Graphs + Mechanically Generated Constraints	Lee Badger, National Institute of Standards and Technology, USA
Wrap-up and end	Bernhard Plattner and Diego Zamboni

Following NIST’s involvement in an area like cloud computing can help you judge the direction the government is heading. Tim Grance presented at the 5th Annual IT Security Automation Conference and Expo Presentations and the presentations have been made available. Grance presented on the Security Content Automation Protocol (SCAP) (see my previous post “Standardization and Interoperability in Security” for additional information on SCAP). A cloud computing track consisting only of slides (no video) was also posted. If lack of video does not concern you, the following conferences have posted slides on cloud security:

CCSW 2009: The ACM Cloud Computing Security Workshop, held November 13th, 2009 in Chicago.
Digital Government Institute’s Cloud Computing 2010: Focus on Operational Efficiency and Security, held December 9, 2009.
Cloud Interoperability Roadmaps Session held in Long Beach, CA on December 10, 2009.

If you prefer to listen and do not need to see slides, Tim Grance can be heard on Dana Gardner’s BriefingsDirect podcast, “Panel Discussion: Is Cloud Computing More or Less Secure than On-Premises IT?.” The discussion includes a panel of all stars from the cloud security community, including Glenn Brunette, distinguished engineer and chief security architect at Sun Microsystems and founding member of the Cloud Security Alliance (CSA); Doug Howard, chief strategy officer of Perimeter eSecurity and president of USA.NET; Christofer Hoff, technical adviser at CSA and director of Cloud and Virtualization Solutions at Cisco Systems; and Dr. Richard Reiner, CEO of Enomaly. The podcast was recorded at the Open Group’s 23rd Enterprise Architecture Practitioners Conference in Toronto on July 20-22, 1009, along with:

Jericho Forum Aims to Guide Enterprises Through Risk Mitigation Landscape for Cloud Adoption where Dana interviews Steve Whitlock, a member of the Jericho Board of Management.
Cloud and Security Join Boundaryless Information as Top-of-Mind Issues for The Open Group where Dana talked with Allen Brown, president and CEO of The Open Group.
XDAS Standard Aims to Empower IT Audit Trails from Across Complex Events where Dana talks with Ian Dobson, director of the Security Forum for The Open Group, as well as Joël Winteregg, CEO and co-founder of NetGuardians. XDAS is an open-source standard that is hopefully going to help in compliance and regulatory issues and in the automation of heterogeneous environments.
New Era Enterprise Architects Need Sweeping Skills to Straddle the IT-Business Alignment Chasm where Dana is joined by James de Raeve, vice president of certification at The Open Group; Len Fehskens, vice president, Skills and Capabilities at The Open Group; David Foote, CEO and co-founder, as well as chief research officer, at Foote Partners, and Jason Uppal, chief architect at QRS.
Cloud Pushes Enterprise Architects’ Scope Beyond IT into Business Process Optimization Role where Dana is joined by Tim Westbrock, managing director of EAdirections; Sandy Kemsley, an independent IT analyst and architect; and John Gotze, international president for the Association of Enterprise Architects.

For more video presentations on the cloud security, awhile back I posted “CERT, CERIAS, the Academy, and Google Video: Training Online.” Two other sources include the SecurityTube and O’Reilly Webcasts. Below are a few examples of the presentations available:

The Belgian Beer Lovers Guide to Cloud Security (Brucon 2009) Tutorial by Craig Balding at Brucon 2009: In this presentation Craig covers why talking about “cloud” is akin to walking into a Belgian bar and asking for “beer”; the common cloud architectures and their implications for you – the security dude; what the beer brewing Trappist Monks can teach us about cloud security; attacking clouds (aka getting free beer); and dealing with the hangover: cloud incident response & forensics.
Evolution of Security (Fsecure) Tutorial by F-Secure: an animated series on the various threats out there on the Internet and also talks about their state of the art AV (self promotion) They also talk about “cloud security” and how the next generation AV will be in the cloud and not isolated.
Cloud Security and Privacy by Tim Mather, Subra Kumaraswamy, Shahed Latif: discusses cloud computing’s SPI delivery model, and its impact on various aspects of enterprise information security (e.g., infrastructure, data, identity and access management, security management), privacy, and compliance. Security-as-a-Service and the impact of cloud computing on corporate IT is also discussed.
Architecting Applications for the Cloud by Jorge Noa: This presentation analyzes aspects of the Amazon EC2 IaaS cloud environment that differ from a traditional data center and introduces general best practices for ensuring data privacy, storage persistence, and reliable DBMS backup.
Cloud Computing: The Next Frontier for Open Source by Bernard Golden: discusses how the trends of open source and cloud computing reinforce one another, and why cloud computing is a significant driver of enterprise open source adoption.
Getting Started with Amazon Web Services by Cloud Security Deep Dive by Subra Kumaraswamy, Shahed Latif, Tim Mather: will take a deep dive into cloud security issues and focus on three specific aspects: (1) data security; (2) identity management in the cloud, and; (3) governance in the cloud (in the context of managing a cloud service provider with respect to security obligations). Each of these three topics will be covered in a 30 minute segment that will include a presentation and Q&A with the audience.
Cloudburst (Hacking 3D and Breaking Out of VMware) Blackhat 2009 by Kostya Kortchinsky: VMware products include implement a lot of functionality, and as such have a decent chance to include some bugs. CLOUDBURST is the combination of 3 of those found in the virtualized video device (more specifically the 3D code). Combined, these allow a user in a Guest to execute code on the Host. Since the virtualized device code is the same for all the branches of the products, this impacts Workstation, as well as Fusion or ESX. Immunity, Inc. will present the various vulnerabilities and the techniques used to exploit the bug reliably, even on platforms with ASLR or DEP such as Vista SP1. Once exploited, Immunity will demonstrate how to establish MOSDEF between the Host and Guest.
Virtualization: Resource Coupling and Security across the Stack by Dennis Moreau, Configuresoft: The session briefly addressed extension to the cloud and utility computing infrastructures to address how to use configuration and behavioral information to address the increased complexity of security, compliance and risk assessment in virtualized environments.

Other BruCON Security Conference (held September 18-19, 2009) videos are available at their vimeo channel. O’Reilly maintains on YouTube an O’Reilly Media Channel along with an area to sign up for future webcasts. Blackhat DC 2009 video, audio, whitepapers, and slides are also available. Content is ever changing, so keep checking the sites.

Remember that Vivek Kundra, Chief Information Officer (CIO) of the United States of America, outlined as his team’s priorities:

Innovation
Lowering the cost of Government
Transparency
Engaging Citizens
Ensuring a safe computing environment

In response, FedScoop! started hosting one event each quarter around these pillars. On October 14 at the Newseum, they did their first event bringing together executives in the White House and federal CIO’s, CTO’s, and decision-makers to talk about lowering the cost of government with technology. Check out the video of the Cyber Security Panel. Since one of the topics was cloud computing, FedScoop! scheduled a follow-up event. On December 9th, 2009, they hosted and posted the “Cloud Computing Shoot Out.”

FederalNewsRadio has posted a three part video series on secure cloud computing. The panelists include Jim Flyzik, President of the Flyzik Group; Henry Sienkiewicz, Technical Program Director, Computer Services, Defense Information Systems Agency; Ronald Bechtold, Army Architecture Integration Center at Headquarters, Department of the Army, Chief Information Office/G6; Curt Aubley, Chief Technology Officer CTO Operations & Next Generation Solutions, Lockheed Martin Information Systems & Global Services; Dale Wickizer, Chief Technology Officer-Public Sector, NetApp, Inc.; and Aileen Black, Vice President of Public Sector VMware Inc.

CNET’s editor of Webware, Rafe Needleman and senir writer Stephen Shankland talked with Christofer Hoff on the Reporters’ Roundtable podcast about the “Dangers of Cloud Computing.” Chris also presented at Microsoft’s BlueHat, “Cloudifornication: Indiscriminate Information Intercourse Involving Internet Infrastructure.” Any presentation with such a great title must be watched. There is a short interview with Chris from Bluehat.

One of my favorite stories of Abraham Lincoln involved the McCormick-Manny case of 1855 where Lincoln was one of Manny’s lawyers. Lincoln basically was pushed aside and humiliated. After the trial, he told Ralph Emerson, a young lawyer who was present at the trial, “I am going home. I am going home to study law.” Emerson asked, “Mr. Lincoln, you stand at the head of the bar in Illinois now! What are you talking about?” Lincoln replied, “Ah, yes, I do occupy a good position there, and I think that I can get along with the way things are done there now. But these college-trained men, who have devoted their whole lives to study, are coming West, don’t you see? And they study their cases as we never do. They have got as far as Cincinnati now. They will soon be in Illinois.” Emerson stated Lincoln turned to him, his countenance suddenly assuming that look of strong determination which those who knew him best sometimes saw upon his face, and said, “I am going home to study law! I am as good as any of them, and when they get out to Illinois, I will be ready for them.”

Change is coming. If you try just to get along, the future will overwhelm you. While we do not live in a world of unlimited funds for conferences and training, people are sharing a wealth of information. Take advantage of it and get ready for whatever might be heading your way.

Implementing a Web Application Firewall with ModSecurity

John Gerber — Fri, 01 Aug 2008 04:02:27 +0000

There are a few topics I have meant to do a post on for awhile. Sometimes having too much interest and information on a topic can be a bad thing. Wanting to pull various postings and articles along with implementation instructions can be a bit time consuming. Plus, in order to demonstrate why someone might be interested in implementing a web application firewall (WAF), I envisioned a post discussing the collapse of the perimeter and addressing points made by the Jericho Forum (see the Security Roundtable podcast for June 2008 for a good discussion on the topic). Raffael Marty makes excellent points in his soon to be released book, “Applied Security Visualization“:

The crime landscape is shifting. Attacks are moving up the network stack. Network-based attacks are not the prime source of security problems anymore. The attacks today are moving into the application layer: Web 2.0, instant messenger attacks, fraud, information theft, and crime-ware are just some examples of new types of attacks that generate a load of data to be collected and analyzed. Beware! Applications are really chatty and generate alot of data.

While my current post is not about security visualization (see earlier post “Security Data Visualization“), I would like to point out that DAVIX, a live CD for data analysis and visualization, is expected to be released August 6th. That should be really cool and fun.

Since application security is a topic of interest for me, I ran into the problem of having too many topics I wanted to discuss when I started trying to write a post on ModSecurity, an open source, free WAF Apache module. Today, rather than waiting for me to integrate the information, I decided to move ahead and do the post while limiting myself to only pointing out the various sources. The reader can follow the links for a more in-depth discussion and understanding on the topic.

Why You Should Care

The Risky Business podcast has come to be one of my favorite podcasts. The host, Patrick Gray and regular guest Munir Kotadia, just cracks me up. Plus the show is informative and features great guests. This week’s show had an interview with H D Moore talking about the DNS bug. Timely and informative; what else can one ask for? The 68th episode, done at the beginning of this month, had an interview with Jeremiah Grossman concerning web application firewalls. As Patrick writes in the show notes, “it takes typical organizations around 130 days to fix sequel injection bugs in code. But you can mitigate these sorts of things with a Web app firewall, and you won’t even have to deal with the development team! Hooray!.”

In Grossman’s blog post “Can WAFs protect against business logic flaws?“, he pointed out that “WAFs don’t defend against every logic flaw, or even every crazy form of SQLi or XSS. Just as white/black box scanners can’t identify every vulnerability and neither can expert pen-testers or source code auditors.” Stuart King, in his article, “Larry David and Web Application Firewalls“, builds upon this idea when he wrote:

Back to the CSO article where the point is made that we are sitting on a huge legacy of insecure code and that “we can’t rewrite history.” So, the argument is that a web application firewall mitigates the risk – note: does not solve the problem – until the code can be replaced.

How much of the risk is mitigated is open to debate, but there are lots of other things to consider too. For instance the cost of redeveloping code against the cost of purchasing and supporting a WAF. We also need to consider the value and risk profile of the product.

Today’s world consist of attackers adjusting focus from network-based attacks to the application layer. Grossman in his post “Website Security Strategies that Work” makes the claim that “9 out of 10 (or more) websites have vulnerabilities as a result of being built by those who didn’t know or appreciate the severity of today’s attacks.” There is no arguing that many organizations are sitting on a huge legacy of insecure code, much of which may have been written before the discovery of prevalent vulnerabilities such as XSS, SQL Injection, CSRF, etc. Even worse, organization often have their security groups focused on network or system security, leaving application level security to developers. Unfortunately, these developers are receiving little or no training, while remaining under pressure to produce code under short deadlines.

Andre Gironda series, starting with “Week of War on WAF’s: Day 1 — Top ten reasons to wait on WAF’s,” provides important reasons why WAFs should not be viewed as a silver bullet solution. Rich Mogull in his post “Web Application Security: We Need Web Application Firewalls To Work. Better” makes the important point:

With old school vulnerabilities we know the details of the specific vulnerability and (usually) exploit mechanism. With WAFs, we are trying to block vulnerability classes instead of specific vulnerabilities. This is a HUGE difference. The WAF doesn’t know the details of the application or any application-specific vulnerabilities, and thus is much more limited in what it can block.

Mogull goes on to state that WAFs can:

no longer be merely external boxes protecting against generic vulnerabilities; they need tighter integration into our applications. In the long term, I’ve branded this Application and Database Monitoring and Protection (ADMP) as we create a dedicated application and database security stack that links from the browser on the front end, to the DB server on the back.

ADMPs, or if you prefer WAFs + Database Activity Monitoring (WAFs+DAM), would be another step in the evolution of WAFs. As Ivan Ristic, creator of ModSecurity, points out in his blog post “What’s the Score of the Game?“:

I feel that one of areas where organizations are failing, with regards to web application security, is that there is a lack of communication between the following three groups: Development teams (who are running source code reviews), InfoSec teams (who are running vulnerability scans) and Operational Security teams (who are running web application firewalls). These three teams each have unique perspectives on the vulnerabilities of the webapps and they should share their data with each other.

Nicely stated. No one is arguing that writing secure code is not the answer. If organization began adapting secure systems development lifecycle (SDLC) models into their business operation, many security problems would go away. Building secure software will require changes in the current development culture, which will include people, processes, and technology. No small task.

Gunnar Peterson has a nice post, “WAF and XSG Risk and Effectiveness at 20,000 feet” where he discusses modeling of combination of risk and effectiveness to identify areas of focus. As Peterson points out in another post, “WAFs are not as static as network firewalls…Instead WAFs collaborate much more directly with development, which is another growth opportunity for security industry.”

This post is going to stay focused on WAFs. With it taking on average 130 days to fix sequel injection bugs, organizations need something they can implement today. WAFs have an important role to play in adding a layer of security and monitoring to a defense in depth security approach. WAFs will evolve. They are in the process of evolving now. Understanding the fundamental ideas and going through the implementation of an open source solution starts us on the path of better understanding of future technologies.

An Implementation Using ModSecurity

Building on previous posts concerning “An Apache Implementation“, “PHP Implementation“, and “Apache and OpenSSL“, we have an Apache web server setup to build upon. For additional details, please get Ivan Ristic’s book, “Apache Security.” It really is a must have book for anyone serious about running an Apache web server. Ristic also maintains the ModSecurity website and blog, which serves as a great source for up-to-date information on ModSecurity.

The Apache module mod_unique_id needs to be installed for ModSecurity to work properly. This module was not installed when we configured Apache. At that time, we did not know we needed it. While it can be somewhat inconvenient, for security reasons it is best not to install modules not needed.

1. Stop Apache Server.

# /usr/local/apache/bin/apachectl stop

2. Install mod_unique_id Module.

For non Mac OS X, do the following:

root# cd /usr/local/src/httpd-2.2.8
/usr/local/src/httpd-2.2.8 root# make clean
/usr/local/src/httpd-2.2.8 root# CC=gcc \
CFLAGS="-O3 -fno-omit-frame-pointer" \
./configure --prefix=/usr/local/apache \
--enable-rewrite \
--enable-so \
--disable-imap \
--disable-userdir --with-mpm=worker --enable-ssl --enable-unique-id --enable-unique-id
/usr/local/src/httpd-2.2.8 root# make
/usr/local/src/httpd-2.2.8 root# make install

For Mac OS X, please do:

root# cd /usr/local/src/httpd-2.2.8
/usr/local/src/httpd-2.2.8 root# make clean
/usr/local/src/httpd-2.2.8 root# CC=gcc \
CFLAGS="-O3 -fno-omit-frame-pointer" \
LDFLAGS="-L/opt/local/lib -L/usr/lib" \
./configure --prefix=/usr/local/apache --enable-rewrite \
--enable-so --disable-imap --disable-userdir \
--with-mpm=worker --enable-ssl --enable-unique-id
/usr/local/src/httpd-2.2.8 root# make
/usr/local/src/httpd-2.2.8 root# make install

3. Install PCRE.

Only under Mac OS X did I have to install Perl Compatible Regular Expressions (PCRE). You may be able to skip this step, depending on your OS.

root# cd /usr/local/src
/usr/local/src root# wget http://downloads.sourceforge.net/pcre/pcre-7.7.tar.gz
/usr/local/src root# tar xzf pcre-7.7.tar.gz
/usr/local/src root# cd pcre-7.7
/usr/local/src/pcre-7.7 root# CC=gcc \
CFLAGS="-O3 -fno-omit-frame-pointer" \
LDFLAGS="-L/opt/local/lib -L/usr/lib" \
./configure
/usr/local/src/pcre-7.7 root# make
/usr/local/src/pcre-7.7 root# make test
/usr/local/src/pcre-7.7 root# make install

4. Install the latest version of libxml2 or Lua.

To quote wikipedia, libxml is “a library for parsing XML documents” and Lua is “a lightweight, reflective, imperative and procedural programming language, designed as a scripting language with extensible semantics as a primary goal.” ModSecurity requires dynamic libraries which are not built by default in the source distribution. Binary distribution is recommended.

I will go through configuration and installation of libxml2 from source and the binary installation of lua under Mac OS X. There is a good chance if you are running a different OS, the libraries will have already been installed.

root# cd  /usr/local/src/
/usr/local/src root# wget ftp://xmlsoft.org/libxml2/libxml2-2.6.32.tar.gz
/usr/local/src root# tar xzf libxml2-2.6.32.tar.gz
/usr/local/src root# cd libxml2-2.6.32
/usr/local/src/cd libxml2-2.6.32 root#  CC=gcc \
CFLAGS="-O3 -fno-omit-frame-pointer" \
LDFLAGS="-L/opt/local/lib -L/usr/lib" \
/usr/local/src/cd libxml2-2.6.32 root# ./configure
/usr/local/src/cd libxml2-2.6.32 root# make
/usr/local/src/cd libxml2-2.6.32 root# make install
/usr/local/src/cd libxml2-2.6.32 root# cd ..
/usr/local/src root# wget http://luaforge.net/frs/download.php/3097/lua5_1_3_Darwin811x86_lib.tar.gz
/usr/local/src root# mkdir lua
/usr/local/src root# cd lua
/usr/local/src/lua root# tar xzf lua5_1_3_Darwin811x86_lib.tar.gz
/usr/local/src/lua root# cp liblua5.1.* /usr/local/lib
/usr/local/src/lua root# cp include/* /usr/local/include

5. Download, unpack, configure, and compile ModSecurity.

If you are interested in connecting a ModSecurity sensor to the central audit log repository, you will want to build the ModSecurity Log Collector below with the command “make mlogc”. Install instructions can be found under apache2/mlogc-src directory. That step will not be included below.

root# cd  /usr/local/src/
/usr/local/src root# wget http://www.modsecurity.org/download/modsecurity-apache_2.5.5.tar.gz
/usr/local/src root# tar xzf modsecurity-apache_2.5.5.tar.gz
/usr/local/src root# cd modsecurity-apache_2.5.5
/usr/local/src/modsecurity-apache_2.5.5 root# cd apache2

For non Mac OS X, configure with the command:

/usr/local/src/modsecurity-apache_2.5.5/apache2 root# ./configure \
--with-apxs=/usr/local/apache/bin/apxs \
--with-apr=/usr/local/apache/bin \
--with-apu=/usr/local/apache/bin

For Mac OS X, use the command:

/usr/local/src/modsecurity-apache_2.5.5/apache2 root# CC=gcc \
CFLAGS="-O3 -fno-omit-frame-pointer" \
LDFLAGS="-L/opt/local/lib -L/usr/lib" \
./configure --with-apxs=/usr/local/apache/bin/apxs

Continue to compile and install with the commands:

/usr/local/src/modsecurity-apache_2.5.5/apache2 root# make
/usr/local/src/modsecurity-apache_2.5.5/apache2 root# make test
/usr/local/src/modsecurity-apache_2.5.5/apache2 root# make install
/usr/local/src/modsecurity-apache_2.5.5/apache2 root# ls -la /usr/local/apache/modules

6. Configure Apache and ModSecurity.

We must now edit the httpd.conf file in order to load libxml2 or lua5.1 modules before the ModSecurity module.

/usr/local/src/modsecurity-apache_2.5.5/apache2 root# vi /usr/local/apache/conf/httpd.conf

Add the lines for non Mac OS X:

#
LoadFile /usr/lib/libxml2.so
LoadFile /usr/lib/liblua5.1.so
#
LoadModule security2_module modules/mod_security2.so

For Mac OS X, add the lines:

#
LoadFile /usr/local/lib/libxml2.2.dylib
LoadFile /usr/local/lib/liblua5.1.so
#
LoadModule security2_module modules/mod_security2.so

Create the ModSecurity configuration file. There is a file modsecurity.conf-minimal present in the /usr/local/src/modsecurity-apache_2.5.5 that can be used. There is also a a Core Rule Set that was included in the /usr/local/src/modsecurity-apache_2.5.5/rules directory courtesy of Breach Security Inc. To quote the README file, “The Core Rule Set provides generic protection from unknown vulnerabilities often found in web application that are in most cases custom coded. The Core Rule Set is heavily commented to allow it to be used as a step-by-step deployment guide for ModSecurity.” Under the rules subdirectory, there a directory “optional” which contains additional possible rules. It is left to the reader which configuration files they may want to include, though it might be wise to start with the minimal and make sure the Apache runs without problems. Then add configurations files as desired.

/usr/local/src/modsecurity-apache_2.5.5/apache2 root# cd ..
/usr/local/src/modsecurity-apache_2.5.5 root# cp  modsecurity.conf-minimal /usr/local/apache/conf/modsecurity.conf
/usr/local/src/modsecurity-apache_2.5.5 root# cp  rules/*.conf /usr/local/apache/conf/

Include the modsecurity.conf, and additional ModSecurity configurations file, in the Apache httpd.conf file.

/usr/local/src/modsecurity-apache_2.5.5 root# vi /usr/local/apache/conf/httpd.conf

Add the line:

Include /usr/local/apache/conf/modsecurity.conf
#Include /usr/local/apache/conf/modsecurity_crs_10_config.conf
#Include /usr/local/apache/conf/modsecurity_crs_21_protocol_anomalies.conf
#Include /usr/local/apache/conf/modsecurity_crs_23_request_limits.conf
#Include /usr/local/apache/conf/modsecurity_crs_30_http_policy.conf
#Include /usr/local/apache/conf/modsecurity_crs_35_bad_robots.conf
#Include /usr/local/apache/conf/modsecurity_crs_40_generic_attacks.conf
#Include /usr/local/apache/conf/modsecurity_crs_45_trojans.conf
#Include /usr/local/apache/conf/modsecurity_crs_50_outbound.conf

Edit /usr/local/apache/conf/modsecurity.conf. The modifications will be very dependent on your environment. See resources listed in the Additional Information section to help with configuration. The default configuration saves the log files relative to the configuration file directory. Change this to where the apache logs are currently being saved.

/usr/local/src/modsecurity-apache_2.5.5 root# vi /usr/local/apache/conf/modsecurity.conf

Change the values to:

SecAuditLog /var/www/logs/modsec_audit.log
SecDebugLog /var/www/logs/modsec_debug.log

Let’s create null files with the correct permissions for Apache.

/usr/local/src/modsecurity-apache_2.5.5 root# touch  /var/www/logs/modsec_audit.log
/usr/local/src/modsecurity-apache_2.5.5 root# chown httpd.httpd /var/www/logs/modsec_audit.log
/usr/local/src/modsecurity-apache_2.5.5 root# touch  /var/www/logs/modsec_debug.log
/usr/local/src/modsecurity-apache_2.5.5 root# chown httpd.httpd /var/www/logs/modsec_debug.log

7. Start Apache.

Check that the configuration file is correct and start up Apache.

/usr/local/src/modsecurity-apache_2.5.5 root# /usr/local/apache/bin/apachectl configtest
Syntax OK
/usr/local/src/modsecurity-apache_2.5.5 root# /usr/local/apache/bin/apachectl start

Check if ModSecurity if configured into running Apache server.

/usr/local/src/modsecurity-apache_2.5.5 root# cat /var/www/logs/error_log | grep ModSecurity
[Thu Jul 31 18:24:59 2008] [notice] ModSecurity for Apache/2.5.5 (http://www.modsecurity.org/) configured.

Additional Information

This post is only to get the basics down. The above information was taken from the ModSecurity documentation install section for version 2.5.5. A great deal more information is available at the ModSecurity blog site and in the book “Apache Security“.

Concluding Remarks

Ivan Ristic and Ofer Shezaf are working on an interesting paper, “Enough With Default Allow in Web Applications!” This paper demonstrates how WAFs are evolving. To quote the paper:

The default allow deployment model, which is commonly used to implement and deploy web applications, is the cause of numerous security problems. We propose a method of modeling web applications in a platform-agnostic way to adopt a default deny model instead, removing several classes of vulnerability altogether and significantly reducing the attack surface of many others. Our approach is best adopted during development, but can be nearly as efficient as an afterthought, or when used at deployment time. What they are looking to do is create a protection layer between the web servers and applications which would increase security and turn applications into verifiable components with external contracts that can be enforced.

Ristic mentions in his post the planned release of “an open source profiling tool (which I will announce next week) to help with the third use case and automate the creation of positive security models (also known as the learning feature of web application firewalls).” Breach Security has also teamed up with WhiteHat Security to add the ability to their Sentinel scanning service to automatically create custom ModSecurity rules for certain classes of vulnerabilities that are found in your web applications. This is the kind of evolution that is required in security and makes ModSecurity such an interesting software package.

Open Source and Service-Orientated Architecture

John Gerber — Mon, 29 Jan 2007 05:48:36 +0000

“These blind men, every one honest in his contentions and certain of having the truth, formed schools and sects and factions…” — Buddha

“The Blind Men and the Elephant” is a classic fable. As the world of IT becomes more complex, the people in IT become more specialized. People become so focused in their areas, they lose the ability to see the big picture. Life becomes the fable. The fable is about a group of blind men who came upon an elephant. The first man, feeling the enormous leg, said, ‘This thing is very much like a tree.’ The second, standing near its ear, reached up and said, ‘This is a winnowing fan!’ ‘No,’ said a third as he grasped the moving trunk. ‘Be careful. This creature is a serpent.’ ‘I disagree,’ said a voice at the other end. ‘It is only a frayed piece of rope’. The last man commented, ‘You are all wrong. I have felt this thing on both sides and it is just a wall.’”

Let’s talk about two parts of the elephant, open source software (OSS) and service-orientated architecture (SOA). John Grimes, assistant secretary of Defense for networks and information integration/chief information officer, told the Network Centric Warfare conference in Washington, D.C., on Jan. 23, “As we go to SOA architecture, we keep the applications behind and share the data on the network, and it becomes very critical that data is understood by everyone.”

“It just eats our lunch every time we get into a proprietary situation, because it’s noncompetitive,” Grimes stated explaining that DOD will increasingly move to SOA because it benefits information sharing and acquisitions. Federal Computing Weekly has an article, “DOD’s Grimes: Our focus is on data” by Josh Rogin. In another article in FCW, Bob Brewinn wrote an article, “DISA Buying into SOA ‘Big Time’.” John Grimes is quoted as stating, “DOD spends too much time and money acquiring individual, highly-tailored systems.” He continues, stating, “It’s time for the department to stop buying things and start buying services.”

In an article in the Linux Insider title “Iona Tightens Open Source, SOA Bond,” Dana Gardner writes:

Open source and SOA are increasingly joined at the hip. These twins are developing in tandem, not sequentially, which is giving CIOs and architects a variety of choices for picking and choosing the projects and products that make up their SOAs.

Darryl K. Taft writes an article for eWeek title, “Web Services, SOA, and Open Source Converge.” Hub Vandervoort, chief technology officer at Sonic Software Corp., Bedford, Mass., was on a panel of heavy hitters at the Web Services/SOA on Wall Street conference on Feb. 27. Vandervoort makes the point that, “SOA as a concept will challenge the whole concept of one throat to choke. SOA means federation and is built from federated components that are boundless.”

Han Zaunere, president of New York PHP, an organization for the Apache, MySQL and PHP community in New York, stated, “”In the long run, as far as looking at what you get, I think open source is more valuable.” He points out, “If I download [licensed] software and in two years it’s obsolete, I have no return on that. When you buy open-source support, the software is secondary.” Hiram Chirino, co-founder and director of architecture at LogicBlaze Inc., of Marina del Ray, Calif., believes that open-source software allows users to scale their systems more easily and cheaply because they can simply add more servers without having to worry about licensing costs.

Bob Sutor has written, and podcasted, extensively on the topic of open source. His four part series, is very interesting. I would recommend folks take a few moment to read it, paying particular attention to part 4, “The SOA Connection.” He covers:

Analyst at Forrester have written a report, “The Future of Enterprise Software.” Andy McCue has written the article “Open Source and SOA to Redefine Software Landscape” where he summarizes the report. The report stated: “Too many IT pros today reject the new ideas behind the four horsemen as ‘not ready for prime time’. Blanket dismissals of new ideas are defensive; IT executives should be looking instead for ways that the four horsemen can drive productive changes for business. These forces will define the future of enterprise software.” The “four horsemen” of commoditization are service oriented architecture (SOA), open source, software as a service and offshoring. Forrester predicts the four horsemen will lead to cheaper prices and a radical change in enterprise software landscape of the future.

SOA is ultimately about integration. It can be integration of open or closed source software packages. SOA brings agility to an enterprise. CEO and CIO are beginning to question the wisdom of getting locked into a software solution. When you take a solution, such as SAP, it is a solution that matches a complex problem with a complex solution. Annrai O’Tool, CEO of Cape Clear, tells the following story on Dana Gardner’s SOA trends webinar:

We have a couple of ex-PeopleSoft people working with us at Cape Clear, and they tell a great story about how they used to do sales pitches against SAP. They went to the customer with a small cup of quick-drying cement and poured it into a mold. By the time they finished the presentation, the cement is set and it has SAP written on it. They then say, “There you go, that’s the deal with SAP.” It is easy to design, but once you get your business process done, it is embedded in cement.

O’Tool points out that for many businesses, the business process is very difficult to change. He goes on to say, “SOA is all about how to use that application in new and more transparent ways that are easier to change and that deliver agility.”

Commercial solutions can have high up cost and high continuous operation costs. Even changing out hardware can end up costing a business significantly due to software licenses. In this rapidly changing IT environment, a company can easily find itself with a solution that no longer fits its business need, but it has too much invested to change course. Even worse, in time their software solution might no longer be actively developed as software companies change directions or go out of business. OSS helps reduce some of these risks. While there are costs, more of the expenditures goes into the companies own people. This helps create a work force better adapted to face a changing IT environment. Hub Vandervoort said it best when he stated “when you buy a software license you are paying for past innovation; when you buy open source, you’re investing in future innovation.”

The Benefits of Open Source

John Gerber — Sun, 28 Jan 2007 19:32:18 +0000

“Knowledge is of two kinds. We know a subject ourselves, or we know where we can find information upon it.” — Samuel Johnson

When I was in college, some fundamental preachers came to campus to let us all know how terrible we were. They even preached against the Mennonite women whose hemlines fell above their ankles. One of the campus Christian groups were so upset with themselves because they froze up when trying to argue via an exchange of Bible verses. The statements the preachers were coming out with were so over the top, the only thing that tends to come to one’s mind is “Are they for real?” You can’t really get past that, rendering you a stammering idiot. In the end, it did not matter. It was not about an exchange of ideas.

I run into some folks who will preach against open source. A classic sound bite argument is, “Is freeware really free?” I remember working for government when government was very anti open source. Nowadays, attitudes have changed. Still, I come across some die hards from the old days. When I encounter such a person, I end up thinking about the news stories of those time capsules recently opened. Were these people somehow trapped in one of those time capsules? I am not an open source zealot. You use the right tool for the right job. I would argue one needs to be aware of the tools at their disposal. In the rapidly changing IT world, OSS can be a valuable tool adding agility to an enterprise solution toolbox. There are so many reports on the business impact of OSS, that one could write several books on the topic. I will spare you that, but still I wanted to post a few links. It is my way of countering the closed source preacher with what is equivalent to quoting open source Bible verses.

A good place to start is discussing the advantages, risk, culture change, and strategies involved with adapting OSS. Ibrahim Haddad has written a Enterprise OpenSource Magazine article titled, “Adapting an Open Source Approach to Software Development, Distribution, and Licensing.” He does a good job of discussing the unique characteristics, development model, the advantages, the risks, cultural changes, and strategies involved with developing and deploying OSS. There is more to open source than just not paying for the software

For a fun 48 page presentation, head over to the Free Electrons site. They have posted the presentation “Advantages of Free Software and Open Source in Embedded Systems.” A newly released European Commission study on the impact of Free, Libre, and Open Source Software (FLOSS) on the European IT sector is summarized by Thomas Claburn of InformationWeek in the article, “Study Finds Open Source Benefits Business.” The Europeans have been major developers of open source projects. Over on the English GBdirect site they have put together an area discussing the benefits of using open source software.

How does OSS fit into American markets? Darryl K. Taft did an article for eWeek on “CIOs: Open-Source Software Offers Cost, Quality Benefits.” Major American companies are also posting information on the benefits of open source. Red Hat, Inc. has posted, “Understanding the Financial Benefits of Open Source.” Sun has posted, “Open Source Is About Participation.” Of course, the US government has been looking into using OSS. Joab Jackson writing for GCN did a report, “Report Advocates Open-Source Approach for Software Acquisition.” It discusses the Office of the Deputy Undersecretary of Defense for Advanced Systems and Concepts commissioning the Open Technology Development road map. That is a fascinating report, which I highly recommend everyone interested in this topic read. A summary of the report can be found in Computer Business article title, “Open Source in the National Interest.”

Additional information on how OSS is being used within DoD can be read in the GCN article, “Open-Source Software Gets Nod from DoD” by Patricia Daukantas. Joris Evers writes for CNET how “Homeland Security Helps Secure Open-Source Code.”

Those articles and reports are just a sampling. They should help folks see the benefits of OSS and danger when implemented in the wrong environments. There are other benefits in terms of the agility open source brings to an organization. Plus, there are ties in to other areas of IT that are gaining major attention in 2007. I’ll write more on those later.

Identity Management

John Gerber — Mon, 18 Dec 2006 06:51:24 +0000

As we increase our use of services being developed as part of Web 2.0, we find more services linked together and more integrated applications. A year ago, Dick Hardt, CEO of Sxip Identity sat down with IT conversations to talk about “Identity 2.0: Identity Protocols, Today and Tomorrow.” ZDNet posted an article back in May, “The Many Players at IIW.” IIW is the Internet Identity Workshop. Of course, I have to point out Phil Windley’s blog Technometria. Phil is the author of the O’Reilly book, “Digital Identity,” a man who knows what he is talking about.

Dick Hardt stated, “The identity management industry needs a common approach to secure, role-based access and compliance reporting for the enterprise and open source projects like Bandit from Novell and Higgins are a great step in that direction. We see this as a natural compliment to the user-centric Identity 2.0 efforts being made with SXIP and DIX and are excited to work with them on adding support of Bandit, Higgins and eDirectoryTM.”

Now talking with a friend of mine, Vincent Tillman, he pointed out that the problem is that at some level projects must interoperate in a large enterprise. The Security Assertion Markup Language (SAML) (for real-time management) and Liberty (federation/trust between systems) seem to always be mentioned in the “going-to-be” supported category. Another, the Service Provisioning Markup Language (SPML), allows resources (e.g. , Oracle Db) and managers (e.g., Tivoli or Sun IdM) to create and manage accounts by calling standard (web) services. SPML and SAML both are Web Services initiatives for standard account and access management. I’ll just mention that the OASIS general membership voted to accept SPML v2.0 as an OASIS Standard. SAML v2.0 was accepted by OASIS back in March 2005.

I am going to quote from IT Conversations, “Hardt differentiates Identity 1.0 from Identity 2.0 by describing the move from a directory centric environment where authentication means simply that your identity is registered on a web site’s directory to a user centric environment where an identity can truly be applied to a variety of web sites. He believes this will happen because the recent history of technological initiatives shows that open and simple wins out.”

To add this idea, I’ll quote redmonk, ” Identity 2.0 systems are interested in using the concept of a user’s identity as a declarative bundle of claims about the user: from things like their name, address, to less traditional things like their desires, customer service history, and other attributes that are usually not so much associated with a user identity. That’s the first big leap of Identity 2.0 think: a user’s attributes should be associated with that user’s identity..”

For an example of Identity 2.0, and keeping it in the open source area, take a look at OpenID. To quote the site, “OpenID starts with the concept that anyone can identify themselves on the Internet the same way websites do-with a URI (also called a URL or web address). Since URIs are at the very core of Web architecture, they provide a solid foundation for user-centric identity.” It is a decentralized digital identity system, in which any user’s online identity is given by URL (such as for a blog or a home page) or an XRI in the latest version, and can be verified by any server running the protocol.

Wikipedia adds, “On OpenID-enabled sites, Internet users don’t need to create and manage a new account for every site before being granted access. Instead, they only need to be able to authenticate with a trusted site that supports OpenID, called the identity provider (or IdP, sometimes called an i-broker). The identity provider can then confirm ownership of the user’s OpenID identifier to other OpenID-enabled sites, called relying parties or RPs. Unlike most single sign-on architectures, OpenID does not specify the authentication mechanism. Therefore, the strength of an OpenID login depends on how much a relying party knows about the authentication policies of the identity provider. Without such knowledge, OpenID is not meant to be used on sensitive accounts (banking, e-commerce transactions, etc.), but if an identity provider uses strong authentication, OpenID can be used for all types of transactions..”

Sounds like an interesting idea. The need to maintain duplicate user data within an organization is a problem. It gets worse as services are moved outside an organization. This is the advantage of a distributed authentication system like OpenID. On the backend, you can authenticate from any data source, including but not limited to LDAP. Even the inclusion of a user information from legacy systems is possible. When Access Control Lists become a reality, it will be possible to eliminate any user data from ever being stored on any site other than the central source accessed by the OpenID server.

After the Digital ID World conference John Fontana, Senior Editor, Infrastructure for Network World magazine, wrote in his article Higgins lays out roadmap for open source identity project that

The Higgins group plans to release a middleware piece called the Identity Attribute Service that acts as a layer on top of identity repositories such as directories or applications. It can aggregate data from multiple sources in real-time and bundle them into a single identity credential. The idea is to link to data without having to move it around the network.

Mark Wahl writes, “There are several ways of looking at these APIs. One is that they are conceptually similar to APIs such as Active Directory Service Interfaces (ADSI) or Java Naming and Directory Interface (JNDI), in that they provide an abstraction to enable an application to be independent of the API of a lower layer access protocol. In this view, Higgins would offer a higher level abstraction as well as a different set of supported protocols: OpenID, WS-Trust and LDAP instead of Novell Netware, NIS and LDAP.”

I have written more than intended on this topic. A very interesting area, which I will revisit later. For now, I just wanted to point out a few concepts and links.

Business Intelligence

John Gerber — Thu, 14 Dec 2006 20:36:34 +0000

I’ll update this post later. I just want to get a few ideas down. I was looking for something, which I don’t even recall what it was now, when I came across Oreon for Nagios. Oreon is a front end providing Nagios with a dashboard. This led me to the idea of open source dashboards, which led me to open source business intelligence systems. I came across Pentaho. While reading about the components of Pentaho, I came across ACEGI and the Eclipse framework. Eclipse has its own Business Intelligence and Reporting Tools (BIRT). If you look at BIRT, you will come across Callisto. Having tossed all those names (and links), let me provide a additional links with descriptions.

Why is this of interest to me? I like to think of myself as an open source integrator. That might just be because the places I work tend to try and cut costs in terms of software. Many organizations are preferring to put money into support, and cut software costs. Many of the open source companies following this business need have a model that I can understand and appreciate. Another interest of mine is the bridging of business and IT focus. My initial interest was in terms of security and auditing. Reliance on IT in business is growing. As IT takes more of the companies budget, the business is going to need to understand what return on investment they are getting. Enter my interest in business intelligence. Like I said, I am an integrator, whether that be in open source packages, or the integration of IT and business needs.
First, a post from me would not be complete without links to podcasts. From IT Conversations, Steve Lucas, VP for Business Objects, talks on Business Intelligence.

The power of building on open source is that it gives the organization agility. No longer does the vendor control innovation. Software components can be brought together, and replaced if they fail to keep up with the changing market. Let me take an example from an InfoWorld article. You can get Alfresco integrated with Asterisk (VoiceRD from Novacoast) and SugarCRM (CRM) today. Then you can extend this. Add some JasperSoft or Pentaho for Business Intelligence (perhaps reporting capabilities). Some DimDim for web conferencing. Some Zimbra or Scalix for email/collaboration. Want to scale this out on a grid, then there is 3Tera. The Open Source Group has An Open Source Strategy for the Open Group, where they outline, much better then I ever could, why an organization might want to utilize open source software.

Not satisfied? Well here are a OSBI links for more information:

OSBI Links

Open Source Solutions BI Wiki: The Open Source Solutions BI wiki will provide research, articles and draft chapters of our book, Open Source Business Intelligence, as they are developed. There are public and private sections of the wiki. For access to the private sections please contact us.
Open Source Solutions Blog: This blog provides timely information about open source solutions for business intelligence, collaboration, project managment & data analytics, and expose the process of three authors writing a book collaboratively. Posts range from news & status of OSBI projects, interviews with OSBI project teams and communities’ members, and describing OSS related events.
Open Source Business Conference: The Open Source Business Conference was held at San Francisco’s Moscone Center on February 14-15. OSBC East is to be held 2006 October 17-18 in Boston, MA.
OSBC Wiki: The Open Source Business Conference has a Wiki on SocialText that supplements the information on the website.
Business Intelligence for Business People: This lens by Tom Hudock is a good source of non-technical information about “Business Intelligence (BI), Performance Management, and Data Warehousing (DW)”

Links to OSS BI Suites

BEE Project: BEE is one of the first open source BI Suites, having been around since 2002. It provides ETL, ROLAP, reporting, integration with the R Project, is written in PERL, and primarily supports MySQL.
Bizgres: Bizgres is a distribution of PostgreSQL with specific modifications to increase performance and use as a data warehouse. In addition, the Bizgres project comes with the KETL ETL tool and JasperReports. The Bizgres project is supported by a consurtium of three companies, Greenplum, Kinetic Networks, and JasperSoft.
DASH Portal: MarvelIT’s Dash Portal solution is based on the Apache Jetspeed portal and their unique charting portlets. Using MARVELit DASH, dashboards can be built very easily using the power of SQL and XML.
Openi: Openi provides a web-driven interface to OLAP, relational, statistical and data mining sources giving BI integrators user interface, report definition and connector tools.
Pentaho: Pentaho has been getting a lot of attention since its launch and funding in 2005. This project has an impressive pedigree in its team leaders, and provides quite an array of capabilities: Reporting, Analysis, Dashboards, Data Mining and Workflow.
SpagoBI: SpagoBI is a BI platform drawing its components from the ObjectWeb consurtium. Tools include metadata management, ETL, Reporting, Analysis, and Dashboards.

Links to OSS BI Development Tools

Eclipse BIRT: Eclipse is the IDE for Java and J2EE, and BIRT is, basically, its reporting plug-in.
EFEU: EFEU is a programming environment to develop C-programs and libraries. It is often pointed to as facilitating the development of ETL and reporting software.
JpGraph: JpGraph is an OO Graph drawing library for PHP that is very useful for data visualization and presentation.
PostgreSQL MDDB: The linked article describes using EFEU with PostgreSQL to create a multi-dimensional database for use in OLAP.

If you these are not enough links, go to the Squidoo site where folks pass on recommendations and posted some of the above information.

Socialtext Information

John Gerber — Wed, 13 Dec 2006 16:32:15 +0000

If you are interested in comparing Wikis, a good place to start is at the WikiMatrix. In my opinion, Socialtext is the company to watch. Not so much because it would be easy to implement under Mac OS X. In fact, their FAQ states MySQL support is still being developed. While, I might agree with the folks at InfoSpaces when they posted their blog, Socialtext 2.0: not yet there, the fast pace development of the software gives me faith that they will get there. What interests me the most is the work Socialtext developers have been doing with the API interfaces and the involvement in the open source community. Check into the CEO, Ross Mayfield Weblog or company’s blog for recent developments. There is even an area maintained for customers to exchange information, call Customer Exchange.

At the O’Reilly Open Source Convention (OSCON), Socialtext released Socialtext Open. Socailtext distribution includes wikiCalc, which is the social spreadsheet, and Wikiwfg, and WYSIWYG editor. Very interesting products.

If you want to read more, TechCrunch has a posting by Marshall Kirkpatrick, “SocialText aims for wiki 2.0.” And of course, there are podcasts. Kirsten Jones talked to Perlcast about the REST API beta. The podcast is available along with reading material. Last month, CEO Ross Mayfield joined Phil Windley, Matt Asay, and Scott C. Lemon in a r eview of the technology news of the week. Before Wikimania 2006, InfoWorld’s Jon Udell had a podcast with Ross Mayfeild about Socialtext’s decision to release Socialtext Open and its relationship with WikiCalc. PodTech has Ross Mayfield talking with Andrew McAfee about Web 2.0 for Enterprise. Jennifer Jones talked with Ross about wikis and their use for companies both large and small in the podcast “Delving Into the Real Value of Wikis.” There is a video cast at the start of this month entitled, “Inside the first Wiki Company, SocialText.” PodTech has additional podcast/video cast available at their site.

It is an interesting company and leader in the Wiki world.

SOA and Open Source

John Gerber — Tue, 05 Dec 2006 14:30:18 +0000

A recently posted podcast on SOA and Open Source was done by Dana Gardner over on ZDnet. Note, this was sponsored by IONA Technologies:

IONA broadens open source approach to SOA with Celtrix Enterprise initiative

Let me quote the accompanying blog:

IONA Technologies on Monday introduced a broader Apache-based set of open source SOA initiatives with the introduction of Celtix Enterprise, which expands the ESB offering to include other open source technologies including Qpid, which implements the Advance Message Queuing Protocol (AMQP) specification, and ActiveMQ, the JMS 1.1-based messaging middleware that uses AMQP. Celtix Enterprise also includes Eclipse-based tooling supporting JBI, Tomcat, and Spring.

I like the idea. Open Source has become more and more popular as the business model on software has shifted. People now tap into the open source community to help develop their software and move towards offering services to help support the software. This is different than the traditional method of in house development with the hope to make money off the sales of software. Though, alot of company that developed software really made their money providing services and support, even when they were selling the software. It is an interesting model.

Introduce a new architecture based on services and it just seems that we are moving in the same circles. SOA is about the architecture. There is alot of cost involved and managers are looking to defray the cost wherever they can. Add to this an architecture which is all about not getting locked into a particular proprietary solution, but being able to communicate between different services. It is all about being agile. Changes are occurring so rapidly, you have to be able to adapt quickly. Investing in your people help create an organization that can adapt to whatever comes tomorrow. If a company only invests in proprietary solutions, the company one day will find itself to be similar to the Titanic; unable to change course quick enough to avoid the unforeseen perils that surely will come its way.

Listen to the podcast.

Of course, when it comes to SOA, we have to see what Dave Linthicum has to say about OSS SOAs:

Open Source ESBs Have Emerged

Truthfully, I don’t much about SOA, though I keep listening and reading. I do know to listen what Dave has to say, even if he doesn’t understand open source. So, keep an eye on Mule. The bottom line is that currently developers who are building SOAs are not hardcore geeks. Sorry, Dave. The less technical guys require modeling tools with features to work collaboratively in conjunction with non-technical people. Commercial companies deliver this. While there are open source solutions, there simply are not many people who can implement them. Companies, while wanting to reduce the cost of the SOA software, lack the personnel. This seems like prime ground for consulting companies. Traditional close source companies are going open source, sort of, and offering consulting and support towards this end. The open solutions are out there and are being developed ever day. It is an interesting area to keep an eye on.