Security Advancements at the Monastery » MySQL

Google Visualization: An Example Graphing NVD CVE Data

John Gerber — Fri, 16 Apr 2010 15:54:42 +0000

Google visualization offers graphing abilities to any number of projects. Why should security professionals care? If you are going to have to collect and present security metrics, it is best to showcase them in the very best manner possible. Andrew Jaquith in his article, “Creating meaningful information security metrics” states, “For 2010, Forrester Research expects that overall security budgets will rise less than 5 percent over 2009 –higher than in the previous year, but not by much.” Andrew goes on to point out, “smart security managers, sensing sudden vulnerability in their budgets, seek better ways to measure and prove the value of what they do every day.”

In today’s work environment there is a need to show changes, potential risks, improved performance, etc. in all areas of the company’s operations. Security professionals need to be prepared to answer the basic question, “why should the CIO or CEO care about security?” CSO Online has a great quote from the post, “From the CIO: Why You Didn’t Get the CISO Job” that challenges us to consider our views when it comes to security. The post states, “laser focus on your speciality is great in middle management. It’s what we want. One of the really hard things about jumping from management to executive is a focus on the whole of the business. It’s a rare person who manages it quickly or easily.” That is basically the problem with metrics. It is a battle between generalization to the point of uselessness and details to the point of not being understandable or collectible. At the end of the day, something needs to be done because the security industry is currently leaving upper management in the position of not understanding what is going on within their business. That is a risk that not acceptable.

Andrew’s article discusses what kind of security metrics should be used. Additional sources of information on security metrics can be found in a previous post entitled “Security Metrics.” The post provides links to wonderful sources on security metric information. You might also want to take a look at the CIS Consensus Security Metrics v1.0.0 guide, NIST Special Publication (SP) 800-55 Rev 1 “Security Metrics Guide for Information Technology Systems”, NIST IR-7564 “Directions in Security Metrics Research”, “Twenty Most Important Controls and Metrics for Effective Cyber Defense and Continuous FISMA Compliance,” and “Metrics, measures & Myths.” Once you have start gathering metrics, you will want to present them in an easy to understand format. This is where Google Visualization can help.

Today’s post walks through an example using the data from the National Institute of Standards and Technology (NIST) National Vulnerability Database (NVD ) Common Vulnerabilities and Exposures (CVE) database. The purpose is to provide a working example from which you can learn and apply to the various metrics gathered at your organization.

Data Source

A previous post, “Standardization and Interoperability in Security,” discussed how the Security Content Automation Protocol (SCAP) is an attempt to help defenders by providing a collection of XML schemas/standards that allow technical security information to be exchanged between tools. SCAP components consists of:

Common Configuration Enumeration (CCE): provide unique identifiers to system configuration issues in order to facilitate fast and accurate correlation of configuration data across multiple information sources and tools.
Common Platform Enumeration (CPE): a structured naming scheme for information technology systems, platforms, and packages.
Common Vulnerability Enumeration (CVE): a dictionary of publicly known information security vulnerabilities and exposures.
Common Vulnerability Scoring System (CVSS): a vulnerability scoring system designed to provide an open and standardized method of rating IT vulnerabilities. NIST has even provided a calculator for creating CVSS vulnerability severity scores.
eXtensible Checklist Configuration Description Format (XCCDF): a specification language for writing security checklists, benchmarks, and related kinds of documents. NIST has released the NIST Interagency Report 7275 Revision 3 “Specification for Extensible Configuration Checklist Description Format (XCCDF) Version 1.1.4.”
Open Vulnerability Assessment Language (OVAL): an information security community standard to promote open and publicly available security content, and to standardize the transfer of this information across security tools and services.

We are going to make use of the data from NVD/CVE XML feed with the Common Vulnerability Scoring System (CVSS) mappings (version 2.0). NIST documentation states:

CVSS provides an open framework for communicating the characteristics and impacts of IT vulnerabilities. Its quantitative model ensures repeatable accurate measurement while enabling users to see the underlying vulnerability characteristics that were used to generate the scores. Thus, CVSS is well suited as a standard measurement system for industries, organizations, and governments that need accurate and consistent vulnerability impact scores. Two common uses of CVSS are prioritization of vulnerability remediation activities and in calculating the severity of vulnerabilities discovered on one’s systems.

NVD provides CVSS ‘base scores‘ representing the innate characteristics of each vulnerability. ‘Temporal scores,’ which change over time due to events external to the vulnerability, are not provided though NVD does provide a CVSS score calculator. This allows an organization to add temporal data and even factor in ‘environmental scores‘ customized to reflect the impact of the vulnerability on the organization. Please refer to the CVSS standards guide and the OWASP Risk Rating Methodology concerning factors involved in estimating the severity of risks to your business.

NVD CVE XML Schema

For our example, we will be using the data feeds nvdcve-2.0-2010.xml and nvdcve-2.0-2009.xml. Examining the CVE XML 2.0 Schema, we are particularly interested in certain vulnerability and CVSS scoring information. For example, for CVE-2010-1228, we will parse and pull the following kind of information:

 id="CVE-2010-1228">
  CVE-2010-1228
  2010-04-01T18:30:00.453-04:00
  
  2010-04-05T00:00:00.000-04:00
  
  
    
      10.0
      NETWORK
      LOW
      NONE
      COMPLETE
      COMPLETE
      COMPLETE
      http://nvd.nist.gov

Using Perl to Retrieve the CVE File

Initially we will read the nvdcve-2.0-2010.xml and nvdcve-2.0-2009.xml files. If we start retrieving the file regularly, we would want to change this to nvdcve-2.0-recent.xml. Of course, previous years can also be read in to provide a longer perspective on vulnerabilities. A simple example of a Perl subroutine to read the NVD CVE file and save it locally would be:

sub readpage {
   my($url,$nvd_file) = @_;
   my($proxy) = "http://your-proxy-server:proxy-port";
   my $ua = new LWP::UserAgent;
   $ua->proxy(http  => $proxy);
   $ua->proxy(ftp => $proxy);
   $ua->proxy(https => $proxy);
   # Go out and retrieve page
   my $req = new HTTP::Request('GET', $url);
   my $res = $ua->request($req);
   my $pjstatus = 1;
   # Check if the requested webpage is there and return results
   if ($res->is_success) { # Request successful
       open(OUTFILE,">$nvd_file") || ($pjstatus = 0);
       if ($pjstatus) {
          print OUTFILE $res->content;
       }
       close(OUTFILE);
   }
   else {
      $pjstatus = 0;
   }
   return($pjstatus);
}

Please substitute “http://your-proxy-server:proxy-port” with your site’s proxy server and port, if applicable.

Creating a MYSQL Table to Hold the Data

There is a great deal of information in the NVD CVE file. You will need to determine what information your organization will be interested in storing and graphing. For better or worse, folks have come to expect vulnerabilities to have a “Low,” “Medium,” or “High” score. NIST has stated concerning the NVD Vulnerability Severity Ratings:

NVD provides severity rankings of “Low,” “Medium,” and “High” in addition to the numeric CVSS scores but these qualitative rankings are simply mapped from the numeric CVSS scores:
1. Vulnerabilities are labeled “Low” severity if they have a CVSS base score of 0.0-3.9.
2. Vulnerabilities will be labeled “Medium” severity if they have a base CVSS score of 4.0-6.9.
3. Vulnerabilities will be labeled “High” severity if they have a CVSS base score of 7.0-10.0.

While preferring quantitative over qualitative values, for this example I would like to create a stacked column chart. We will add a severity column which is based on the CVSS score. An example table follows:

CREATE DATABASE vulnerabilities;
USE vulnerabilities;
DROP TABLE IF EXISTS `nvdcve`;
CREATE TABLE `nvdcve` (
  `cve_id` varchar(13) NOT NULL,
  `published` datetime default NULL,
  `modified` datetime default NULL,
  `score` DECIMAL(5,2) default '0.0',
  `severity` varchar(6) default 'LOW',
  `vector` varchar(25) default NULL,
  `complexity` varchar(25) default NULL,
  `authentication` varchar(25) default NULL,
  `confidentiality` varchar(25) default 'NONE',
  `integrity` varchar(25) default 'NONE',
  `availability` varchar(25) default 'NONE',
  `summary` varchar(512) default NULL,
  PRIMARY KEY  (`cve_id`),
  INDEX (score),
  INDEX (vector)
)

Using Perl Populating the Database

Populating the database table is simply a matter of reading the file and adding the entries to the table. An example Perl subroutine follows:

sub readxml {
   my($nvd_file, $dbh) = @_;
   my $parser = XML::LibXML-> new();
   my $doc    = $parser-> parse_file($nvd_file);
   my $xc     = XML::LibXML::XPathContext-> new( $doc->documentElement() );
   $xc-> registerNs(
      def  => 'http://scap.nist.gov/schema/feed/vulnerability/2.0' );
   $xc-> registerNs(
     vuln => 'http://scap.nist.gov/schema/vulnerability/0.4' );
   $xc-> registerNs( cvss => 'http://scap.nist.gov/schema/cvss-v2/0.2' );
   for my $entry ($xc-> findnodes("/def:nvd/def:entry")) {
      my $cve = $xc-> find('vuln:cve-id',$entry);
      my $published = $xc-> find('vuln:published-datetime', $entry);
      my $modified = $xc-> find('vuln:last-modified-datetime', $entry);
      my $summary = $xc-> find('vuln:summary', $entry);
      my $skip = 0;
      my ($metrics) = $xc-> findnodes('vuln:cvss/cvss:base_metrics', $entry) or ($skip = 1);
      if (! $skip) {
         my $score = $xc-> find('cvss:score', $metrics);
         my $vector = $xc-> find('cvss:access-vector', $metrics);
         my $complexity = $xc-> find('cvss:access-complexity', $metrics);
         my $authentication = $xc-> find('cvss:authentication', $metrics);
         my $confidentiality =
            $xc-> find('cvss:confidentiality-impact', $metrics);
         my $integrity = $xc-> find('cvss:integrity-impact', $metrics);
         my $availability = $xc-> find('cvss:availability-impact', $metrics);
         my $severity = "LOW";
         if (int($score) >= 7) {
            $severity = "HIGH";
         }
         elsif (int($score) >= 4) {
            $severity = "MEDIUM";
         }
         my $sql = qq{ SELECT count(*) FROM nvdcve WHERE cve_id=? };
         my $sth = $dbh->prepare( $sql );
         my $rc = $sth->execute($cve);
         if ( $rc) {
            my($exist) = $sth->fetchrow_array();
            if (! $exist) {
                $sql = qq{ INSERT INTO nvdcve SET cve_id=?,
published=?, modified=?, score=?, severity=?, vector=?, complexity=?,
authentication=?, confidentiality=?, integrity=?,availability=?, summary=? };
               $sth = $dbh->prepare( $sql );
               $rc = $sth->execute($cve,$published,$modified,$score,
$severity,$vector,$complexity,$authentication,
$confidentiality,$integrity,$availability,$summary);
            }
         }
      }
   }
}

The Perl Program to Pull It All Together

The above subroutines use the Perl modules LWP::UserAgent, XML::LibXML, XML::LibXML::XPathContext, and DBI. A sample Perl program that calls the above subroutines to pull down the NVD CVE data and load it into a MySQL table would be:

#!/usr/local/bin/perl -w
use LWP::UserAgent;
use XML::LibXML;
use XML::LibXML::XPathContext;
use DBI;
BEGIN{push @INC, "/home/jgerber/projects/nvd/perl"}
use nvdsubs qw($db_host $db $mysql_user $mysql_passwd $mysql.sock
readpage readxml );
# Main
my $datadir = "/home/johngerber/projects/nvd/data";
my @timeData = localtime(time);
my $year = 1900 + $timeData[5];
my $prev_year = 1900 + $timeData[5] - 1;
my $url = "http://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-" .
    $year . ".xml";
my $prev_url = "http://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-" .
    $prev_year . ".xml";
my $nvd_file = $datadir  . "/nvdcve-". $year . ".xml";
my $prev_nvd_file = $datadir  . "/nvdcve-". $prev_year . ".xml";
$db = "vulnerabilities";
local($dbh) = DBI->connect("DBI:mysql:mysql_socket=$mysql.sock;$db:$db_host",
$mysql_user, $mysql_passwd) || die "ERROR: Connecting: $DBI::errstr\n";
my ($pjstatus) = &readpage($prev_url,$prev_nvd_file);
if ($pjstatus) {
   &readxml($prev_nvd_file,$dbh);
}
$pjstatus = &readpage($url,$nvd_file);
if ($pjstatus) {
   &readxml($nvd_file,$dbh);
}
exit;

The nvdsubs.pm file will not be included in this post. The subroutines are defined and the only pieces missing are the MySQL database username and password. You don’t need mine. Add your own. At this point, we have everything we need to finally use Google Visualization to create a graph.

Google Visualization

We are going to create a Perl program that will read our MySQL nvdcve table and generate the JavaScript that will render our charts on the client’s browser. First, we want to define the JavaScript we want to produce. Just to alleviate some concerns, with Google Visualization your data is only shared between your server and the client connecting. This is unlike Google Charts where your data is sent to Google where it is made into a chart and the result is sent back. Google states concerning the logging of chart data (via Google Charts), “The chart data included in the HTTP request is saved in temporary logs for no longer than two weeks for internal testing and debugging purposes.” Every example in the Google Visualization Gallery will state the data policy. For Google Charts, stated at the bottom of the page for each gadget description the data policy:

While Google Visualization gadgets will have the following stated data policy:

Loading Google Libraries

The first thing the JavaScript needs to do is load the required libraries. This is accomplished with the lines:

Area Chart and Table

In this example we are going to create an column chart. In a later section, “Other Charting Options” (see below) we define different Google Visualization charting options.

JavaScript code for a sample column chart would be:

    <script type='text/javascript'>
      google.load('visualization', '1', {packages:['columnchart']});
      google.setOnLoadCallback(drawChart);
      function drawChart() {
        var data = new google.visualization.DataTable();
        data.addColumn('date', 'Date');
        data.addColumn('number', 'High');
        data.addColumn('number', 'Medium');
        data.addColumn('number', 'Low');
        data.addRows([
           [new Date(2009, 0, 30),92,97,3],
           [new Date(2009, 1, 27),168,142,25],
           [new Date(2009, 2, 31),141,165,9],
           [new Date(2009, 3, 30),132,203,12],
           [new Date(2009, 4, 29),158,153,8],
           [new Date(2009, 5, 30),200,199,22],
           [new Date(2009, 6, 31),190,195,11],
           [new Date(2009, 7, 31),127,139,14],
           [new Date(2009, 8, 30),233,208,14],
           [new Date(2009, 9, 30),163,167,18],
           [new Date(2009, 10, 30),129,172,8],
           [new Date(2009, 11, 31),200,211,19],
           [new Date(2010, 0, 29),157,139,14],
           [new Date(2010, 1, 26),137,143,12],
           [new Date(2010, 2, 31),252,242,18],
           [new Date(2010, 3, 13),92,118,17]
        ]);
        var chart = new google.visualization.ColumnChart(document.getElementById('s4graph'));
        chart.draw(data, {displayAnnotations:true, is3D: true, isStacked: true, min: 0,
          allowHtml: true, colors:[{color:'#E41B17', darker:'#C11B17'}, {color:'#FFA500', darker:'#E56717'}, {color:'#FFE87C', darker:'#C8B560'}]});
      }
    script>

The resulting image would be the following column chart:

Rendering the Table

When providing qualitative results, I like to back them up with more accurate numeric values. Let us include a table with links to the CVSS scores for each vulnerability.

    <script type='text/javascript'>
      google.load('visualization', '1', {packages:['table']});
      google.setOnLoadCallback(drawChart);
      function drawChart() {
        var data2 = new google.visualization.DataTable();
        data2.addColumn('date', 'Date');
        data2.addColumn('number', 'High');
        data2.addColumn('number', 'Medium');
        data2.addColumn('number', 'Low');
        data2.addRows([
           [{v:new Date(2009, 0, 30),
              f:'2009-01-30'}, 92,97,3],
           [{v:new Date(2009, 1, 27),
              f:'2009-02-27'}, 168,142,25],
           [{v:new Date(2009, 2, 31),
              f:'2009-03-31'}, 141,165,9],
           [{v:new Date(2009, 3, 30),
              f:'2009-04-30'}, 132,203,12],
           [{v:new Date(2009, 4, 29),
              f:'2009-05-29'}, 158,153,8],
           [{v:new Date(2009, 5, 30),
              f:'2009-06-30'}, 200,199,22],
           [{v:new Date(2009, 6, 31),
              f:'2009-07-31'}, 190,195,11],
           [{v:new Date(2009, 7, 31),
              f:'2009-08-31'}, 127,139,14],
           [{v:new Date(2009, 8, 30),
              f:'2009-09-30'}, 233,208,14],
           [{v:new Date(2009, 9, 30),
              f:'2009-10-30'}, 163,167,18],
           [{v:new Date(2009, 10, 30),
              f:'2009-11-30'}, 129,172,8],
           [{v:new Date(2009, 11, 31),
              f:'2009-12-31'}, 200,211,19],
           [{v:new Date(2010, 0, 29),
              f:'2010-01-29'}, 157,139,14],
           [{v:new Date(2010, 1, 26),
              f:'2010-02-26'}, 137,143,12],
           [{v:new Date(2010, 2, 31),
              f:'2010-03-31'}, 252,242,18],
           [{v:new Date(2010, 3, 13),
              f:'2010-04-13'}, 92,118,17],
        ]);
        var table = new google.visualization.Table(document.getElementById('s4graph_tab'));
        table.draw(data2, {showRowNumber: true, sortAscending: false, sortColumn: 0, allowHtml: true});
      }
    script>

The JavaScript code assumes there is a PHP program called cvealerts.php under the /nvd directory on your web server. Adjust to your environment. A sample PHP program that could be used for cvealerts.php is provided below. The resulting table chart would look like:

Handling Events: Interactions Between Graphs

We now have two different types of graphs representing the same data. We want to add interaction between the graphs so the viewer can see the relationship. With tables rows are selected when the user clicks, which correspond to the whole column of the stacked column chart. It is not a perfect fit, but it does demonstrate nicely use of adding interactions.

        // Set a 'select' event listener for the table.
        // When the table is selected,
        // we set the selection on the line graph.
        google.visualization.events.addListener(table, 'select', function() {
          chart.setSelection([{row: table.getSelection()[0].row, column: 1}]);
         });
        // Set a 'select' event listener for the graph.
        // When the graph is selected,
        // we set the selection on the table.
        google.visualization.events.addListener(chart, 'select', function() {
           table.setSelection([{row: chart.getSelection()[0].row}]);
        });

Providing Detailed Information

When the table chart link is clicked, we would like to provide some detailed information about the vulnerability. For this example, we will do this with a simple PHP program placed in the /nvd directory on the web server. The program is called cvealerts.php.


session_start();
function db_connect($table) {
   $result = mysql_pconnect(":", "", "");
   if (!$result) return false;
   if (!mysql_select_db($table)) return false;
   return $result;
}
function do_html_header($title,$checkuser,$logpage) {
?>
  <html> <head> <title>$title?>title>head>
  <body bgcolor="#FFFFFF">

}
function do_html_footer() {
?>
<table>
<tr><td ALIGN=CENTER NOWRAP WIDTH="590">font>
<font face="Verdana, Arial, Helvetica" size=-2>Notice to Users: Use
of this system constitutes consent to security monitoring and testing.
<br>All activity is logged with your host name and IP address.font>
td>tr>
table>
body>
 html>

}
// Main
$dates= array();
$stringlist = "";
if (isset($_GET['date'])) {
    $passdates = explode(",",$_GET['date']);
    for ($index=0; $index<count($passdates); $index++) {
       array_push($dates, $passdates[$index]);
       $stringlist .= $passdates[$index] . " ";
    }
}
else {
  print("Confusion over how you arrived at this page.\n");
  exit;
}
$stringlist = preg_replace("/ $/", "",$stringlist);
do_html_header("Review NVD CVE Announcements for Month Ending $stringlist",1,1);
$nvd_host = "http://web.nvd.nist.gov/view/vuln/detail?vulnId=";
$conn = db_connect("vulnerabilities");
if (!$conn)
   logit("Could not connect to database vulnerabilities - please try later.\n",1);
for ($index=0; $index<count($dates); $index++) {
   $rule = $dates[$index];
   $sql = "SELECT cve_id,score,published,vector,severity,complexity,left(summary,50)
    FROM vulnerabilities.nvdcve
      WHERE date_format(published,'%Y-%m')='$rule'
       ORDER BY (score+0)";
   $result = mysql_query($sql,$conn);
   if (!$result)
       logit("Problem with $sql\n",1);
   print("
\n");for($count=1;list($cve_id,$score,$date,$vector,$severity,$complexity,$shortsum)=mysql_fetch_array($result, MYSQL_NUM);++$count){?><tr><td CLASS="plfieldhdrleft" WIDTH="20%" BGCOLOR='#F0F5FF'>  print("$cve_id"); ?>
      td>
      <td CLASS="plfieldhdrleft" BGCOLOR='#F9FCFF'>
        print($score); ?>
      td>
      <td CLASS="plfieldhdrleft" BGCOLOR='#F0F5FF'>
        print($date); ?>
      td>
      <td CLASS="plfieldhdrleft" BGCOLOR='#F9FCFF'>
        print($vector); ?>
      td>
      <td CLASS="plfieldhdrleft" BGCOLOR='#F0F5FF'>
        print($severity); ?>
      td>
      <td CLASS="plfieldhdrleft" BGCOLOR='#F9FCFF'>
        print($complexity); ?>
      td>
      <td CLASS="plfieldhdrleft" BGCOLOR='#F0F5FF'>
        print($shortsum); ?>
      td>
      tr>

   }
}
print("
Bulletin 
Impact 
Date 
Vector 
Severity 
Complexity 
Short Summary
            
        

      
      
");
do_html_footer();

The PHP program would generate a HTML table displaying the NVD CVE alerts for that month. The table would look like:

When the CVE link is clicked on, the user is taken to the NIST NVD site where additional information is available.

Using Perl to Create the JavaScript

The Perl code is rather simple now that we have the MySQL tables defined and the JavaScript we want to generate. Much of the code consists of the JavaScript listed above.

#!/usr/local/bin/perl -w
use DBI;
use Time::Local;
use POSIX qw(strftime);
use LWP::UserAgent;
BEGIN{push @INC, "/home/jgerber/projects/nvd/perl"}
use ornl_feds qw($db_host $db $mysql_user $mysql_passwd );
sub slide_nvd_alerts {
  my($min_date,$graph_name,$web_link,$dbh) = @_;
  my $slide = "";
  my $slide_head = qq!
    
!;
   if ($min_date eq "") {
      my $sql2 = qq{ SELECT min(published) FROM vulnerabilities.nvdcve };
      my $sth2 = $dbh->prepare( $sql2 );
      my $rc2 = $sth2->execute();
      if ($rc2) {
         $min_date = $sth2->fetchrow_array();
      }
   }
   my $table_data = "";
   my $graph_data = "";
   my $sql2 = qq{ select date_format(published,'%Y-%m'),severity,count(severity)
      FROM vulnerabilities.nvdcve where published >= ? group by date_format(published,'%Y-%m'),severity };
   my $sth2 = $dbh->prepare( $sql2 );
   my $rc2 = $sth2->execute($min_date);
   if ($rc2) {
      my ($change,$virgin,$ht,$mt,$lt,$mmax_date) = ("",1,0,0,0,"");
      while (my($snapshot_date, $severity, $pcount) = $sth2->fetchrow_array()) {
         my $sql3 = qq{ SELECT max(published) FROM vulnerabilities.nvdcve where
date_format(published,'%Y-%m')=? };
         my $sth3 = $dbh->prepare( $sql3 );
         my $rc3 = $sth3->execute($snapshot_date);
         $max_date =  $sth3->fetchrow_array();
         $max_date =~ s/ \S+$//;
         if ($change ne $snapshot_date) {
            if (! $virgin) {
                my($year,$month,$day) = split("-",$mmax_date);
                my $mmonth = $month;
                $month--;
                $graph_data .= qq!           [new Date($year, $month, $day),$ht,$mt,$lt],
!;
                $table_data .= qq!           [{v:new Date($year, $month, $day),
              f:'$mmax_date'}, $ht,$mt,$lt],
!;
                ($ht,$mt,$lt) = (0,0,0);
             }
             $change = $snapshot_date;
          }
          if ($severity eq "HIGH") { $ht = $pcount; }
          elsif ($severity eq "MEDIUM") { $mt = $pcount; }
          elsif ($severity eq "LOW") { $lt = $pcount; }
          if ($mmax_date eq "") { $mmax_date = $max_date; }
          if ($mmax_date lt $max_date) { $mmax_date = $max_date; }
          $virgin = 0;
      }
      my($year,$month,$day) = split("-",$mmax_date);
      my $mmonth = $month;
      $month--;
      $graph_data .= qq!           [new Date($year, $month, $day),$ht,$mt,$lt]
!;
     $table_data .= qq!           [{v:new Date($year, $month, $day),
              f:'$mmax_date'}, $ht,$mt,$lt],
!;
   }
   $table_data .= "        ]);\n";
   $graph_data .= "        ]);\n";
   $slide = $slide_head .  $graph_data . $slide_head_table . $table_data . $slide_tail;
   return($slide);
}
sub slide_body {
  my($graph_name,$title,$style) = @_;
  my $table_name = $graph_name . "_tab";
  my $table_text = "div id=\"$table_name\"";
  if ($style ne "") {
     $table_text .= " style=\'$style\'";
  }
  my $slide2 = "$title
\n";
  my $itext = "div id=\"$graph_name\"";
  if ($style ne "") {
     $itext .= " style=\'$style\'";
  }
  $slide2 .= qq{
    
    <$itext>
    <$table_text>
       
    
    
  };
  return($slide2);
}
# Main
my $web_link = "/nvd";
my $results_dir = "/data/html" . $web_link;
my $result_file = $results_dir . "/nvdcve_stats.html";
my $debug = 1;
my $db = "vulnerabilities";
local($dbh) = DBI->connect("DBI:mysql:$db:$db_host", $mysql_user, $mysql_passwd) ||
   die "ERROR: Connecting: $DBI::errstr\n";
$slides_data .= &slide_body("s4graph","NVD CVE Alerts","width:700px; height:400px;");
$slides_head .= &slide_nvd_alerts("","s4graph",$web_link,$dbh);
open(OUTFILE,">$result_file");
print OUTFILE "\nNVD CVE Statistics\n";
print OUTFILE "\n";
print OUTFILE $slides_head;
print OUTFILE "\n\n";
print OUTFILE $slides_data;
print OUTFILE "\n";
close(OUTFILE);
exit;

Other Charting Options

Google, Google users, and other companies have shared some JavaScript visualizations built on the Google Visualization API to help you get started. Below are some example:

	Annotated Time Line (GWT Integrated) An animated time series chart. By: Google
	Area Chart (GWT Integrated) Interactive area chart. By: Google
	Bar Chart (GWT Integrated) Interactive bar chart. By: Google
	Bars of Stuff Fun bar charts using images of trains, chocolate, worms, and more. By: The visapi project
	Bio Heat Map Heatmaps are a useful way to visualize matricies of data. Scientists often use green-black-red heatmaps to visualize gene expression data from microarrays. This visualization supports both three color heatmaps (ex: green to black to red) and two color heatmaps (ex: white to yellow). By: Institute for Systems Biology
	Column Chart (GWT Integrated) Interactive column chart. By: Google
	Drastic Treemap A dynamic treemap in Flash. By: DrasticData
	Dygraphs The dygraphs JavaScript library produces interactive, zoomable charts of time series. By: Dan Vanderkam
	Filters A Visualization that acts as a control over other visualizations. It is rendered within the browser using HTML. This visualization offers the ability to select some criteria to filter the DataTable used by the controlled visualizations. By: Institute for Systems Biology
	Gauge (GWT Integrated) Each numeric value is displayed as a gauge. By: Google
	Geo Map (GWT Integrated) A map of a country, continent, or region map, with colors and values assigned to specific regions. Values are displayed as a color scale, and you can specify optional hovertext for regions. By: Google
	Intensity Map (GWT Integrated) An intensity map that highlights regions or countries based on relative values. By: Google
	Line Chart (GWT Integrated) Interactive line chart. By: Google
	Magic-Table The Magic Table is a JavaScript library that allows you to see more in your data by applying some simple visual techniques to transform a table. The table is displayed in the browser by the canvas element. Internet Explorer is not supported. By: Greg Ross
	Map (GWT Integrated) An interactive map that uses the Google Maps API. By: Google
	Motion Chart (GWT Integrated) Motion Chart: A dynamic flash based chart to explore several indicators over time. Required columns: bubble name, time and 2 columns of numeric values. Optional columns: Numeric values or categories. By: Google
	Organizational Chart A GWT Integrated simple organizational chart. By: Google
	Parallel Coordinates Chart Parallel Coordinates is a method of visualizing multivariate data. An n-dimensional space is represented as n parallel lines. Works for browsers based on Gecko or Presto (does not work in IE). This is written in Javascript, no Flash required. By: Sri Harsha Allamraju
	Pie Chart (GWT Integrated) Interactive pie chart. By: Google
	Piles of Money Column chart made of of money bills. By: The visapi project
	Scatter Chart (GWT Integrated) Interactive scatter chart. By: Google
	Table (GWT Integrated) A highly customizable table with sorting, paging and selection capabilities. By: Google
	TermCloud A list of terms, where the size and color of each word is determined by a specified frequency value (typically the number of times it appears in some text). By: The visapi project
	Thematic Mapping API Enables visualization of data in Google Earth or other geobrowsers through the use of the Google Visualization API and KML. By: Thematicmapping.org
	WordCloud Displays all words in text with size and color based on the number of time each word appears. By: The visapi project

Additional Information

Below is the talk that Itai Raz, the lead engineer for the Visualization API product at Google, gave at Google I/O 2009 titled “Using the Visualization API with GWT:”

Additional Possibilities

The work above is meant only to serve as a starting point. There is a great deal more information to expand upon. For example, we began this post pulling some information from the XML schema for CVE-2010-1228. One field we did not pull out from the XML file is:

The Common Weakness Enumeration (CWE) represents vulnerability types and NIST provides a CWE Cross Section Mapped into by NVD table. In the above example, we see an entry:

Name	CWE-ID	Description
Race Conditions	CWE-362	The state of a resource can change between the time the resource is checked to when it is accessed.

Clicking on the link will take us to the MITRE site that provides a great deal more information on CWE entries. It is easy enough to expand on the above program to harvest this information for a richer information database.

Another possibility is to expand the above program to pull additional information on the CVE entry. In additional to the data in the NVD CVE XML file, we could pull information from the NVD site. Using CVE-2010-1228 as an example, we could have the program pull down the page:

http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2010-1228

Notice the line:

CVSS v2 Base Score:10.0 (HIGH) (AV:N/AC:L/Au:N/C:C/I:C/A:C) (legend)

The (AV:N/AC:L/Au:N/C:C/I:C/A:C) provides values that were used in determining the base score. If you follow the link, you will see the values used in the calculations:

CVSS Base Score: 10
- Impact Subscore: 10
- Exploitability Subscore: 10
CVSS Temporal Score: Undefined
CVSS Environmental Score: Undefined
Overall CVSS Score: 10

NVD has made available the equations used in calculating the CVSS base score, temporal score, and environmental score.

Three other pieces of information that might provide interesting groupings are:

Access Complexity: Low **NOTE: Access Complexity scored Low due to insufficient information
Authentication: Not required to exploit
Impact Type: Allows unauthorized disclosure of information; Allows unauthorized modification; Allows disruption of service

What information is of interest and how it is used will be dependent on your organization. There is a great deal of information available and many directions you start examining.

Final Thoughts

I am often reminded of the old phrase, “Trust us, we are from the government.” No one really trusts anyone, especially when it comes to matters they do not understand. Just because you are from the security group at your organization, is that reason enough for the CEO to give you unlimited money and authority to do what you see fit? Of course not. While management might trust you, they may not believe that you are capable of seeing the big picture. That is after all their job.

Another great old saying is that “the devil is in the details.” Those details will likely fall in the security domain. In organization across the planet there is a tug of war between the details and the big picture with multiple groups adding in their opinions and views. You need to make the details understandable to your higher management to effectively argue your view. Finding effective metrics and finding clear representation is essential in today’s business. Google Visualization can be a useful tool in accomplishing this task.

RTIR: Adding Incident Response Capabilities to RT

John Gerber — Thu, 07 Aug 2008 21:11:37 +0000

In our last post, “Request Tracker Installation (Part 2 of 2),” we implemented Request Track (RT) on an Apache web server. As a reminder, RT is an enterprise-grade ticketing system which allows for the checking of the status of various tasks including when the tasks were requested, who requested the tasks and why, when the tasks were completed, prioritizing, etc. It was developed by the folks over at Best Practical, and is the leading open-source issue tracking system. Best Practical has also created RT for Incident Response (RTIR), which is “an Open Source incident handling system designed with the needs of CERT teams and other incident-response teams in mind.” The posting “RTIR: RT for Incident Response” outlines the added features of RTIR. Today’s post will build upon the implementation of RT and will walk through the steps to implement RTIR.

Before going any further, I did want to give credit to the Doctor, who has been good enough to post over on his ITIL Service Management blog a mind mapping of the incident management process under ITIL. That mind map is the image used in today’s blog. While it has nothing directly to do with RTIR, it is interesting in terms of incident management and I wanted to give proper credit.

Installation

Below are the basic steps involved with installing RTIR.

1. Download and install required Perl modules.

There are several Perl packages, which are dependent on other packages, and so the cycle goes. You can install there packages using cpan, with commands like:

 /usr/local/src root# perl -MCPAN -e 'install Business::Hours'
 /usr/local/src root# perl -MCPAN -e 'install Net::Whois::RIPE'

Sometimes you can run into problems installing modules in that manner. Another alternative is to pull down the package, untar, configure, compile, and install in a manner similar to:

 root# cd /usr/local/src
 /usr/local/src root# wget http://search.cpan.org/CPAN/authors/\
id/M/MR/MRSAM/Net-CIDR-0.11.tar.gz
 /usr/local/src root# tar xzf Net-CIDR-0.11.tar.gz
 /usr/local/src root# cd Net-CIDR-0.11
 /usr/local/src/Net-CIDR-0.11 root# perl Makefile.PL
 /usr/local/src/Net-CIDR-0.11root# make test
 /usr/local/src/Net-CIDR-0.11 root# make install

To make installation of RTIR as easy as possibly, you may want to make sure the following packages are installed.

Package	File
Business::Hours	http://search.cpan.org/CPAN/authors/id/J/JE/JESSE/\ Business-Hours-0.07.tar.gz
Net::Whois::RIPE	http://search.cpan.org/CPAN/authors/id/P/PA/PAULG/ Net-Whois-RIPE-1.23.tar.gz
Net::CIDR	http://search.cpan.org/CPAN/authors/id/M/MR/MRSAM/\ Net-CIDR-0.11.tar.gz
Business::SLA	http://search.cpan.org/CPAN/authors/id/R/RU/RUZ/\ Business-SLA-0.05.tar.gz
Regexp::Common::net::CIDR	http://search.cpan.org/CPAN/authors/id/R/RU/RUZ/\ Regexp-Common-net-CIDR-0.02.tar.gz
Devel::Cycle	http://search.cpan.org/CPAN/authors/id/L/LD/LDS/\ Devel-Cycle-1.10.tar.gz
PadWalker	http://search.cpan.org/CPAN/authors/id/R/RO/ROBIN/\ PadWalker-1.7.tar.gz
Devel::Symdump	http://search.cpan.org/CPAN/authors/id/A/AN/ANDK/\ Devel-Symdump-2.08.tar.gz
Pod::Coverage	http://search.cpan.org/CPAN/authors/id/R/RC/RCLAMP/\ Pod-Coverage-0.19.tar.gz
Test::Pod::Coverage	http://search.cpan.org/CPAN/authors/id/P/PE/PETDANCE/\ Test-Pod-Coverage-1.08.tar.gz
Test::Memory::Cycle	http://search.cpan.org/CPAN/authors/id/P/PE/PETDANCE/\ Test-Memory-Cycle-1.04.tar.gz
Test::Taint	http://search.cpan.org/CPAN/authors/id/P/PE/PETDANCE/\ Test-Taint-1.04.tar.gz
WWW::Mechanize	http://search.cpan.org/CPAN/authors/id/P/PE/PETDANCE/\ WWW-Mechanize-1.34.tar.gz
Test::HTTP::Server::Simple	http://search.cpan.org/CPAN/authors/id/J/JE/JESSE/\ Test-HTTP-Server-Simple-0.09.tar.gz

2. Download, configure, and install RTFM.

The stable RTIR package was released back to 2004, way before the most recent RT 3 release. Pull down the most recently updated for RT 3 release of the RT FAQ Manager (RTFM). To quote from the FAQ, “RTFM lets you open, categorize and search for ‘articles.’ Like RT, RTFM lets your users contribute additional information to existing articles and makes sure that each article’s full history is preserved for future inspection. RTFM makes it easy to quickly search the knowledge base and find critical information.”

 root# cd /usr/local/src
 /usr/local/src root# wget http://download.bestpractical.com/pub/rt/devel/\
RTIR_M3/RTFM-2.2.2.tar.gz
 /usr/local/src root# tar xzf RTFM-2.2.2.tar.gz
 /usr/local/src root# cd RTFM-2.2.2
 /usr/local/src/RTFM-2.2.2 root# perl Makefile.PL
 /usr/local/src/RTFM-2.2.2 root# make
 /usr/local/src/RTFM-2.2.2 root# make install
 /usr/local/src/RTFM-2.2.2 root# make initdb
 /usr/local/src/RTFM-2.2.2 root# cd ..
 /usr/local/src root# wget http://download.bestpractical.com/pub/rt/devel/\
RTIR_M3/RTFM-Extension-ArticleTemplates-0.01.tar.gz
 /usr/local/src root# tar xzf RTFM-Extension-ArticleTemplates-0.01.tar.gz
 /usr/local/src root# cd RTFM-Extension-ArticleTemplates-0.01
 /usr/local/src/RTFM-Extension-ArticleTemplates-0.01root# perl Makefile.PL
 /usr/local/src/RTFM-Extension-ArticleTemplates-0.01root# make
 /usr/local/src/RTFM-Extension-ArticleTemplates-0.01root# make install
 /usr/local/src/RTFM-Extension-ArticleTemplates-0.01root# cd ..

3. Download, configure, and install RTIR.

Pull down the most recently updated for RT 3 release of RTIR.

 root# cd /usr/local/src
 /usr/local/src root# wget http://download.bestpractical.com/pub/rt/devel/\
RTIR_M3/RT-IR-2.3.17.tar.gz
 /usr/local/src root# tar xzf RT-IR-2.3.17.tar.gz
 /usr/local/src root# cd RT-IR-2.3.17
 /usr/local/src/RT-IR-2.3.17 root# perl Makefile.PL
 /usr/local/src/RT-IR-2.3.17 root# make install

4. Edit RT configuration file.

Edit the RT configuration file /opt/rt3/etc/RT_SiteConfig.pm adding the RTIR configuration file /opt/rt3/local/plugins/RT-IR/etc/RTIR_Config.pm.

 /usr/local/src/RT-IR-2.3.17 root#  vi /opt/rt3/etc/RT_SiteConfig.pm

Add the lines:

# The RTIR config file
$RTIR_CONFIG_FILE = "/opt/rt3/local/plugins/RT-IR/etc/RTIR_Config.pm";
require $RTIR_CONFIG_FILE || die ("Couldn't load RTIR config file '$RTIR_CONFIG_FILE'\n$@");
Set(@Plugins, 'RT::FM', 'RT::IR');

5. Initialize the database.

Update the RT database.

 /usr/local/src/RT-IR-2.3.17 root# make initdb

6. Stop and restart the Apache server.

For good measure, restart the Apache server.

 /usr/local/src/RT-IR-2.3.17 root# /usr/local/apache/bin/apachectl stop
 /usr/local/src/RT-IR-2.3.17 root# /usr/local/apache/bin/apachectl start

7. Configure RTIR.

Configuration of RT and RTIR is a topic for another post; maybe even a book. I will include below basic instructions from RTIR readme. Please see the next section for additional documentation.

1) Using RT’s configuration interface, add the email address
of the Network Operations Team (the people who will handle
activating and removing Blocks) as AdminCC on the Blocks queue.
RT -> Configuration -> Queues -> Blocks -> Watchers

2) You may want to modify the email messages that are automatically
sent on the creation of Investigations and Blocks.
RT -> Queues ->