Security Advancements at the Monastery » Google

Google Visualization: An Example Graphing NVD CVE Data

John Gerber — Fri, 16 Apr 2010 15:54:42 +0000

Google visualization offers graphing abilities to any number of projects. Why should security professionals care? If you are going to have to collect and present security metrics, it is best to showcase them in the very best manner possible. Andrew Jaquith in his article, “Creating meaningful information security metrics” states, “For 2010, Forrester Research expects that overall security budgets will rise less than 5 percent over 2009 –higher than in the previous year, but not by much.” Andrew goes on to point out, “smart security managers, sensing sudden vulnerability in their budgets, seek better ways to measure and prove the value of what they do every day.”

In today’s work environment there is a need to show changes, potential risks, improved performance, etc. in all areas of the company’s operations. Security professionals need to be prepared to answer the basic question, “why should the CIO or CEO care about security?” CSO Online has a great quote from the post, “From the CIO: Why You Didn’t Get the CISO Job” that challenges us to consider our views when it comes to security. The post states, “laser focus on your speciality is great in middle management. It’s what we want. One of the really hard things about jumping from management to executive is a focus on the whole of the business. It’s a rare person who manages it quickly or easily.” That is basically the problem with metrics. It is a battle between generalization to the point of uselessness and details to the point of not being understandable or collectible. At the end of the day, something needs to be done because the security industry is currently leaving upper management in the position of not understanding what is going on within their business. That is a risk that not acceptable.

Andrew’s article discusses what kind of security metrics should be used. Additional sources of information on security metrics can be found in a previous post entitled “Security Metrics.” The post provides links to wonderful sources on security metric information. You might also want to take a look at the CIS Consensus Security Metrics v1.0.0 guide, NIST Special Publication (SP) 800-55 Rev 1 “Security Metrics Guide for Information Technology Systems”, NIST IR-7564 “Directions in Security Metrics Research”, “Twenty Most Important Controls and Metrics for Effective Cyber Defense and Continuous FISMA Compliance,” and “Metrics, measures & Myths.” Once you have start gathering metrics, you will want to present them in an easy to understand format. This is where Google Visualization can help.

Today’s post walks through an example using the data from the National Institute of Standards and Technology (NIST) National Vulnerability Database (NVD ) Common Vulnerabilities and Exposures (CVE) database. The purpose is to provide a working example from which you can learn and apply to the various metrics gathered at your organization.

Data Source

A previous post, “Standardization and Interoperability in Security,” discussed how the Security Content Automation Protocol (SCAP) is an attempt to help defenders by providing a collection of XML schemas/standards that allow technical security information to be exchanged between tools. SCAP components consists of:

Common Configuration Enumeration (CCE): provide unique identifiers to system configuration issues in order to facilitate fast and accurate correlation of configuration data across multiple information sources and tools.
Common Platform Enumeration (CPE): a structured naming scheme for information technology systems, platforms, and packages.
Common Vulnerability Enumeration (CVE): a dictionary of publicly known information security vulnerabilities and exposures.
Common Vulnerability Scoring System (CVSS): a vulnerability scoring system designed to provide an open and standardized method of rating IT vulnerabilities. NIST has even provided a calculator for creating CVSS vulnerability severity scores.
eXtensible Checklist Configuration Description Format (XCCDF): a specification language for writing security checklists, benchmarks, and related kinds of documents. NIST has released the NIST Interagency Report 7275 Revision 3 “Specification for Extensible Configuration Checklist Description Format (XCCDF) Version 1.1.4.”
Open Vulnerability Assessment Language (OVAL): an information security community standard to promote open and publicly available security content, and to standardize the transfer of this information across security tools and services.

We are going to make use of the data from NVD/CVE XML feed with the Common Vulnerability Scoring System (CVSS) mappings (version 2.0). NIST documentation states:

CVSS provides an open framework for communicating the characteristics and impacts of IT vulnerabilities. Its quantitative model ensures repeatable accurate measurement while enabling users to see the underlying vulnerability characteristics that were used to generate the scores. Thus, CVSS is well suited as a standard measurement system for industries, organizations, and governments that need accurate and consistent vulnerability impact scores. Two common uses of CVSS are prioritization of vulnerability remediation activities and in calculating the severity of vulnerabilities discovered on one’s systems.

NVD provides CVSS ‘base scores‘ representing the innate characteristics of each vulnerability. ‘Temporal scores,’ which change over time due to events external to the vulnerability, are not provided though NVD does provide a CVSS score calculator. This allows an organization to add temporal data and even factor in ‘environmental scores‘ customized to reflect the impact of the vulnerability on the organization. Please refer to the CVSS standards guide and the OWASP Risk Rating Methodology concerning factors involved in estimating the severity of risks to your business.

NVD CVE XML Schema

For our example, we will be using the data feeds nvdcve-2.0-2010.xml and nvdcve-2.0-2009.xml. Examining the CVE XML 2.0 Schema, we are particularly interested in certain vulnerability and CVSS scoring information. For example, for CVE-2010-1228, we will parse and pull the following kind of information:

 id="CVE-2010-1228">
  CVE-2010-1228
  2010-04-01T18:30:00.453-04:00
  
  2010-04-05T00:00:00.000-04:00
  
  
    
      10.0
      NETWORK
      LOW
      NONE
      COMPLETE
      COMPLETE
      COMPLETE
      http://nvd.nist.gov

Using Perl to Retrieve the CVE File

Initially we will read the nvdcve-2.0-2010.xml and nvdcve-2.0-2009.xml files. If we start retrieving the file regularly, we would want to change this to nvdcve-2.0-recent.xml. Of course, previous years can also be read in to provide a longer perspective on vulnerabilities. A simple example of a Perl subroutine to read the NVD CVE file and save it locally would be:

sub readpage {
   my($url,$nvd_file) = @_;
   my($proxy) = "http://your-proxy-server:proxy-port";
   my $ua = new LWP::UserAgent;
   $ua->proxy(http  => $proxy);
   $ua->proxy(ftp => $proxy);
   $ua->proxy(https => $proxy);
   # Go out and retrieve page
   my $req = new HTTP::Request('GET', $url);
   my $res = $ua->request($req);
   my $pjstatus = 1;
   # Check if the requested webpage is there and return results
   if ($res->is_success) { # Request successful
       open(OUTFILE,">$nvd_file") || ($pjstatus = 0);
       if ($pjstatus) {
          print OUTFILE $res->content;
       }
       close(OUTFILE);
   }
   else {
      $pjstatus = 0;
   }
   return($pjstatus);
}

Please substitute “http://your-proxy-server:proxy-port” with your site’s proxy server and port, if applicable.

Creating a MYSQL Table to Hold the Data

There is a great deal of information in the NVD CVE file. You will need to determine what information your organization will be interested in storing and graphing. For better or worse, folks have come to expect vulnerabilities to have a “Low,” “Medium,” or “High” score. NIST has stated concerning the NVD Vulnerability Severity Ratings:

NVD provides severity rankings of “Low,” “Medium,” and “High” in addition to the numeric CVSS scores but these qualitative rankings are simply mapped from the numeric CVSS scores:
1. Vulnerabilities are labeled “Low” severity if they have a CVSS base score of 0.0-3.9.
2. Vulnerabilities will be labeled “Medium” severity if they have a base CVSS score of 4.0-6.9.
3. Vulnerabilities will be labeled “High” severity if they have a CVSS base score of 7.0-10.0.

While preferring quantitative over qualitative values, for this example I would like to create a stacked column chart. We will add a severity column which is based on the CVSS score. An example table follows:

CREATE DATABASE vulnerabilities;
USE vulnerabilities;
DROP TABLE IF EXISTS `nvdcve`;
CREATE TABLE `nvdcve` (
  `cve_id` varchar(13) NOT NULL,
  `published` datetime default NULL,
  `modified` datetime default NULL,
  `score` DECIMAL(5,2) default '0.0',
  `severity` varchar(6) default 'LOW',
  `vector` varchar(25) default NULL,
  `complexity` varchar(25) default NULL,
  `authentication` varchar(25) default NULL,
  `confidentiality` varchar(25) default 'NONE',
  `integrity` varchar(25) default 'NONE',
  `availability` varchar(25) default 'NONE',
  `summary` varchar(512) default NULL,
  PRIMARY KEY  (`cve_id`),
  INDEX (score),
  INDEX (vector)
)

Using Perl Populating the Database

Populating the database table is simply a matter of reading the file and adding the entries to the table. An example Perl subroutine follows:

sub readxml {
   my($nvd_file, $dbh) = @_;
   my $parser = XML::LibXML-> new();
   my $doc    = $parser-> parse_file($nvd_file);
   my $xc     = XML::LibXML::XPathContext-> new( $doc->documentElement() );
   $xc-> registerNs(
      def  => 'http://scap.nist.gov/schema/feed/vulnerability/2.0' );
   $xc-> registerNs(
     vuln => 'http://scap.nist.gov/schema/vulnerability/0.4' );
   $xc-> registerNs( cvss => 'http://scap.nist.gov/schema/cvss-v2/0.2' );
   for my $entry ($xc-> findnodes("/def:nvd/def:entry")) {
      my $cve = $xc-> find('vuln:cve-id',$entry);
      my $published = $xc-> find('vuln:published-datetime', $entry);
      my $modified = $xc-> find('vuln:last-modified-datetime', $entry);
      my $summary = $xc-> find('vuln:summary', $entry);
      my $skip = 0;
      my ($metrics) = $xc-> findnodes('vuln:cvss/cvss:base_metrics', $entry) or ($skip = 1);
      if (! $skip) {
         my $score = $xc-> find('cvss:score', $metrics);
         my $vector = $xc-> find('cvss:access-vector', $metrics);
         my $complexity = $xc-> find('cvss:access-complexity', $metrics);
         my $authentication = $xc-> find('cvss:authentication', $metrics);
         my $confidentiality =
            $xc-> find('cvss:confidentiality-impact', $metrics);
         my $integrity = $xc-> find('cvss:integrity-impact', $metrics);
         my $availability = $xc-> find('cvss:availability-impact', $metrics);
         my $severity = "LOW";
         if (int($score) >= 7) {
            $severity = "HIGH";
         }
         elsif (int($score) >= 4) {
            $severity = "MEDIUM";
         }
         my $sql = qq{ SELECT count(*) FROM nvdcve WHERE cve_id=? };
         my $sth = $dbh->prepare( $sql );
         my $rc = $sth->execute($cve);
         if ( $rc) {
            my($exist) = $sth->fetchrow_array();
            if (! $exist) {
                $sql = qq{ INSERT INTO nvdcve SET cve_id=?,
published=?, modified=?, score=?, severity=?, vector=?, complexity=?,
authentication=?, confidentiality=?, integrity=?,availability=?, summary=? };
               $sth = $dbh->prepare( $sql );
               $rc = $sth->execute($cve,$published,$modified,$score,
$severity,$vector,$complexity,$authentication,
$confidentiality,$integrity,$availability,$summary);
            }
         }
      }
   }
}

The Perl Program to Pull It All Together

The above subroutines use the Perl modules LWP::UserAgent, XML::LibXML, XML::LibXML::XPathContext, and DBI. A sample Perl program that calls the above subroutines to pull down the NVD CVE data and load it into a MySQL table would be:

#!/usr/local/bin/perl -w
use LWP::UserAgent;
use XML::LibXML;
use XML::LibXML::XPathContext;
use DBI;
BEGIN{push @INC, "/home/jgerber/projects/nvd/perl"}
use nvdsubs qw($db_host $db $mysql_user $mysql_passwd $mysql.sock
readpage readxml );
# Main
my $datadir = "/home/johngerber/projects/nvd/data";
my @timeData = localtime(time);
my $year = 1900 + $timeData[5];
my $prev_year = 1900 + $timeData[5] - 1;
my $url = "http://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-" .
    $year . ".xml";
my $prev_url = "http://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-" .
    $prev_year . ".xml";
my $nvd_file = $datadir  . "/nvdcve-". $year . ".xml";
my $prev_nvd_file = $datadir  . "/nvdcve-". $prev_year . ".xml";
$db = "vulnerabilities";
local($dbh) = DBI->connect("DBI:mysql:mysql_socket=$mysql.sock;$db:$db_host",
$mysql_user, $mysql_passwd) || die "ERROR: Connecting: $DBI::errstr\n";
my ($pjstatus) = &readpage($prev_url,$prev_nvd_file);
if ($pjstatus) {
   &readxml($prev_nvd_file,$dbh);
}
$pjstatus = &readpage($url,$nvd_file);
if ($pjstatus) {
   &readxml($nvd_file,$dbh);
}
exit;

The nvdsubs.pm file will not be included in this post. The subroutines are defined and the only pieces missing are the MySQL database username and password. You don’t need mine. Add your own. At this point, we have everything we need to finally use Google Visualization to create a graph.

Google Visualization

We are going to create a Perl program that will read our MySQL nvdcve table and generate the JavaScript that will render our charts on the client’s browser. First, we want to define the JavaScript we want to produce. Just to alleviate some concerns, with Google Visualization your data is only shared between your server and the client connecting. This is unlike Google Charts where your data is sent to Google where it is made into a chart and the result is sent back. Google states concerning the logging of chart data (via Google Charts), “The chart data included in the HTTP request is saved in temporary logs for no longer than two weeks for internal testing and debugging purposes.” Every example in the Google Visualization Gallery will state the data policy. For Google Charts, stated at the bottom of the page for each gadget description the data policy:

While Google Visualization gadgets will have the following stated data policy:

Loading Google Libraries

The first thing the JavaScript needs to do is load the required libraries. This is accomplished with the lines:

Area Chart and Table

In this example we are going to create an column chart. In a later section, “Other Charting Options” (see below) we define different Google Visualization charting options.

JavaScript code for a sample column chart would be:

    <script type='text/javascript'>
      google.load('visualization', '1', {packages:['columnchart']});
      google.setOnLoadCallback(drawChart);
      function drawChart() {
        var data = new google.visualization.DataTable();
        data.addColumn('date', 'Date');
        data.addColumn('number', 'High');
        data.addColumn('number', 'Medium');
        data.addColumn('number', 'Low');
        data.addRows([
           [new Date(2009, 0, 30),92,97,3],
           [new Date(2009, 1, 27),168,142,25],
           [new Date(2009, 2, 31),141,165,9],
           [new Date(2009, 3, 30),132,203,12],
           [new Date(2009, 4, 29),158,153,8],
           [new Date(2009, 5, 30),200,199,22],
           [new Date(2009, 6, 31),190,195,11],
           [new Date(2009, 7, 31),127,139,14],
           [new Date(2009, 8, 30),233,208,14],
           [new Date(2009, 9, 30),163,167,18],
           [new Date(2009, 10, 30),129,172,8],
           [new Date(2009, 11, 31),200,211,19],
           [new Date(2010, 0, 29),157,139,14],
           [new Date(2010, 1, 26),137,143,12],
           [new Date(2010, 2, 31),252,242,18],
           [new Date(2010, 3, 13),92,118,17]
        ]);
        var chart = new google.visualization.ColumnChart(document.getElementById('s4graph'));
        chart.draw(data, {displayAnnotations:true, is3D: true, isStacked: true, min: 0,
          allowHtml: true, colors:[{color:'#E41B17', darker:'#C11B17'}, {color:'#FFA500', darker:'#E56717'}, {color:'#FFE87C', darker:'#C8B560'}]});
      }
    script>

The resulting image would be the following column chart:

Rendering the Table

When providing qualitative results, I like to back them up with more accurate numeric values. Let us include a table with links to the CVSS scores for each vulnerability.

    <script type='text/javascript'>
      google.load('visualization', '1', {packages:['table']});
      google.setOnLoadCallback(drawChart);
      function drawChart() {
        var data2 = new google.visualization.DataTable();
        data2.addColumn('date', 'Date');
        data2.addColumn('number', 'High');
        data2.addColumn('number', 'Medium');
        data2.addColumn('number', 'Low');
        data2.addRows([
           [{v:new Date(2009, 0, 30),
              f:'2009-01-30'}, 92,97,3],
           [{v:new Date(2009, 1, 27),
              f:'2009-02-27'}, 168,142,25],
           [{v:new Date(2009, 2, 31),
              f:'2009-03-31'}, 141,165,9],
           [{v:new Date(2009, 3, 30),
              f:'2009-04-30'}, 132,203,12],
           [{v:new Date(2009, 4, 29),
              f:'2009-05-29'}, 158,153,8],
           [{v:new Date(2009, 5, 30),
              f:'2009-06-30'}, 200,199,22],
           [{v:new Date(2009, 6, 31),
              f:'2009-07-31'}, 190,195,11],
           [{v:new Date(2009, 7, 31),
              f:'2009-08-31'}, 127,139,14],
           [{v:new Date(2009, 8, 30),
              f:'2009-09-30'}, 233,208,14],
           [{v:new Date(2009, 9, 30),
              f:'2009-10-30'}, 163,167,18],
           [{v:new Date(2009, 10, 30),
              f:'2009-11-30'}, 129,172,8],
           [{v:new Date(2009, 11, 31),
              f:'2009-12-31'}, 200,211,19],
           [{v:new Date(2010, 0, 29),
              f:'2010-01-29'}, 157,139,14],
           [{v:new Date(2010, 1, 26),
              f:'2010-02-26'}, 137,143,12],
           [{v:new Date(2010, 2, 31),
              f:'2010-03-31'}, 252,242,18],
           [{v:new Date(2010, 3, 13),
              f:'2010-04-13'}, 92,118,17],
        ]);
        var table = new google.visualization.Table(document.getElementById('s4graph_tab'));
        table.draw(data2, {showRowNumber: true, sortAscending: false, sortColumn: 0, allowHtml: true});
      }
    script>

The JavaScript code assumes there is a PHP program called cvealerts.php under the /nvd directory on your web server. Adjust to your environment. A sample PHP program that could be used for cvealerts.php is provided below. The resulting table chart would look like:

Handling Events: Interactions Between Graphs

We now have two different types of graphs representing the same data. We want to add interaction between the graphs so the viewer can see the relationship. With tables rows are selected when the user clicks, which correspond to the whole column of the stacked column chart. It is not a perfect fit, but it does demonstrate nicely use of adding interactions.

        // Set a 'select' event listener for the table.
        // When the table is selected,
        // we set the selection on the line graph.
        google.visualization.events.addListener(table, 'select', function() {
          chart.setSelection([{row: table.getSelection()[0].row, column: 1}]);
         });
        // Set a 'select' event listener for the graph.
        // When the graph is selected,
        // we set the selection on the table.
        google.visualization.events.addListener(chart, 'select', function() {
           table.setSelection([{row: chart.getSelection()[0].row}]);
        });

Providing Detailed Information

When the table chart link is clicked, we would like to provide some detailed information about the vulnerability. For this example, we will do this with a simple PHP program placed in the /nvd directory on the web server. The program is called cvealerts.php.


session_start();
function db_connect($table) {
   $result = mysql_pconnect(":", "", "");
   if (!$result) return false;
   if (!mysql_select_db($table)) return false;
   return $result;
}
function do_html_header($title,$checkuser,$logpage) {
?>
  <html> <head> <title>$title?>title>head>
  <body bgcolor="#FFFFFF">

}
function do_html_footer() {
?>
<table>
<tr><td ALIGN=CENTER NOWRAP WIDTH="590">font>
<font face="Verdana, Arial, Helvetica" size=-2>Notice to Users: Use
of this system constitutes consent to security monitoring and testing.
<br>All activity is logged with your host name and IP address.font>
td>tr>
table>
body>
 html>

}
// Main
$dates= array();
$stringlist = "";
if (isset($_GET['date'])) {
    $passdates = explode(",",$_GET['date']);
    for ($index=0; $index<count($passdates); $index++) {
       array_push($dates, $passdates[$index]);
       $stringlist .= $passdates[$index] . " ";
    }
}
else {
  print("Confusion over how you arrived at this page.\n");
  exit;
}
$stringlist = preg_replace("/ $/", "",$stringlist);
do_html_header("Review NVD CVE Announcements for Month Ending $stringlist",1,1);
$nvd_host = "http://web.nvd.nist.gov/view/vuln/detail?vulnId=";
$conn = db_connect("vulnerabilities");
if (!$conn)
   logit("Could not connect to database vulnerabilities - please try later.\n",1);
for ($index=0; $index<count($dates); $index++) {
   $rule = $dates[$index];
   $sql = "SELECT cve_id,score,published,vector,severity,complexity,left(summary,50)
    FROM vulnerabilities.nvdcve
      WHERE date_format(published,'%Y-%m')='$rule'
       ORDER BY (score+0)";
   $result = mysql_query($sql,$conn);
   if (!$result)
       logit("Problem with $sql\n",1);
   print("
\n");for($count=1;list($cve_id,$score,$date,$vector,$severity,$complexity,$shortsum)=mysql_fetch_array($result, MYSQL_NUM);++$count){?><tr><td CLASS="plfieldhdrleft" WIDTH="20%" BGCOLOR='#F0F5FF'>  print("$cve_id"); ?>
      td>
      <td CLASS="plfieldhdrleft" BGCOLOR='#F9FCFF'>
        print($score); ?>
      td>
      <td CLASS="plfieldhdrleft" BGCOLOR='#F0F5FF'>
        print($date); ?>
      td>
      <td CLASS="plfieldhdrleft" BGCOLOR='#F9FCFF'>
        print($vector); ?>
      td>
      <td CLASS="plfieldhdrleft" BGCOLOR='#F0F5FF'>
        print($severity); ?>
      td>
      <td CLASS="plfieldhdrleft" BGCOLOR='#F9FCFF'>
        print($complexity); ?>
      td>
      <td CLASS="plfieldhdrleft" BGCOLOR='#F0F5FF'>
        print($shortsum); ?>
      td>
      tr>

   }
}
print("
Bulletin 
Impact 
Date 
Vector 
Severity 
Complexity 
Short Summary
            
        

      
      
");
do_html_footer();

The PHP program would generate a HTML table displaying the NVD CVE alerts for that month. The table would look like:

When the CVE link is clicked on, the user is taken to the NIST NVD site where additional information is available.

Using Perl to Create the JavaScript

The Perl code is rather simple now that we have the MySQL tables defined and the JavaScript we want to generate. Much of the code consists of the JavaScript listed above.

#!/usr/local/bin/perl -w
use DBI;
use Time::Local;
use POSIX qw(strftime);
use LWP::UserAgent;
BEGIN{push @INC, "/home/jgerber/projects/nvd/perl"}
use ornl_feds qw($db_host $db $mysql_user $mysql_passwd );
sub slide_nvd_alerts {
  my($min_date,$graph_name,$web_link,$dbh) = @_;
  my $slide = "";
  my $slide_head = qq!
    
!;
   if ($min_date eq "") {
      my $sql2 = qq{ SELECT min(published) FROM vulnerabilities.nvdcve };
      my $sth2 = $dbh->prepare( $sql2 );
      my $rc2 = $sth2->execute();
      if ($rc2) {
         $min_date = $sth2->fetchrow_array();
      }
   }
   my $table_data = "";
   my $graph_data = "";
   my $sql2 = qq{ select date_format(published,'%Y-%m'),severity,count(severity)
      FROM vulnerabilities.nvdcve where published >= ? group by date_format(published,'%Y-%m'),severity };
   my $sth2 = $dbh->prepare( $sql2 );
   my $rc2 = $sth2->execute($min_date);
   if ($rc2) {
      my ($change,$virgin,$ht,$mt,$lt,$mmax_date) = ("",1,0,0,0,"");
      while (my($snapshot_date, $severity, $pcount) = $sth2->fetchrow_array()) {
         my $sql3 = qq{ SELECT max(published) FROM vulnerabilities.nvdcve where
date_format(published,'%Y-%m')=? };
         my $sth3 = $dbh->prepare( $sql3 );
         my $rc3 = $sth3->execute($snapshot_date);
         $max_date =  $sth3->fetchrow_array();
         $max_date =~ s/ \S+$//;
         if ($change ne $snapshot_date) {
            if (! $virgin) {
                my($year,$month,$day) = split("-",$mmax_date);
                my $mmonth = $month;
                $month--;
                $graph_data .= qq!           [new Date($year, $month, $day),$ht,$mt,$lt],
!;
                $table_data .= qq!           [{v:new Date($year, $month, $day),
              f:'$mmax_date'}, $ht,$mt,$lt],
!;
                ($ht,$mt,$lt) = (0,0,0);
             }
             $change = $snapshot_date;
          }
          if ($severity eq "HIGH") { $ht = $pcount; }
          elsif ($severity eq "MEDIUM") { $mt = $pcount; }
          elsif ($severity eq "LOW") { $lt = $pcount; }
          if ($mmax_date eq "") { $mmax_date = $max_date; }
          if ($mmax_date lt $max_date) { $mmax_date = $max_date; }
          $virgin = 0;
      }
      my($year,$month,$day) = split("-",$mmax_date);
      my $mmonth = $month;
      $month--;
      $graph_data .= qq!           [new Date($year, $month, $day),$ht,$mt,$lt]
!;
     $table_data .= qq!           [{v:new Date($year, $month, $day),
              f:'$mmax_date'}, $ht,$mt,$lt],
!;
   }
   $table_data .= "        ]);\n";
   $graph_data .= "        ]);\n";
   $slide = $slide_head .  $graph_data . $slide_head_table . $table_data . $slide_tail;
   return($slide);
}
sub slide_body {
  my($graph_name,$title,$style) = @_;
  my $table_name = $graph_name . "_tab";
  my $table_text = "div id=\"$table_name\"";
  if ($style ne "") {
     $table_text .= " style=\'$style\'";
  }
  my $slide2 = "$title
\n";
  my $itext = "div id=\"$graph_name\"";
  if ($style ne "") {
     $itext .= " style=\'$style\'";
  }
  $slide2 .= qq{
    
    <$itext>
    <$table_text>
       
    
    
  };
  return($slide2);
}
# Main
my $web_link = "/nvd";
my $results_dir = "/data/html" . $web_link;
my $result_file = $results_dir . "/nvdcve_stats.html";
my $debug = 1;
my $db = "vulnerabilities";
local($dbh) = DBI->connect("DBI:mysql:$db:$db_host", $mysql_user, $mysql_passwd) ||
   die "ERROR: Connecting: $DBI::errstr\n";
$slides_data .= &slide_body("s4graph","NVD CVE Alerts","width:700px; height:400px;");
$slides_head .= &slide_nvd_alerts("","s4graph",$web_link,$dbh);
open(OUTFILE,">$result_file");
print OUTFILE "\nNVD CVE Statistics\n";
print OUTFILE "\n";
print OUTFILE $slides_head;
print OUTFILE "\n\n";
print OUTFILE $slides_data;
print OUTFILE "\n";
close(OUTFILE);
exit;

Other Charting Options

Google, Google users, and other companies have shared some JavaScript visualizations built on the Google Visualization API to help you get started. Below are some example:

	Annotated Time Line (GWT Integrated) An animated time series chart. By: Google
	Area Chart (GWT Integrated) Interactive area chart. By: Google
	Bar Chart (GWT Integrated) Interactive bar chart. By: Google
	Bars of Stuff Fun bar charts using images of trains, chocolate, worms, and more. By: The visapi project
	Bio Heat Map Heatmaps are a useful way to visualize matricies of data. Scientists often use green-black-red heatmaps to visualize gene expression data from microarrays. This visualization supports both three color heatmaps (ex: green to black to red) and two color heatmaps (ex: white to yellow). By: Institute for Systems Biology
	Column Chart (GWT Integrated) Interactive column chart. By: Google
	Drastic Treemap A dynamic treemap in Flash. By: DrasticData
	Dygraphs The dygraphs JavaScript library produces interactive, zoomable charts of time series. By: Dan Vanderkam
	Filters A Visualization that acts as a control over other visualizations. It is rendered within the browser using HTML. This visualization offers the ability to select some criteria to filter the DataTable used by the controlled visualizations. By: Institute for Systems Biology
	Gauge (GWT Integrated) Each numeric value is displayed as a gauge. By: Google
	Geo Map (GWT Integrated) A map of a country, continent, or region map, with colors and values assigned to specific regions. Values are displayed as a color scale, and you can specify optional hovertext for regions. By: Google
	Intensity Map (GWT Integrated) An intensity map that highlights regions or countries based on relative values. By: Google
	Line Chart (GWT Integrated) Interactive line chart. By: Google
	Magic-Table The Magic Table is a JavaScript library that allows you to see more in your data by applying some simple visual techniques to transform a table. The table is displayed in the browser by the canvas element. Internet Explorer is not supported. By: Greg Ross
	Map (GWT Integrated) An interactive map that uses the Google Maps API. By: Google
	Motion Chart (GWT Integrated) Motion Chart: A dynamic flash based chart to explore several indicators over time. Required columns: bubble name, time and 2 columns of numeric values. Optional columns: Numeric values or categories. By: Google
	Organizational Chart A GWT Integrated simple organizational chart. By: Google
	Parallel Coordinates Chart Parallel Coordinates is a method of visualizing multivariate data. An n-dimensional space is represented as n parallel lines. Works for browsers based on Gecko or Presto (does not work in IE). This is written in Javascript, no Flash required. By: Sri Harsha Allamraju
	Pie Chart (GWT Integrated) Interactive pie chart. By: Google
	Piles of Money Column chart made of of money bills. By: The visapi project
	Scatter Chart (GWT Integrated) Interactive scatter chart. By: Google
	Table (GWT Integrated) A highly customizable table with sorting, paging and selection capabilities. By: Google
	TermCloud A list of terms, where the size and color of each word is determined by a specified frequency value (typically the number of times it appears in some text). By: The visapi project
	Thematic Mapping API Enables visualization of data in Google Earth or other geobrowsers through the use of the Google Visualization API and KML. By: Thematicmapping.org
	WordCloud Displays all words in text with size and color based on the number of time each word appears. By: The visapi project

Additional Information

Below is the talk that Itai Raz, the lead engineer for the Visualization API product at Google, gave at Google I/O 2009 titled “Using the Visualization API with GWT:”

Additional Possibilities

The work above is meant only to serve as a starting point. There is a great deal more information to expand upon. For example, we began this post pulling some information from the XML schema for CVE-2010-1228. One field we did not pull out from the XML file is:

The Common Weakness Enumeration (CWE) represents vulnerability types and NIST provides a CWE Cross Section Mapped into by NVD table. In the above example, we see an entry:

Name	CWE-ID	Description
Race Conditions	CWE-362	The state of a resource can change between the time the resource is checked to when it is accessed.

Clicking on the link will take us to the MITRE site that provides a great deal more information on CWE entries. It is easy enough to expand on the above program to harvest this information for a richer information database.

Another possibility is to expand the above program to pull additional information on the CVE entry. In additional to the data in the NVD CVE XML file, we could pull information from the NVD site. Using CVE-2010-1228 as an example, we could have the program pull down the page:

http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2010-1228

Notice the line:

CVSS v2 Base Score:10.0 (HIGH) (AV:N/AC:L/Au:N/C:C/I:C/A:C) (legend)

The (AV:N/AC:L/Au:N/C:C/I:C/A:C) provides values that were used in determining the base score. If you follow the link, you will see the values used in the calculations:

CVSS Base Score: 10
- Impact Subscore: 10
- Exploitability Subscore: 10
CVSS Temporal Score: Undefined
CVSS Environmental Score: Undefined
Overall CVSS Score: 10

NVD has made available the equations used in calculating the CVSS base score, temporal score, and environmental score.

Three other pieces of information that might provide interesting groupings are:

Access Complexity: Low **NOTE: Access Complexity scored Low due to insufficient information
Authentication: Not required to exploit
Impact Type: Allows unauthorized disclosure of information; Allows unauthorized modification; Allows disruption of service

What information is of interest and how it is used will be dependent on your organization. There is a great deal of information available and many directions you start examining.

Final Thoughts

I am often reminded of the old phrase, “Trust us, we are from the government.” No one really trusts anyone, especially when it comes to matters they do not understand. Just because you are from the security group at your organization, is that reason enough for the CEO to give you unlimited money and authority to do what you see fit? Of course not. While management might trust you, they may not believe that you are capable of seeing the big picture. That is after all their job.

Another great old saying is that “the devil is in the details.” Those details will likely fall in the security domain. In organization across the planet there is a tug of war between the details and the big picture with multiple groups adding in their opinions and views. You need to make the details understandable to your higher management to effectively argue your view. Finding effective metrics and finding clear representation is essential in today’s business. Google Visualization can be a useful tool in accomplishing this task.

Santa’s Secrets Leaked

John Gerber — Mon, 21 Dec 2009 22:00:11 +0000

In a stunning possible security breach, Gregory Mone reveals in his book, “The Truth About Santa: Wormholes, Robots, and What Really Happens on Christmas Eve” some of the sensitive data loss that has been occurring in Father Christmas‘ organization. Answers to questions that have plagued human kind since the first report of the jolly old elf are beginning to appear on the Internet. Through heavy research and interviews with scientists and other field experts (management consultants, Berkeley astrophysicists, Navy SEALs) Mone was able to discover that Baba Noel accomplishes the seemingly impossible annual mission using advanced science and technology. Mone claims no information was obtained through disgruntled elves.

On a mission to reveal the truth behind Pere Noel, Mone took time for an interview on NPR’s Morning Edition and did an one hour lecture at MIT. Shaula Clark reporting for the Boston Phoenix on the MIT lecture, exposed some of Babbo Natale’s trade secrets:

Kanakaloka is not immortal, but retains his jolly vigor with the help of organ printers.
Swiety Mikolaj does not, in fact, leave toys under the tree; instead, he comes bearing complex chemical reactions — toys assemble themselves in their packaging.
Ded Moroz’s Christmas Eve rounds are actually accomplished via several teams of Santa-recruited lieutenants, a series of short-distance wormholes, and time travel.
Papai Noel’s base of operations (actually in Greenland, not the North Pole) is greatly threatened by global warming — to keep his unfathomably large server farm cool, he needs the Arctic chill. Papai Noel’s own green initiatives include planting trees and cloning his elves (“because he wouldn’t want [them] breeding on their own”).

According to Mone, Sinter Klaas uses tools that are hundreds of years beyond what we have at our disposal. For example, “Santa’s suit is laden with what are called metamaterials, which have the effect of bending light around a person so that they turn invisible” — which can come in handy if there are curious children peeking during his Christmas deliveries.

Questions on the Internet have been raised as to where Mone may have obtained his information. At the beginning of the month, Mone traveled to Google allegedly to take part in the Authors@Google series. During the talk Mone discussed how implanted listening devices in the ornaments help Hoteiosho keep the naughty and nice kids straight. Also discussed was the use of cloning and wormhole technology to help Baba Chaghaloo get to every household. A few posts on the Internet question whether Google could be providing information to Shengdan Laoren through advance data mining in exchange for some of the advance technologies.

Could the US government also be involved? Those Internet posts point to the partnership between Google and NORAD (the North American Aerospace Defense Command), a bi-national United States and Canadian organization. NORAD and Google are helping children track the journey of Jolasveinar around the world using Google Maps and Google Earth. In a possible attempt to gain patents and disrupt Google market shares, there are even rumors that Gaghant Baba’s workshop has been purchased by Bill Gates. Could a secret message exist behind the Microsoft Bing commercial about Daidi na Nollag?

Google maintains that they take user privacy very seriously. In this case, I believe them. If there is trickery, Tomten would likely be behind it. How can one trust a person who goes by so many names? And what exactly is his past? Every country provides a different story. If he is a jolly old elf, there are reports that elves have used trickery as a means to an end. Local and federal governments across the world have gift policies limiting the the value and number of gifts that can be given to government employees. Gifts can be used as bribes. One could begin to wonder if the gift bearing holiday might be a cover for a massive yearly bribery event. More troubling, attempts to trace those questioning Internet posts lead back to ISPs in Greenland. Maybe Jack Bauer is needed to get at the truth.

I am not saying Chimney John is not a jolly nice fellow. I am just not a great believer in security through obscurity. There is a great deal we don’t know about Samichlaus. As security minded people, we need to be always questioning. Video of Mone’s Google talk has been made available. View it below and judge for yourself:

Wishing you a great holiday, wherever you may be and whatever you may believe.

Remembering 9/11: The Personal Side

John Gerber — Sat, 12 Sep 2009 01:39:44 +0000

If you are just looking for information on 9/11, the Library of Congress has pulled together a nice collection of 2,313 sites in their browse collection and an additional 30,000 URLs to sites. As I was going through some of these links, I found myself getting very annoyed. I kept reading one conspiracy theory, political rant, or self promotion spiel after another. Just as I was about to call it quits, I found out that the folks over at Google are helping launch Make History, a website created by the National September 11th Memorial & Museum in partnership with design firm Local Projects. The site provides the personal side of 9/11 by helping people share photos, videos, and their stories. “They say that 9/11 was the most digitally documented event of all time,” said Alice Greenwald, director of the National September 11 Memorial & Museum. “We’re asking people everywhere to help us tell the story.”

What happened eight years ago is not just a news story. It is personal. My childhood home was within fifteen miles of the World Trade Center. I could see the skyline from my bedroom window. My family attended St. Joseph’s church where Father Mychal Judge, affectionately known as Father Mike, was our friar. Father Mike was also the Fire Department chaplain killed eight years ago following the World Trade Center attacks. He died after being struck by falling debris as he anointed a firefighter and a fallen office worker.

While this partnership is just starting out, the site is powerful and will become more moving as additional content is added. The images, videos and personal stories contributed to the Make History initiative will be time stamped and layered over the Street View imagery of Google Maps so they can be placed into a chronological timeline of the events of 9/11.

Filmmakers Steve Rosenbaum and Pamela Yoder donated rights to preserve and display 500 hours of video footage collected in The CameraPlanet Archive. In the wake of 9/11, Rosenbaum and Yoder collected, organized, catalogued and dubbed hundreds of hours of first-person video tape of the events and their aftermath.

David W. Dunlap, from the New York Times, has posted the blog entry “From the Archive: Moving Images.” Mr. Dunlap quotes Michael Shulan, the creative director of the museum, who worked with Mr. Rosenbaum and Ms. Yoder to acquire the archive, “In many of the videos, the shaking of the camera, the breathing of the cameraman, the dust getting on the lens — which would normally be edited out of network footage — are very powerful.” Mr. Shulan explains that the archive fits the museum’s mission to be “as broad-reaching as possible in the representation of the event” and to “allow everyone to have a place in the narrative.” Below are a few of the first-person videos:

A City Within a City

Robert Leiblein observational footage of the World Trade Center provides a simple and yet riveting view of the buildings and the busy pedestrian traffic years before the attacks. In particular, the video shows some of the most memorable sculptures, plazas, and at at time code 6:00 you can see people walking down the now the now historic Vesey Staircase.

I was standing in the clouds

For most folks, the view from the observation deck at the World Trade Center was as close to walking in the clouds as they might ever be. But for the team that rode the scaffold and washed windows at the Trade Center – life in the clouds was part of the job.

These Papers Came From Over There

Jen S., a newly arrived New Yorker, finds that from her street in Brooklyn the papers from the World Trade Center are filling the sky.

Should I Call Someone?

Suzanne Kaufman and her husband watch and record what they see out their window. In their words, and their silence, you can hear all of New York holding its breath.

The Storm Breaks

Mike C, a computer programmer and amateur videographer took his fish eye lens, his folding bike, and a long camera pole – and found himself in the middle of the stormcloud of smoke and dust.

Darkness Descends 1

“I have a fisheye lense, a camera pole, and a scooter” In a city full of observers, this nighttime journey through the dust covered streets of New York is a rare look at a city just after a stunning blow. Mike C’s fisheye lense and scooter journey takes from the streets to the rooftops. And once there – the camera provides a unblinking look at what remains of the World Trade Center.

Darkness Descends 2

David Goldberg, who returns to the haunted streets of Lower Manhattan. On September 11th, 2001 walk through the ashes, the smoke, the dazed rescue workers -and see what they saw through their own eyes.

Night of 9/11

On September 11th, 2001 two young filmmakers walked downtown and found themselves staring up at a smoldering wreckage of what had been The World Trade Center. Their footage, and the slow and careful view of the scene bring people inside what it might have looked like to the rescue workers who stood silently at the site.

Giving Thanks

I just wandered toward the West Side Highway… and the crowds…” So much of what we forget about 9/11 is the spontaneous outpouring of concern, appreciation, and community. In the days while smoke still billowed from the site of The World Trade Center, the West Side Highway became a improvised parade route for rescue workers and volunteers.

Lincoln Center Candlelight Vigil

College Student Jenny Tolan had her camera in hand when she discovered this candlelight vigil at Lincoln Center. The sounds, the images, and the emotion were very raw.

StoryCorps, the national oral history initiative, is also working with the museum to collect oral histories of friends, families, rescue and recovery workers and neighbors. The plan is to collect at least one recording about each of the nearly 3,000 lives lost in the attacks along with stories from survivors, rescue workers, and those most personally affected by the events of 9/11.

NPR’s Morning Edition, has been running a series on StoryCorps personal histories. The latest post “Firefighter Father Recalls Losing Sons On 9/11,” tells the story of John Vigiano Jr., a New York City firefighter and his younger brother, Joe, a NYC police detective. Both were killed in the attacks.

Additional StoryCorps stories can be found on the Public Radio Exchange (PRX), an online marketplace for distribution, review, and licensing of public radio programming. These stories include that of Monique Ferrer’s ex-husband, Michael Trinidad, who worked on the 103rd floor of the World Trade Center’s North Tower. He called after the building had been struck to tell Monique that he still loved her — and to ask her current husband to be a good father to his kids. Jessica DeRubbio, talks about losing her father, New York City firefighter David DeRubbi, when she was just twelve. Arlene Sullivan remembers her son, Tommy Sullivan, in an interview with her daughter Norene Schneider. Tommy Sullivan was a stockbroker at Harvey Young and Yurman and was having his weekly breakfast at Windows on the World. The editorial staff of PRX has also created a playlist with selected programs like “We Were on Duty,” a first-person oral history of the September 11th attack on the Pentagon.

Washington Irving, an American author, wrote, “There is a sacredness in tears. They are not the mark of weakness, but of power. They speak more eloquently than ten thousand tongues. They are messengers of overwhelming grief…and unspeakable love.” Today is a day to remember those we have lost, shed tears for the sadness that came into our lives, and most importantly always remember the love we shared.