Security Advancements at the Monastery » Visualization

Google Visualization: An Example Graphing NVD CVE Data

John Gerber — Fri, 16 Apr 2010 15:54:42 +0000

Google visualization offers graphing abilities to any number of projects. Why should security professionals care? If you are going to have to collect and present security metrics, it is best to showcase them in the very best manner possible. Andrew Jaquith in his article, “Creating meaningful information security metrics” states, “For 2010, Forrester Research expects that overall security budgets will rise less than 5 percent over 2009 –higher than in the previous year, but not by much.” Andrew goes on to point out, “smart security managers, sensing sudden vulnerability in their budgets, seek better ways to measure and prove the value of what they do every day.”

In today’s work environment there is a need to show changes, potential risks, improved performance, etc. in all areas of the company’s operations. Security professionals need to be prepared to answer the basic question, “why should the CIO or CEO care about security?” CSO Online has a great quote from the post, “From the CIO: Why You Didn’t Get the CISO Job” that challenges us to consider our views when it comes to security. The post states, “laser focus on your speciality is great in middle management. It’s what we want. One of the really hard things about jumping from management to executive is a focus on the whole of the business. It’s a rare person who manages it quickly or easily.” That is basically the problem with metrics. It is a battle between generalization to the point of uselessness and details to the point of not being understandable or collectible. At the end of the day, something needs to be done because the security industry is currently leaving upper management in the position of not understanding what is going on within their business. That is a risk that not acceptable.

Andrew’s article discusses what kind of security metrics should be used. Additional sources of information on security metrics can be found in a previous post entitled “Security Metrics.” The post provides links to wonderful sources on security metric information. You might also want to take a look at the CIS Consensus Security Metrics v1.0.0 guide, NIST Special Publication (SP) 800-55 Rev 1 “Security Metrics Guide for Information Technology Systems”, NIST IR-7564 “Directions in Security Metrics Research”, “Twenty Most Important Controls and Metrics for Effective Cyber Defense and Continuous FISMA Compliance,” and “Metrics, measures & Myths.” Once you have start gathering metrics, you will want to present them in an easy to understand format. This is where Google Visualization can help.

Today’s post walks through an example using the data from the National Institute of Standards and Technology (NIST) National Vulnerability Database (NVD ) Common Vulnerabilities and Exposures (CVE) database. The purpose is to provide a working example from which you can learn and apply to the various metrics gathered at your organization.

Data Source

A previous post, “Standardization and Interoperability in Security,” discussed how the Security Content Automation Protocol (SCAP) is an attempt to help defenders by providing a collection of XML schemas/standards that allow technical security information to be exchanged between tools. SCAP components consists of:

Common Configuration Enumeration (CCE): provide unique identifiers to system configuration issues in order to facilitate fast and accurate correlation of configuration data across multiple information sources and tools.
Common Platform Enumeration (CPE): a structured naming scheme for information technology systems, platforms, and packages.
Common Vulnerability Enumeration (CVE): a dictionary of publicly known information security vulnerabilities and exposures.
Common Vulnerability Scoring System (CVSS): a vulnerability scoring system designed to provide an open and standardized method of rating IT vulnerabilities. NIST has even provided a calculator for creating CVSS vulnerability severity scores.
eXtensible Checklist Configuration Description Format (XCCDF): a specification language for writing security checklists, benchmarks, and related kinds of documents. NIST has released the NIST Interagency Report 7275 Revision 3 “Specification for Extensible Configuration Checklist Description Format (XCCDF) Version 1.1.4.”
Open Vulnerability Assessment Language (OVAL): an information security community standard to promote open and publicly available security content, and to standardize the transfer of this information across security tools and services.

We are going to make use of the data from NVD/CVE XML feed with the Common Vulnerability Scoring System (CVSS) mappings (version 2.0). NIST documentation states:

CVSS provides an open framework for communicating the characteristics and impacts of IT vulnerabilities. Its quantitative model ensures repeatable accurate measurement while enabling users to see the underlying vulnerability characteristics that were used to generate the scores. Thus, CVSS is well suited as a standard measurement system for industries, organizations, and governments that need accurate and consistent vulnerability impact scores. Two common uses of CVSS are prioritization of vulnerability remediation activities and in calculating the severity of vulnerabilities discovered on one’s systems.

NVD provides CVSS ‘base scores‘ representing the innate characteristics of each vulnerability. ‘Temporal scores,’ which change over time due to events external to the vulnerability, are not provided though NVD does provide a CVSS score calculator. This allows an organization to add temporal data and even factor in ‘environmental scores‘ customized to reflect the impact of the vulnerability on the organization. Please refer to the CVSS standards guide and the OWASP Risk Rating Methodology concerning factors involved in estimating the severity of risks to your business.

NVD CVE XML Schema

For our example, we will be using the data feeds nvdcve-2.0-2010.xml and nvdcve-2.0-2009.xml. Examining the CVE XML 2.0 Schema, we are particularly interested in certain vulnerability and CVSS scoring information. For example, for CVE-2010-1228, we will parse and pull the following kind of information:

 id="CVE-2010-1228">
  CVE-2010-1228
  2010-04-01T18:30:00.453-04:00
  
  2010-04-05T00:00:00.000-04:00
  
  
    
      10.0
      NETWORK
      LOW
      NONE
      COMPLETE
      COMPLETE
      COMPLETE
      http://nvd.nist.gov

Using Perl to Retrieve the CVE File

Initially we will read the nvdcve-2.0-2010.xml and nvdcve-2.0-2009.xml files. If we start retrieving the file regularly, we would want to change this to nvdcve-2.0-recent.xml. Of course, previous years can also be read in to provide a longer perspective on vulnerabilities. A simple example of a Perl subroutine to read the NVD CVE file and save it locally would be:

sub readpage {
   my($url,$nvd_file) = @_;
   my($proxy) = "http://your-proxy-server:proxy-port";
   my $ua = new LWP::UserAgent;
   $ua->proxy(http  => $proxy);
   $ua->proxy(ftp => $proxy);
   $ua->proxy(https => $proxy);
   # Go out and retrieve page
   my $req = new HTTP::Request('GET', $url);
   my $res = $ua->request($req);
   my $pjstatus = 1;
   # Check if the requested webpage is there and return results
   if ($res->is_success) { # Request successful
       open(OUTFILE,">$nvd_file") || ($pjstatus = 0);
       if ($pjstatus) {
          print OUTFILE $res->content;
       }
       close(OUTFILE);
   }
   else {
      $pjstatus = 0;
   }
   return($pjstatus);
}

Please substitute “http://your-proxy-server:proxy-port” with your site’s proxy server and port, if applicable.

Creating a MYSQL Table to Hold the Data

There is a great deal of information in the NVD CVE file. You will need to determine what information your organization will be interested in storing and graphing. For better or worse, folks have come to expect vulnerabilities to have a “Low,” “Medium,” or “High” score. NIST has stated concerning the NVD Vulnerability Severity Ratings:

NVD provides severity rankings of “Low,” “Medium,” and “High” in addition to the numeric CVSS scores but these qualitative rankings are simply mapped from the numeric CVSS scores:
1. Vulnerabilities are labeled “Low” severity if they have a CVSS base score of 0.0-3.9.
2. Vulnerabilities will be labeled “Medium” severity if they have a base CVSS score of 4.0-6.9.
3. Vulnerabilities will be labeled “High” severity if they have a CVSS base score of 7.0-10.0.

While preferring quantitative over qualitative values, for this example I would like to create a stacked column chart. We will add a severity column which is based on the CVSS score. An example table follows:

CREATE DATABASE vulnerabilities;
USE vulnerabilities;
DROP TABLE IF EXISTS `nvdcve`;
CREATE TABLE `nvdcve` (
  `cve_id` varchar(13) NOT NULL,
  `published` datetime default NULL,
  `modified` datetime default NULL,
  `score` DECIMAL(5,2) default '0.0',
  `severity` varchar(6) default 'LOW',
  `vector` varchar(25) default NULL,
  `complexity` varchar(25) default NULL,
  `authentication` varchar(25) default NULL,
  `confidentiality` varchar(25) default 'NONE',
  `integrity` varchar(25) default 'NONE',
  `availability` varchar(25) default 'NONE',
  `summary` varchar(512) default NULL,
  PRIMARY KEY  (`cve_id`),
  INDEX (score),
  INDEX (vector)
)

Using Perl Populating the Database

Populating the database table is simply a matter of reading the file and adding the entries to the table. An example Perl subroutine follows:

sub readxml {
   my($nvd_file, $dbh) = @_;
   my $parser = XML::LibXML-> new();
   my $doc    = $parser-> parse_file($nvd_file);
   my $xc     = XML::LibXML::XPathContext-> new( $doc->documentElement() );
   $xc-> registerNs(
      def  => 'http://scap.nist.gov/schema/feed/vulnerability/2.0' );
   $xc-> registerNs(
     vuln => 'http://scap.nist.gov/schema/vulnerability/0.4' );
   $xc-> registerNs( cvss => 'http://scap.nist.gov/schema/cvss-v2/0.2' );
   for my $entry ($xc-> findnodes("/def:nvd/def:entry")) {
      my $cve = $xc-> find('vuln:cve-id',$entry);
      my $published = $xc-> find('vuln:published-datetime', $entry);
      my $modified = $xc-> find('vuln:last-modified-datetime', $entry);
      my $summary = $xc-> find('vuln:summary', $entry);
      my $skip = 0;
      my ($metrics) = $xc-> findnodes('vuln:cvss/cvss:base_metrics', $entry) or ($skip = 1);
      if (! $skip) {
         my $score = $xc-> find('cvss:score', $metrics);
         my $vector = $xc-> find('cvss:access-vector', $metrics);
         my $complexity = $xc-> find('cvss:access-complexity', $metrics);
         my $authentication = $xc-> find('cvss:authentication', $metrics);
         my $confidentiality =
            $xc-> find('cvss:confidentiality-impact', $metrics);
         my $integrity = $xc-> find('cvss:integrity-impact', $metrics);
         my $availability = $xc-> find('cvss:availability-impact', $metrics);
         my $severity = "LOW";
         if (int($score) >= 7) {
            $severity = "HIGH";
         }
         elsif (int($score) >= 4) {
            $severity = "MEDIUM";
         }
         my $sql = qq{ SELECT count(*) FROM nvdcve WHERE cve_id=? };
         my $sth = $dbh->prepare( $sql );
         my $rc = $sth->execute($cve);
         if ( $rc) {
            my($exist) = $sth->fetchrow_array();
            if (! $exist) {
                $sql = qq{ INSERT INTO nvdcve SET cve_id=?,
published=?, modified=?, score=?, severity=?, vector=?, complexity=?,
authentication=?, confidentiality=?, integrity=?,availability=?, summary=? };
               $sth = $dbh->prepare( $sql );
               $rc = $sth->execute($cve,$published,$modified,$score,
$severity,$vector,$complexity,$authentication,
$confidentiality,$integrity,$availability,$summary);
            }
         }
      }
   }
}

The Perl Program to Pull It All Together

The above subroutines use the Perl modules LWP::UserAgent, XML::LibXML, XML::LibXML::XPathContext, and DBI. A sample Perl program that calls the above subroutines to pull down the NVD CVE data and load it into a MySQL table would be:

#!/usr/local/bin/perl -w
use LWP::UserAgent;
use XML::LibXML;
use XML::LibXML::XPathContext;
use DBI;
BEGIN{push @INC, "/home/jgerber/projects/nvd/perl"}
use nvdsubs qw($db_host $db $mysql_user $mysql_passwd $mysql.sock
readpage readxml );
# Main
my $datadir = "/home/johngerber/projects/nvd/data";
my @timeData = localtime(time);
my $year = 1900 + $timeData[5];
my $prev_year = 1900 + $timeData[5] - 1;
my $url = "http://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-" .
    $year . ".xml";
my $prev_url = "http://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-" .
    $prev_year . ".xml";
my $nvd_file = $datadir  . "/nvdcve-". $year . ".xml";
my $prev_nvd_file = $datadir  . "/nvdcve-". $prev_year . ".xml";
$db = "vulnerabilities";
local($dbh) = DBI->connect("DBI:mysql:mysql_socket=$mysql.sock;$db:$db_host",
$mysql_user, $mysql_passwd) || die "ERROR: Connecting: $DBI::errstr\n";
my ($pjstatus) = &readpage($prev_url,$prev_nvd_file);
if ($pjstatus) {
   &readxml($prev_nvd_file,$dbh);
}
$pjstatus = &readpage($url,$nvd_file);
if ($pjstatus) {
   &readxml($nvd_file,$dbh);
}
exit;

The nvdsubs.pm file will not be included in this post. The subroutines are defined and the only pieces missing are the MySQL database username and password. You don’t need mine. Add your own. At this point, we have everything we need to finally use Google Visualization to create a graph.

Google Visualization

We are going to create a Perl program that will read our MySQL nvdcve table and generate the JavaScript that will render our charts on the client’s browser. First, we want to define the JavaScript we want to produce. Just to alleviate some concerns, with Google Visualization your data is only shared between your server and the client connecting. This is unlike Google Charts where your data is sent to Google where it is made into a chart and the result is sent back. Google states concerning the logging of chart data (via Google Charts), “The chart data included in the HTTP request is saved in temporary logs for no longer than two weeks for internal testing and debugging purposes.” Every example in the Google Visualization Gallery will state the data policy. For Google Charts, stated at the bottom of the page for each gadget description the data policy:

While Google Visualization gadgets will have the following stated data policy:

Loading Google Libraries

The first thing the JavaScript needs to do is load the required libraries. This is accomplished with the lines:

Area Chart and Table

In this example we are going to create an column chart. In a later section, “Other Charting Options” (see below) we define different Google Visualization charting options.

JavaScript code for a sample column chart would be:

    <script type='text/javascript'>
      google.load('visualization', '1', {packages:['columnchart']});
      google.setOnLoadCallback(drawChart);
      function drawChart() {
        var data = new google.visualization.DataTable();
        data.addColumn('date', 'Date');
        data.addColumn('number', 'High');
        data.addColumn('number', 'Medium');
        data.addColumn('number', 'Low');
        data.addRows([
           [new Date(2009, 0, 30),92,97,3],
           [new Date(2009, 1, 27),168,142,25],
           [new Date(2009, 2, 31),141,165,9],
           [new Date(2009, 3, 30),132,203,12],
           [new Date(2009, 4, 29),158,153,8],
           [new Date(2009, 5, 30),200,199,22],
           [new Date(2009, 6, 31),190,195,11],
           [new Date(2009, 7, 31),127,139,14],
           [new Date(2009, 8, 30),233,208,14],
           [new Date(2009, 9, 30),163,167,18],
           [new Date(2009, 10, 30),129,172,8],
           [new Date(2009, 11, 31),200,211,19],
           [new Date(2010, 0, 29),157,139,14],
           [new Date(2010, 1, 26),137,143,12],
           [new Date(2010, 2, 31),252,242,18],
           [new Date(2010, 3, 13),92,118,17]
        ]);
        var chart = new google.visualization.ColumnChart(document.getElementById('s4graph'));
        chart.draw(data, {displayAnnotations:true, is3D: true, isStacked: true, min: 0,
          allowHtml: true, colors:[{color:'#E41B17', darker:'#C11B17'}, {color:'#FFA500', darker:'#E56717'}, {color:'#FFE87C', darker:'#C8B560'}]});
      }
    script>

The resulting image would be the following column chart:

Rendering the Table

When providing qualitative results, I like to back them up with more accurate numeric values. Let us include a table with links to the CVSS scores for each vulnerability.

    <script type='text/javascript'>
      google.load('visualization', '1', {packages:['table']});
      google.setOnLoadCallback(drawChart);
      function drawChart() {
        var data2 = new google.visualization.DataTable();
        data2.addColumn('date', 'Date');
        data2.addColumn('number', 'High');
        data2.addColumn('number', 'Medium');
        data2.addColumn('number', 'Low');
        data2.addRows([
           [{v:new Date(2009, 0, 30),
              f:'2009-01-30'}, 92,97,3],
           [{v:new Date(2009, 1, 27),
              f:'2009-02-27'}, 168,142,25],
           [{v:new Date(2009, 2, 31),
              f:'2009-03-31'}, 141,165,9],
           [{v:new Date(2009, 3, 30),
              f:'2009-04-30'}, 132,203,12],
           [{v:new Date(2009, 4, 29),
              f:'2009-05-29'}, 158,153,8],
           [{v:new Date(2009, 5, 30),
              f:'2009-06-30'}, 200,199,22],
           [{v:new Date(2009, 6, 31),
              f:'2009-07-31'}, 190,195,11],
           [{v:new Date(2009, 7, 31),
              f:'2009-08-31'}, 127,139,14],
           [{v:new Date(2009, 8, 30),
              f:'2009-09-30'}, 233,208,14],
           [{v:new Date(2009, 9, 30),
              f:'2009-10-30'}, 163,167,18],
           [{v:new Date(2009, 10, 30),
              f:'2009-11-30'}, 129,172,8],
           [{v:new Date(2009, 11, 31),
              f:'2009-12-31'}, 200,211,19],
           [{v:new Date(2010, 0, 29),
              f:'2010-01-29'}, 157,139,14],
           [{v:new Date(2010, 1, 26),
              f:'2010-02-26'}, 137,143,12],
           [{v:new Date(2010, 2, 31),
              f:'2010-03-31'}, 252,242,18],
           [{v:new Date(2010, 3, 13),
              f:'2010-04-13'}, 92,118,17],
        ]);
        var table = new google.visualization.Table(document.getElementById('s4graph_tab'));
        table.draw(data2, {showRowNumber: true, sortAscending: false, sortColumn: 0, allowHtml: true});
      }
    script>

The JavaScript code assumes there is a PHP program called cvealerts.php under the /nvd directory on your web server. Adjust to your environment. A sample PHP program that could be used for cvealerts.php is provided below. The resulting table chart would look like:

Handling Events: Interactions Between Graphs

We now have two different types of graphs representing the same data. We want to add interaction between the graphs so the viewer can see the relationship. With tables rows are selected when the user clicks, which correspond to the whole column of the stacked column chart. It is not a perfect fit, but it does demonstrate nicely use of adding interactions.

        // Set a 'select' event listener for the table.
        // When the table is selected,
        // we set the selection on the line graph.
        google.visualization.events.addListener(table, 'select', function() {
          chart.setSelection([{row: table.getSelection()[0].row, column: 1}]);
         });
        // Set a 'select' event listener for the graph.
        // When the graph is selected,
        // we set the selection on the table.
        google.visualization.events.addListener(chart, 'select', function() {
           table.setSelection([{row: chart.getSelection()[0].row}]);
        });

Providing Detailed Information

When the table chart link is clicked, we would like to provide some detailed information about the vulnerability. For this example, we will do this with a simple PHP program placed in the /nvd directory on the web server. The program is called cvealerts.php.


session_start();
function db_connect($table) {
   $result = mysql_pconnect(":", "", "");
   if (!$result) return false;
   if (!mysql_select_db($table)) return false;
   return $result;
}
function do_html_header($title,$checkuser,$logpage) {
?>
  <html> <head> <title>$title?>title>head>
  <body bgcolor="#FFFFFF">

}
function do_html_footer() {
?>
<table>
<tr><td ALIGN=CENTER NOWRAP WIDTH="590">font>
<font face="Verdana, Arial, Helvetica" size=-2>Notice to Users: Use
of this system constitutes consent to security monitoring and testing.
<br>All activity is logged with your host name and IP address.font>
td>tr>
table>
body>
 html>

}
// Main
$dates= array();
$stringlist = "";
if (isset($_GET['date'])) {
    $passdates = explode(",",$_GET['date']);
    for ($index=0; $index<count($passdates); $index++) {
       array_push($dates, $passdates[$index]);
       $stringlist .= $passdates[$index] . " ";
    }
}
else {
  print("Confusion over how you arrived at this page.\n");
  exit;
}
$stringlist = preg_replace("/ $/", "",$stringlist);
do_html_header("Review NVD CVE Announcements for Month Ending $stringlist",1,1);
$nvd_host = "http://web.nvd.nist.gov/view/vuln/detail?vulnId=";
$conn = db_connect("vulnerabilities");
if (!$conn)
   logit("Could not connect to database vulnerabilities - please try later.\n",1);
for ($index=0; $index<count($dates); $index++) {
   $rule = $dates[$index];
   $sql = "SELECT cve_id,score,published,vector,severity,complexity,left(summary,50)
    FROM vulnerabilities.nvdcve
      WHERE date_format(published,'%Y-%m')='$rule'
       ORDER BY (score+0)";
   $result = mysql_query($sql,$conn);
   if (!$result)
       logit("Problem with $sql\n",1);
   print("
\n");for($count=1;list($cve_id,$score,$date,$vector,$severity,$complexity,$shortsum)=mysql_fetch_array($result, MYSQL_NUM);++$count){?><tr><td CLASS="plfieldhdrleft" WIDTH="20%" BGCOLOR='#F0F5FF'>  print("$cve_id"); ?>
      td>
      <td CLASS="plfieldhdrleft" BGCOLOR='#F9FCFF'>
        print($score); ?>
      td>
      <td CLASS="plfieldhdrleft" BGCOLOR='#F0F5FF'>
        print($date); ?>
      td>
      <td CLASS="plfieldhdrleft" BGCOLOR='#F9FCFF'>
        print($vector); ?>
      td>
      <td CLASS="plfieldhdrleft" BGCOLOR='#F0F5FF'>
        print($severity); ?>
      td>
      <td CLASS="plfieldhdrleft" BGCOLOR='#F9FCFF'>
        print($complexity); ?>
      td>
      <td CLASS="plfieldhdrleft" BGCOLOR='#F0F5FF'>
        print($shortsum); ?>
      td>
      tr>

   }
}
print("
Bulletin 
Impact 
Date 
Vector 
Severity 
Complexity 
Short Summary
            
        

      
      
");
do_html_footer();

The PHP program would generate a HTML table displaying the NVD CVE alerts for that month. The table would look like:

When the CVE link is clicked on, the user is taken to the NIST NVD site where additional information is available.

Using Perl to Create the JavaScript

The Perl code is rather simple now that we have the MySQL tables defined and the JavaScript we want to generate. Much of the code consists of the JavaScript listed above.

#!/usr/local/bin/perl -w
use DBI;
use Time::Local;
use POSIX qw(strftime);
use LWP::UserAgent;
BEGIN{push @INC, "/home/jgerber/projects/nvd/perl"}
use ornl_feds qw($db_host $db $mysql_user $mysql_passwd );
sub slide_nvd_alerts {
  my($min_date,$graph_name,$web_link,$dbh) = @_;
  my $slide = "";
  my $slide_head = qq!
    
!;
   if ($min_date eq "") {
      my $sql2 = qq{ SELECT min(published) FROM vulnerabilities.nvdcve };
      my $sth2 = $dbh->prepare( $sql2 );
      my $rc2 = $sth2->execute();
      if ($rc2) {
         $min_date = $sth2->fetchrow_array();
      }
   }
   my $table_data = "";
   my $graph_data = "";
   my $sql2 = qq{ select date_format(published,'%Y-%m'),severity,count(severity)
      FROM vulnerabilities.nvdcve where published >= ? group by date_format(published,'%Y-%m'),severity };
   my $sth2 = $dbh->prepare( $sql2 );
   my $rc2 = $sth2->execute($min_date);
   if ($rc2) {
      my ($change,$virgin,$ht,$mt,$lt,$mmax_date) = ("",1,0,0,0,"");
      while (my($snapshot_date, $severity, $pcount) = $sth2->fetchrow_array()) {
         my $sql3 = qq{ SELECT max(published) FROM vulnerabilities.nvdcve where
date_format(published,'%Y-%m')=? };
         my $sth3 = $dbh->prepare( $sql3 );
         my $rc3 = $sth3->execute($snapshot_date);
         $max_date =  $sth3->fetchrow_array();
         $max_date =~ s/ \S+$//;
         if ($change ne $snapshot_date) {
            if (! $virgin) {
                my($year,$month,$day) = split("-",$mmax_date);
                my $mmonth = $month;
                $month--;
                $graph_data .= qq!           [new Date($year, $month, $day),$ht,$mt,$lt],
!;
                $table_data .= qq!           [{v:new Date($year, $month, $day),
              f:'$mmax_date'}, $ht,$mt,$lt],
!;
                ($ht,$mt,$lt) = (0,0,0);
             }
             $change = $snapshot_date;
          }
          if ($severity eq "HIGH") { $ht = $pcount; }
          elsif ($severity eq "MEDIUM") { $mt = $pcount; }
          elsif ($severity eq "LOW") { $lt = $pcount; }
          if ($mmax_date eq "") { $mmax_date = $max_date; }
          if ($mmax_date lt $max_date) { $mmax_date = $max_date; }
          $virgin = 0;
      }
      my($year,$month,$day) = split("-",$mmax_date);
      my $mmonth = $month;
      $month--;
      $graph_data .= qq!           [new Date($year, $month, $day),$ht,$mt,$lt]
!;
     $table_data .= qq!           [{v:new Date($year, $month, $day),
              f:'$mmax_date'}, $ht,$mt,$lt],
!;
   }
   $table_data .= "        ]);\n";
   $graph_data .= "        ]);\n";
   $slide = $slide_head .  $graph_data . $slide_head_table . $table_data . $slide_tail;
   return($slide);
}
sub slide_body {
  my($graph_name,$title,$style) = @_;
  my $table_name = $graph_name . "_tab";
  my $table_text = "div id=\"$table_name\"";
  if ($style ne "") {
     $table_text .= " style=\'$style\'";
  }
  my $slide2 = "$title
\n";
  my $itext = "div id=\"$graph_name\"";
  if ($style ne "") {
     $itext .= " style=\'$style\'";
  }
  $slide2 .= qq{
    
    <$itext>
    <$table_text>
       
    
    
  };
  return($slide2);
}
# Main
my $web_link = "/nvd";
my $results_dir = "/data/html" . $web_link;
my $result_file = $results_dir . "/nvdcve_stats.html";
my $debug = 1;
my $db = "vulnerabilities";
local($dbh) = DBI->connect("DBI:mysql:$db:$db_host", $mysql_user, $mysql_passwd) ||
   die "ERROR: Connecting: $DBI::errstr\n";
$slides_data .= &slide_body("s4graph","NVD CVE Alerts","width:700px; height:400px;");
$slides_head .= &slide_nvd_alerts("","s4graph",$web_link,$dbh);
open(OUTFILE,">$result_file");
print OUTFILE "\nNVD CVE Statistics\n";
print OUTFILE "\n";
print OUTFILE $slides_head;
print OUTFILE "\n\n";
print OUTFILE $slides_data;
print OUTFILE "\n";
close(OUTFILE);
exit;

Other Charting Options

Google, Google users, and other companies have shared some JavaScript visualizations built on the Google Visualization API to help you get started. Below are some example:

	Annotated Time Line (GWT Integrated) An animated time series chart. By: Google
	Area Chart (GWT Integrated) Interactive area chart. By: Google
	Bar Chart (GWT Integrated) Interactive bar chart. By: Google
	Bars of Stuff Fun bar charts using images of trains, chocolate, worms, and more. By: The visapi project
	Bio Heat Map Heatmaps are a useful way to visualize matricies of data. Scientists often use green-black-red heatmaps to visualize gene expression data from microarrays. This visualization supports both three color heatmaps (ex: green to black to red) and two color heatmaps (ex: white to yellow). By: Institute for Systems Biology
	Column Chart (GWT Integrated) Interactive column chart. By: Google
	Drastic Treemap A dynamic treemap in Flash. By: DrasticData
	Dygraphs The dygraphs JavaScript library produces interactive, zoomable charts of time series. By: Dan Vanderkam
	Filters A Visualization that acts as a control over other visualizations. It is rendered within the browser using HTML. This visualization offers the ability to select some criteria to filter the DataTable used by the controlled visualizations. By: Institute for Systems Biology
	Gauge (GWT Integrated) Each numeric value is displayed as a gauge. By: Google
	Geo Map (GWT Integrated) A map of a country, continent, or region map, with colors and values assigned to specific regions. Values are displayed as a color scale, and you can specify optional hovertext for regions. By: Google
	Intensity Map (GWT Integrated) An intensity map that highlights regions or countries based on relative values. By: Google
	Line Chart (GWT Integrated) Interactive line chart. By: Google
	Magic-Table The Magic Table is a JavaScript library that allows you to see more in your data by applying some simple visual techniques to transform a table. The table is displayed in the browser by the canvas element. Internet Explorer is not supported. By: Greg Ross
	Map (GWT Integrated) An interactive map that uses the Google Maps API. By: Google
	Motion Chart (GWT Integrated) Motion Chart: A dynamic flash based chart to explore several indicators over time. Required columns: bubble name, time and 2 columns of numeric values. Optional columns: Numeric values or categories. By: Google
	Organizational Chart A GWT Integrated simple organizational chart. By: Google
	Parallel Coordinates Chart Parallel Coordinates is a method of visualizing multivariate data. An n-dimensional space is represented as n parallel lines. Works for browsers based on Gecko or Presto (does not work in IE). This is written in Javascript, no Flash required. By: Sri Harsha Allamraju
	Pie Chart (GWT Integrated) Interactive pie chart. By: Google
	Piles of Money Column chart made of of money bills. By: The visapi project
	Scatter Chart (GWT Integrated) Interactive scatter chart. By: Google
	Table (GWT Integrated) A highly customizable table with sorting, paging and selection capabilities. By: Google
	TermCloud A list of terms, where the size and color of each word is determined by a specified frequency value (typically the number of times it appears in some text). By: The visapi project
	Thematic Mapping API Enables visualization of data in Google Earth or other geobrowsers through the use of the Google Visualization API and KML. By: Thematicmapping.org
	WordCloud Displays all words in text with size and color based on the number of time each word appears. By: The visapi project

Additional Information

Below is the talk that Itai Raz, the lead engineer for the Visualization API product at Google, gave at Google I/O 2009 titled “Using the Visualization API with GWT:”

Additional Possibilities

The work above is meant only to serve as a starting point. There is a great deal more information to expand upon. For example, we began this post pulling some information from the XML schema for CVE-2010-1228. One field we did not pull out from the XML file is:

The Common Weakness Enumeration (CWE) represents vulnerability types and NIST provides a CWE Cross Section Mapped into by NVD table. In the above example, we see an entry:

Name	CWE-ID	Description
Race Conditions	CWE-362	The state of a resource can change between the time the resource is checked to when it is accessed.

Clicking on the link will take us to the MITRE site that provides a great deal more information on CWE entries. It is easy enough to expand on the above program to harvest this information for a richer information database.

Another possibility is to expand the above program to pull additional information on the CVE entry. In additional to the data in the NVD CVE XML file, we could pull information from the NVD site. Using CVE-2010-1228 as an example, we could have the program pull down the page:

http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2010-1228

Notice the line:

CVSS v2 Base Score:10.0 (HIGH) (AV:N/AC:L/Au:N/C:C/I:C/A:C) (legend)

The (AV:N/AC:L/Au:N/C:C/I:C/A:C) provides values that were used in determining the base score. If you follow the link, you will see the values used in the calculations:

CVSS Base Score: 10
- Impact Subscore: 10
- Exploitability Subscore: 10
CVSS Temporal Score: Undefined
CVSS Environmental Score: Undefined
Overall CVSS Score: 10

NVD has made available the equations used in calculating the CVSS base score, temporal score, and environmental score.

Three other pieces of information that might provide interesting groupings are:

Access Complexity: Low **NOTE: Access Complexity scored Low due to insufficient information
Authentication: Not required to exploit
Impact Type: Allows unauthorized disclosure of information; Allows unauthorized modification; Allows disruption of service

What information is of interest and how it is used will be dependent on your organization. There is a great deal of information available and many directions you start examining.

Final Thoughts

I am often reminded of the old phrase, “Trust us, we are from the government.” No one really trusts anyone, especially when it comes to matters they do not understand. Just because you are from the security group at your organization, is that reason enough for the CEO to give you unlimited money and authority to do what you see fit? Of course not. While management might trust you, they may not believe that you are capable of seeing the big picture. That is after all their job.

Another great old saying is that “the devil is in the details.” Those details will likely fall in the security domain. In organization across the planet there is a tug of war between the details and the big picture with multiple groups adding in their opinions and views. You need to make the details understandable to your higher management to effectively argue your view. Finding effective metrics and finding clear representation is essential in today’s business. Google Visualization can be a useful tool in accomplishing this task.

Security Visualization: FODAVA

John Gerber — Mon, 26 Oct 2009 04:15:51 +0000

Interpreting data is at the heart of security. Transforming large, often streaming data sets, e-mails, images, numbers and sounds into a form that better supports analytic reasoning continue to become more important as organization have to deal with an ever increasing amount of varied data. While there are many folks doing fascinating work, today’s posts will highlight some of the work being done by researchers as part of the NSF’s Foundations of Data Analysis and Visual Analytics (FODAVA ).

Background

In March 2004, DHS established the National Visualization and Analytics Center (NVAC) . The center is led by the Department of Energy’s (DOE) Pacific Northwest National Laboratory (PNNL) in Richland, Washington. NVAC is tasked with providing “scientific guidance and coordination for the research and development of new tools and methods that Homeland Security has identified as required for managing, visually representing, and analyzing enormous amounts of diverse data and information.”

Penn State develops visualization tools designed to extract and safely store pertinent information, such as place and time, from a variety of data formats which can help analysts anticipate, prevent and respond to major events. Purdue and IUSM focus is on three homeland security areas – intelligence analysis; emergency planning and response; and healthcare monitoring and management. Stanford performs research on network traffic analysis for intrusion detection; cognitive and perceptual principles supporting reasoning with space and time; and methods to support exploratory analysis of graphs in relational databases. UNC Charlotte and Georgia Tech develop techniques and tools to assist homeland security analysts and then combine the tools in an artificial analytic reasoning system. UW established a Pacific Rim regional center, which includes experts from UW, British Columbia, Australia, New Zealand and Hawaii working together to conduct their analysis, a process known as “collaborative visual analytics.”

FODAVA

In July 2007, DHS announced a partnership between NVAC and National Science Foundation (NSF) to conduct a joint research program in data and visual analytics. The Georgia Institute of Technology was tasked with leading and coordinating the initiative. The NSF’s project, known as the FODAVA, was established to build a community of researchers consisting of Cornell University, Duke University, Northwestern University, Standford University, University of California, Davis, University of Illinois at Chicago, University of Illinois at Urbana-Champaign, University of Michigan. Research is performed in massive data analysis and visual analytics through such areas as machine learning, numeric and geometric computing, optimization, computation statistics, and information visualization.

Sample Work

For an understanding of the type of research being done, take a moment to examine one sample project, Jigsaw. The project is intended to help analysts better assess, analyze and make sense of large document collections. To quote the Jigsaw site description:

Jigsaw is “a visual analytics system to help analysts better assess, analyze, and make sense of such document collections. Our specific objective is to help analysts reach more timely and accurate understandings of the larger stories embedded throughout textual reports. Jigsaw provides a collection of visualizations that each portray different aspects of the documents. We particularly focus on presenting the identifiable important entities (people, places, organizations, etc.) and their direct or indirect connections. Textual processing extracts the important entities from the documents and then the visualizations help an analyst to explore the relationships and connections among the entities.

There is even a tutorial-style video that shows a number of short segments about the different views in Jigsaw. mov (11 MB)

A few additional projects developed by the Information Interfaces Group, an HCI research group in the GVU Center at Georgia Tech, include:


	Jigsaw Using visualization and visual analytics to help analysis and sensemaking on document collections.
	Information Visualization and Visual Analytics Helping people understand and analyze data through interactive visualization techniques and systems.
	Imprint Empowering workers to use visualizations of printer data as a basis for conversation and reflection.
	SellTrend Enabling real-time awareness and exploratory analysis of temporal, categorical event transactions.
	The Buzz Supporting end-user mashup creation and content aggregation onto photo and text collages.
	InfoCanvas Developing Information Art: Virtual paintings that peripherally convey information to people.
	FundExplorer Aiding equity investors with mutual fund portfolio diversification through the use of Context Treemaps.
	Dust & Magnet Assisting people understand multivariate data sets using a magnet metaphor-based visualization.
	Sports Visualization Promoting advanced statistical analysis in sports through the use of information visualization.
	SunBurst Developing circular, space-filling visualizations for depicting information hierarchies and trees.
	Software Visualization Helping people understand software through visualization of programs, data structures, algorithms, and executions.

Below are some of the presentations done for the FODAVA Distinguished Lecture Series:

Alan Turner (US Government), "Mathematical Foundations as a Key Enabler of Agile Human Performance in Visual Analytics Environments". [pdf]
William S. Cleveland (Shanti S. Gupta Professor of Statistics and Professor of Computer Science, Purdue University), "The Disappearing Second Derivative of Quadratics: Perceptual, Mathematical, and Statistical Properties of Judging Dependence on Visual Displays". [pdf]
Joseph Kielman (Science and Technology Directorate, Department of Homeland Security), "Visual Analytics – Past, Present, and Future". [pdf]
Alexey Chervonenkis (Russian Academy of Science and Royal Holloway University of London), "Model Complexity Optimization".
Vladimir Vapnik (NEC Laboratories, Columbia University, and Royal Holloway University of London), "Learning with Teacher: Learning Using Hidden Information". [pdf]

Final Thoughts

Visualization is not the solution for every security problem. The work being done by researchers can often seem impractical to those in operations, where unfortunately few of us have high performance supercomputers at our disposal. Still, it is similar to why car manufacturers will take part in building race cars. When you push the envelop, the knowledge learned just might be applicable to those fighting the good fight in organizations across the planet. Watching the work being done at research projects like FODAVA provide a view of interesting possibilities for the future. In combination with existing tools, visualization tools promises to help explore data, discover insights, and provide a way to effectively communicate results. Visualization is a most interesting field offering intriguing possibilities.

TOTEM: Threat Observation, Tracking, and Evaluation Model

John Gerber — Sun, 07 Jun 2009 01:29:06 +0000

This week I had the pleasure of presenting two talks at the National Laboratories Information Technology (NLIT) 2009 Summit held in Oak Ridge, TN. Everyone involved was great and I had a fun time. Since the presentations have been posted to the NLIT site, I am free to post now.

The original slides made heavy use of the Microsoft PowerPoint animation feature. Unfortunately, SlideShare does not currently support animation. You can download the presentation and the animations will work, but I ended up modifying the slides so they are more viewable online. SlideBoom will keep the animation, but it does it by creating a video of the presentation. I decided to stick with SlideShare and spare you the resulting nine minute video. While I should add audio and make a SlideCast, this post might never be completed if I wait until I have time to create a really nice web presentation.

Merriam-Webster defines a totem as any supposed entity that watches over or assists a group of people, such as a family, clan, or tribe. In this presentation I focused on how TOTEM assists in watching over and evaluating the threat an IP represents. The idea behind TOTEM is simple: compare threat information from sources such as watchlists (DShield, Emerging Threats, SenderBase, etc.) to activities with the organization (IDS/IPS, flow logs, etc.) and other locations (SANS ISC, DOE federated model, etc.). As new threat information and activity sources are added, a better evaluation can be rendered.

TOTEM: Threat Observation, Tracking, and Evaluation Model

View more presentations by John Gerber.

The purpose of this presentation has been to share the basic ideas behind TOTEM with the hope that others may provide helpful insight. So far I have not disappointed. I wanted to thank everyone for I have received some very intriguing ideas.

Setting Up Prefuse Flare Under Unix

John Gerber — Sun, 31 May 2009 19:59:08 +0000

After working through the steps involved in “Setting Up Axiis for Security Visualization,” I became interested in taking a look at Prefuse Flare. Flare is a visualization toolkit written for Flash in ActionScript. It is a creation of the UC Berkeley Visualization Lab. Sasha Dzeletovic, a person who has worked with Flare, has done a very nice post comparing the two, “Flare vs. Axiis.” Both Tom Gonzalez, one of the co-founders of Axiis, and Sasha seem to agree that one big difference between Flare and Axiis is that, to quote Tom, “Axiis allows you to describe complex layout algorithms in markup in a more concise, and I like to think, intuitive manner.”

The Flare tutorial provides some useful links to resources intended to help with ActionScript 3:

Adobe provides an Overview of AS3, with links to additional resources.
Essential ActionScript 3 by Colin Moock from O'Reilly publishing is a great book to help you get started. You can access it online here (some institutions, such as universities, provide access for free).
The Adobe Flex API Reference is invaluable for understanding the different classes and methods available. We will be focused only on the classes in the flash.* packages.

Installation is very similar to Axiis. Install Flex SDK and/or Adobe Flex Builder. Installing Flex Builder is straight forward, but regrettably development for Adobe Flex Builder for Linux has been put on hold. We will be using Flex SDK. We will want Apache Ant. Flare is package with a build.xml file, so all we need to do is change the first few lines to point to out Flex SDK installation, and we are ready to use ant to compile the libraries.

While I could duplicate the steps for installing Flex SDK and Apache ant, it is simpler to point to “Setting Up Axiis for Security Visualization.” Once you complete those steps, you are ready to install Flare. Continuing from my previous post, let us make sure the required software is installed and our environmental variables are properly set:

root# declare -x JAVA_HOME="/usr/java/latest"
root# declare -x PATH="${JAVA_HOME}/bin:${PATH}"
root# java -version
java version "1.6.0_13"
Java(TM) SE Runtime Environment (build 1.6.0_13-b03)
Java HotSpot(TM) Server VM (build 11.3-b02, mixed mode)

root# declare -x ANT_HOME="/work/software/ant"
# ant -v
Apache Ant version 1.7.1 compiled on June 27 2008
Buildfile: build.xml does not exist!
Build failed

root# declare -x PATH="${PATH}:${ANT_HOME}/bin"
root# export PATH=/work/software/flex/bin:${PATH}
root# mxmlc --version
Version 3.3.0 build 4852

At this point, we are interested in downloading the Flare file and unzipping it.

root# cd /work/software
/work/software root# mkdir flare
/work/software root# cd flare
/work/software/flare root# wget http://flare.prefuse.org/download
/work/software/flare root# unzip prefuse.flare-alpha-20090124.zip
/work/software/flare root# vi build.xml

Now we need to modify the first few lines of the build.xml file that is part of the Flare zipped file. We will want to adjust FLEX_HOME, asdoc.

At this point, build all Flare targets with ant.

/work/software/flare root# ant all
/work/software/flare root# ls build
DependencyGraph.swf  flare.swc        JobVoyager.swf
flare.demos.swf      flare.tests.swf  PackageMap.swf

You can now use your favorite browser to open file /work/software/flare/build/flare.demos.swf. The program will call the other Flash executables.

The Flare tutorial provides a great starting point in working with Flare. The post “Flex and Flare Installation (K)Ubuntu 8.10” will take you through building some of the examples from the tutorial.

Check out the different examples and start experimenting. Two other projects that caught my attention and might be of interest:

JuiceKit, which is for building graphically rich and interactive information displays. JuiceKit is particularly interesting because it is based on Flare.
BirdEye, which states “the actionscript-based library enables users to create multi-dimensional data visualization interfaces for the analysis and presentation of information.” BirdEye caught my attention because it was listed in the article “Visualizing Relational Data Using Graph Theory” by Jason Bellone and Daniel Lang.

With Axiis and Prefuse Flare, you have some powerful tools to start visualizing your security data.

Setting Up Axiis for Security Visualization

John Gerber — Thu, 28 May 2009 15:44:28 +0000

Last week Axiis, an open source data visualization framework, was released. Axiis is built upon Degrafa, an open source declarative graphics framework, and Adobe Flex 3. Thanks to Nathan Yau for pointing this out in his post “Open Source Data Visualization Framework – Axiis.” There were also great comments to Nathan’s post, including a comparison of Axiis to Berkley’s Prefuse Flare. Flare is also a visualization toolkit, but it is written for Flash in ActionScript 3. For a discussion of Flash, Flex, and ActionScript 3, check out Lee Brimelow post “Flash Builder rebrand FAQ.”

Today we are going to walk through the steps of setting up Axiis on a Linux platform.

Install a Java JDK

Get the Sun JDK 6 from Sun’s website. Sun requires you to agree to terms, so you’ll need to go there and agree. Run the installer which gets downloaded. Agree again to the terms. The installer will install a few rpms and jars.

root# /bin/sh jdk-6u13-linux-i586-rpm.bin
root# ls /usr/java
default  jdk1.6.0_13  latest
root# declare -x JAVA_HOME="/usr/java/latest"
root# declare -x PATH="${JAVA_HOME}/bin:${PATH}"
root# java -version
java version "1.6.0_13"
Java(TM) SE Runtime Environment (build 1.6.0_13-b03)
Java HotSpot(TM) Server VM (build 11.3-b02, mixed mode)

Install Another Neat Tool (ANT)

Apache Ant is a free and open source tool for automating software build processes. It is similar to make but is implemented using the Java language using XML configuration files. It is platform-neutral.

root# cd  /work/src/
/work/src root# wget http://www.uniontransit.com/\
apache/ant/binaries/apache-ant-1.7.1-bin.tar.gz
/work/src root# md5sum apache-ant-1.7.1-bin.tar.gz
cc5777c57c4e8269be5f3d1dc515301c  apache-ant-1.7.1-bin.tar.gz
/work/src root# tar xzf apache-ant-1.7.1-bin.tar.gz
/work/src root# mv apache-ant-1.7.1 /work/software
/work/src root# cd  /work/software
/work/software root# ln -s apache-ant-1.7.1 ant
/work/software root# declare -x ANT_HOME="/work/software/ant"
/work/software root# declare -x PATH="${PATH}:${ANT_HOME}/bin"
/work/software root# ant
Buildfile: build.xml does not exist!
Build failed

The error above indicates that ant command is recognized by shell but it did not find build.xml file that needed to compile ant projects. So, it’s absolutely normal and the installation was successful. We will be creating build.xml files below.

Install the Flex SDK

The Adobe Flex 3.3 Software Development Kit (SDK) includes the Flex framework (component class library) and Flex compiler along with:

Automated testing framework.
Memory/Performance profiler.
Certain components, such as the charting and the AdvancedDataGrid component (the code will work, but a watermark image is layered over it).

Installation is fairly straight forward.

root# cd  /work/software
/work/software root# mkdir flex
/work/software root# cd flex
/work/software/flex root# wget \

http://download.macromedia.com/pub/flex/sdk/flex_sdk_3.zip

/work/software/flex root# unzip flex_sdk_3.zip
/work/software/flex root# export PATH=/work/software/flex/bin:${PATH}
/work/software/flex root# mxmlc --version
Version 3.3.0 build 4852

Test the installation out by creating the traditional “hello world” program. First, under your favorite editor, create the file helloworld.mxml file:

/home/ger/flex root# vi helloworld.mxml

Compile the program:

/home/ger/flex root# mxmlc --strict=true helloworld.mxml
/home/ger/flex root# ls helloworld.swf
helloworld.swf

Use your favorite browser to view the helloworld.swf flash file. You now need to copy the flexTasks.jar file to the ant’s lib folder. The flexTasks.jar file contains the ant task definitions for compiling flex applications.

root# cd  /work/software/flex
/work/software/flex root# cp /work/software/flex/ant/lib/flexTasks.jar \
/work/software/ant/lib

Create a new file build.xml with the following content (make the appropriate changes to FLEX_HOME and APP_ROOT):

/home/ger/flex root# vi build.xml

From the directory where build.xml file exist, execute ant:

/home/ger/flex root# ant
Buildfile: build.xml

build:
    [mxmlc] Loading configuration file /work/software/flex/frameworks/flex-config.xml
    [mxmlc] /home/ger/flex/helloworld.swf (181669 bytes)

BUILD SUCCESSFUL
Total time: 7 seconds

The flash file helloworld.swf was created when you ran the command ant. As with the make command, you now have a way to specify build options within the build.xml file. See “Talk:Flex Ant Tasks” for a much more detail description of setting up ant to perform Flex tasks.

Eclipse

Eclipse is an open source plug-in—based editor and IDE framework. It does a good job of code editing while allowing third parties to extend its capabilities through modules. We are going to pull down the “Eclipse IDE for C/C++ Developers” bundle, which already contains the CDT (C/C++ Development Tools). Your Java version will need to be 1.5 or higher.

/home/ger/flex root# cd /work/src
/work/src  root# wget http://mirrors.med.harvard.edu/eclipse//technology/epp/downloads/\
release/ganymede/SR2/eclipse-cpp-ganymede-SR2-linux-gtk.tar.gz
/work/src root# tar xzf eclipse-cpp-ganymede-SR2-linux-gtk.tar.gz
/work/src root# mv eclipse /work/software
/work/src root# su - ger
[ger@loxias ~]$ declare -x JAVA_HOME="/usr/java/latest"
[ger@loxias ~]$ declare -x PATH="${JAVA_HOME}/bin:${PATH}"
[ger@loxias ~]$ java -version
java version "1.6.0_13"
Java(TM) SE Runtime Environment (build 1.6.0_13-b03)
Java HotSpot(TM) Server VM (build 11.3-b02, mixed mode)
[ger@loxias ~]$ /work/software/eclipse/eclipse

Max Berger has written a great HOWTO “Setting up Eclipse” that will walk you through using Eclipse.

Aptana

One Eclipse plug-in, Aptana Studio, comes in both commercial and Community Edition. Since you have already installed Eclipse, you can install Aptana as a plug-in directly into your current Eclipse configuration. Follow the directions from the Aptana site.

From the Help menu in Eclipse, select Software Updates …
Select the Available Software tab
Click the “Add Site…” button.
Specify the Location Url update site: http://update.aptana.com/update/studio/3.4/ and click OK
Select the checkbox next to the added update site.
Click the Install.. button.
Complete instruction to install from update site.

If you would like to use the Aptana perspective, navigate and select Window > Open Perspective > Aptana. You will also want to change the default editor.

Flex Builder

Using the Flex SDK is more complicated and involves more work than using Flex Builder. Flex Builder comes with many nice features, such as tag completion, code hinting, built-in API reference manual, automatic importing of libraries, wizards, visual layout and styling, and automatic builds. There is a Flex Builder version for Linux, though it is a plugin-only version and unfortunately development has been put on hold. If you work in the education field, Adobe Flex Builder 3 Pro is available for free under Microsoft Windows and Mac OS X.

To get Flex Builder running on your system is simple:

root# cd  /work/src/
/work/src root# wget http://download.macromedia.com/pub/labs\
/flex/flexbuilder_linux/flexbuilder_linux_install_a4_081408.bin
/work/src root#  /bin/sh flexbuilder_linux_install_a4_081408.bin

At which point, the installation program will walk you through the installation process asking for the Eclipse folder. If you are following this document, the Eclipse directory would be /work/software/eclipse.

/work/src root#  su - ger
[ger@loxias ~]$ declare -x JAVA_HOME="/usr/java/latest"
[ger@loxias ~]$ declare -x PATH="${JAVA_HOME}/bin:${PATH}"
[ger@loxias ~]$ /bin/sh /work/software/Adobe_Flex_Builder_Linux/Adobe_Flex_builder.sh

Once you are running Eclipse, yo may choose the traditional Flex Builder perspective by navigate and select Window > Open Perspective > Other… > Flex Development to change it to the traditional Flex Builder look.

Degrafa

Degrafa is a declarative graphics framework that allows you to draw shapes and objects in your Flex application using an MXML type syntax. Degrafa is required for Axiis. Juan Sanchez in the post “An Introduction to Degrafa” provides a nice introduction.

root# cd  /work/software/
/work/software root# mkdir  degrafa
/work/software root# cd degrafa
/work/software/degrafa root# wget http://degrafa.googlecode.com/files/Degrafa_Beta3.1_Flex3.zip
/work/software/degrafa root# unzip Degrafa_Beta3.1_Flex3.zip
/work/software/degrafa root# cp /work/software/degrafa/Degrafa_Beta3.1_Flex3.swc  \
/work/software/ant/lib

I ran into memory problems with the Linux version of Flex Builder. It may be due to the software being Alpha. For the rest of this document, I will use ant to compile my programs with the idea that I am showing the more difficult method. If Flex Builder is working well for you, the ideas below end up being the same. It is always good to have an idea of what is going on behind a nice GUI application like Eclipse.

Below we walk through pulling down sample Degrafa code and compile the code. Please note that I start using the compiler.include-libraries attribute to specify the Degrafa library needs to be used in compiling the DegrafaPieMenu.mxml program.

root# cd  /home/ger/workspace
/home/ger/workspace root# mkdir ex3
/home/ger/workspace root# cd ex3
/home/ger/workspace/ex3 root# wget http://www.finflex.fi/projects/DegrafaPieMenu/srcview\
/DegrafaPieMenu.zip
/home/ger/workspace/ex3 root# unzip DegrafaPieMenu.zip
/home/ger/workspace/ex3 root# cd src
/home/ger/workspace/ex3/src root# vi build.xml

At this point, use the command ant to compile the program.

/home/ger/workspace/ex3/src root# ant
Buildfile: build.xml

build:
    [mxmlc] Loading configuration file /work/software/flex/frameworks/flex-config.xml
    [mxmlc] /home/ger/workspace/ex3/src/DegrafaPieMenu.swf (247701 bytes)

BUILD SUCCESSFUL
Total time: 8 seconds

Use your favorite web browser top open the file DegrafaPieMenu.swf. With the necessary software and libraries installed, we are finally ready for Axiis.

Axiis

We are going to pull down the Axiis libraries, place them with the Ant libraries, pull down sample Axiis code, setup a build.xml file, run ant, and view the results. We do everything the same as what we have done before.

root# cd  /home/ger/workspace
/work/software root# mkdir  axiis
/work/software root# cd axiis
/work/software/axiis root# wget http://axiis.googlecode.com/files/Axiis_Library.zip
/work/software/axiis root# unzip Axiis_Library.zip
/work/software/axiis root#  cp libs/* /work/software/ant/lib
/work/software/axiis root# cd /home/ger/workspace
/home/ger/workspace root# mkdir axiis
/home/ger/workspace root# cd axiis
/home/ger/workspace/axiis root# wget http://axiis.googlecode.com/files/Axiis_Examples.zip
/home/ger/workspace/axiis root# unzip Axiis_Examples.zip
/home/ger/workspace/axiis root# cd src
/home/ger/workspace/axiis/src root# vi build.xml

Trying to compile this with ant/mxmlc, produces an error message stating “/home/ger/workspace/axiis/src/Examples/ExampleBackground.mxml(40): Error: unable to resolve ‘Examples/axiis_logo_shadow.png’ for transcoding“. This is because ExampleBackground.mxml is in the same directory as axiis_logo_shadow.png, but the program specifies it is in “Examples/axiis_logo_shadow.png.” Modify ExampleBackground.mxml to just have embedded in line 40 “source=’axiis_logo_shadow.png’”. In other words, do not include the “Examples/” path.

Compile with the ant command:

/home/ger/workspace/axiis/src root# ant
Buildfile: build.xml

build:
    [mxmlc] Loading configuration file /work/software/flex/frameworks/flex-config.xml
    [mxmlc] /home/ger/workspace/axiis/src/LineAreaSeriesExample.swf (547707 bytes)

BUILD SUCCESSFUL
Total time: 10 seconds

From your favorite web browser, open the file LineAreaSeriesExample.swf to see:

Final Thoughts

Examine the LineAreaSeriesExample.mxml file. On line 40, the data used in plotting the graph is specified with “.” Examine the data/LineSeriesData.csv file. This would be the data file that gets generated by your security program. You could always modify the Flex program to access your favorite database. Adobe has a “Simple MySQL to Flex” example. There is also AS3FlexDB, which is an open source library that allows Adobe Flex applications to connect to a MySQL server. Alessandro Crugnola has written “Connect to MySQL in Adobe Flex using AS3FlexDB” to step you through the process. The point is, there are many possibilities. Play around with the different examples and start experimenting with different ways to visualize your security data. It is a great way to learn.

Some Ideas on Geocoding Security

John Gerber — Mon, 08 Dec 2008 03:02:50 +0000

Mark Twain once said, “The reports of my death are greatly exaggerated.” A thousands apologies for not posting in awhile. My only excuse is that I have been buried in work. Even today’s post will be brief. I wanted a few moments to indicate some of the work I am doing and provide a few pointers. I hope to follow this post with more details later.

First, a little about some of the work. I have had to evaluate IPs for an indication of their security threat. One method of evaluation is to compare the IPs to know bad actors. In this post, we will discuss a few data sources that are freely available, a few software packages that might prove useful, and finish up pointing to some sources for further evaluation.

Data Source

You can use various data feeds. Misbehaving IPs that are identified by your IDSP/IPS, honeypots, firewall logs, router logs, syslog servers, etc. will be of particular interest, being specific to your organization. For the sake of discussion, I wanted to point out some freely available sources of IPs that are blacklisted by the Internet community.

The Harimau Watchlist – Mel Mudin (spoonfork) provides this valuable source of information. Please read his post, “The Harimau Watchlist” for additional information. The information is updated daily.
Malware Domain Blocklist – this information is maintained as part of the DNS-BH project and represents a list of domains that are known to be used to propagate malware and spyware.

The sources for the Harimau Watchlist include:

Dshield Top IPs
Dshield Top Blocks
ShadowServer’s Know Russian Business Network
ShadowServer’s Known Bot Command & Control IPs/Blocks
EmergingThreats Known Compromised IPs/Blocks
Spamhaus Top IPs
Atlas (Arbor Networks) Top Threat Source
TrustedSource.org Top Email Senders
TrustedSource.org Most Active Storm Web Proxies
TrustedSource.org Most Newly Activated Storm Web Proxies
TrustedSource.org Most Recently Seen Storm Web Proxies
Projecthoneypot.org’s Most Recent Email Harvesters
Projecthoneypot.org’s Most Recent Spam Servers
Projecthoneypot.org’s Most Recent Comment Spammers
Projecthoneypot.org’s Most Recent Dictionary Attackers
Senderbase.org Top 100 Spammers
Senderbase.org Top 100 Virus Senders

The Malware Domain Blocklist sources include ddanchev.blogspot.com, www.matchent.com, siteadvisor, threatexpert, and many more. For more details, see the site.

Programming

I will not go into details now, but it is easy enough to setup a cron job to pull the information down and add the IPs to a database. If you decide to do this in Perl, a few modules that will come in handy:

LWP::UserAgent – can be used to dispatch web requests.
DBI – Perl database interface.
Net::Abuse::Utils – provides functions to lookup information about an IP or ASN. Information includes country code for an IP or ASN, ASN announcing an IP via BGP, CIDR network an IP is announced in, contact email addresses based on IP whois info, contact email addresses for a domain based on abuse.net data, contact email address from the SOA record for the rDNS zone for an IP, and listing information for an IP in a specific DNSBL.
Geo::IP – provides a simple file-based database. The GeoIP database simply contains IP blocks as keys, and countries as values. The data contains all public IP addresses and should be more complete and accurate than reverse DNS lookups.
Net::DNS – allows the programmer to perform nearly any type of DNS query.

A few other software packages you will likely use:

MySQL – is a multi-threaded and multi-user SQL (Structured Query Language) database server.
GeoLite Country – is similar to the GeoIP Country database, but is slightly less accurate. Please review Instructions on how to use our CSV databases with a SQL database.
GeoLite City – is similar to the GeoIP City database, but is less accurate.
Geo/IPfree – Perl module for looking up country of IP Address.

A Few Interesting Possibilities

One thing that can be done with the IPs is to map them using Google Earth. This will require you to create KML files, which are not difficult once you have the IPs along with their DNS and GeoIP data. Two scripts that help generate KML files from security data are:

Cosight – the security log file visualization tool used by the Colorado ISOC. Cosight parses logfiles looking for connections to or from internet addresses. It then uses the geolocation database from Maxmind to convert those addresses to coordinates for output as a KML overlay file.
KisGearth – a small perl script to convert kismet xml and gps logfiles to google earth kml files.

A few months ago, I did a post “Unclear and Present Danger.” The post outlined some of the electronic dangers facing an organization on the Internet. Thanks to the fantastic work done by the Shadowserver Foundation, we have a nice collection of some very interesting statistics mapped by country. Those examples can be very useful when mapping misbehaving IPs. Rather than repeat what has previously been posted, I’ll leave it to the reader to visit that entry.

While searching for an interesting way to represent and drill down from continents, to countries etc., I came across GeoTree, a hierarchical toponym browser for GeoNames. GeoNames is part of the Linked Data project, which brings together data from public sources and builds a web of open and free data where data sets are interlinked with each other. The Linked Data project represents a great wealth of information. Below is a mapping done by Richard Cyganiak of the projects involved in the Linked Data projects:

Walter Rafelsberger provides two interesting examples, that can be adapted for security representation and interpretation. Both examples make use of the Processing language. Processing is a data visualization programming language. Read more about Processing on Ben Fry’s or Casey Reas‘ blog.

Geosketch of world cities with a population of more than 1000, labeling those cities with more than 5 million:

The second example visualizes conversations of about 1500 users from Twitter. The arcs link positions of people who talk to each other:

Nathan Yau, from Flowing Data posted about “40 Essential Tools and Resources to Visualize Data.” The post contains valuable information with additional resource links. I came across Nathan’s post, while checking out FlowingData’s graphic post “Watching the Growth of Walmart Across America.” I was not able to embed the object. You will need to click on the image to view the growth of Walmart.

What is really nice is that you can downloaded the code, including the Actionscripts with the openings data from FlowingData’s site . With that code other types of growth can be illustrated in a similar manner. That is really nice. Modest Maps, a BSD-licensed display and interaction library for tile-based maps in Flash (ActionScript 2.0 and ActionScript 3.0) and Python was used to map the data. This reminds me of code_swarm:

code_swarm – Eclipse (short ver.) from Michael Ogawa on Vimeo.

If you have never watched the code_swarm video, you have to check it out. It was done by Michael Ogawa. The example above shows the commit history of the Eclipse open source project. To quote Michael:

code_swarm, shows the history of commits in a software project. A commit happens when a developer makes changes to the code or documents and transfers them into the central project repository. Both developers and files are represented as moving elements. When a developer commits a file, it lights up and flies towards that developer. Files are colored according to their purpose, such as whether they are source code or a document. If files or developers have not been active for a while, they will fade away. A histogram at the bottom keeps a reminder of what has come before.

It is a great example of visualizing something we traditionally would not think of outside of your run of the mill reports and numbers.

Take a look at Jamie Wilkinson’s post “Obama Wikipedia page edits,” which is a visualization of people who have contributed to the Barack Obama page on Wikipedia between October 2005 – November 2008. Users who edit a lot drift toward the center. Visualized using code_swarm (Processing) and Jamie’s Wikipedia page history parser Wikiswarm (Ruby). Code and instructions on how Jamie created this visualization can be found in his post “Wikiswarm: visualize Wikipedia page histories.”

Obama Wikipedia page edits from Jamie Dubs on Vimeo.

Most important, the code_swarm source if freely available.

Final Words

Today we explored a few interesting paths for representing data. Three excellent books to help guide us further on the visualization paths are:

Security Data Visualization by Greg Conti.
Applied Security Visualization by Raffael Marty.
Processing: A Programming Handbook for Visual Designers and Artists by Casey Reas and Ben Fry (forward by John Maeda) .

We have all heard the proverb, “A picture is worth a thousand words.” Another famous quote states, “The devil is in the details.” Or, if you prefer, “God is in the details.” If life was a Star Trek episode, Kirk could have used those two quotes to cause a computer to explode. Both statements are true and false, depending on the circumstances.

It is wise to remember the words of Siddhartha Gautama: “These blind men, every one honest in his contentions and certain of having the truth, formed schools and sects and factions.” Geocoding and data visualization simply provide tools to help interpret information. Interpretations are not absolute. If you are looking for a silver bullet that will help the blind see, and the ignorant smart, I am afraid your search must continue. The author A. L. Linall, Jr. once wrote, “Visualization and belief in a pattern of reality, activates the creative power of realization.” The best solutions will come from using a combination of tools to help explore the possibilities, discover insights, view the results from different views which helps with realization, and provide a way to effectively communicate results.

Security Data Visualization

John Gerber — Sun, 21 Oct 2007 05:39:31 +0000

“Visualization and belief in a pattern of reality, Activates the creative power of Realization.” — A. L. Linall, Jr.

As a follow up to my posting, “Traditional Thinking,” I wanted to examine one nontraditional solution that is still in the early stage of development. From watching detective shows, we all know that when a crimes occurs the police always need to first look towards family and friends. If it is not about passion, it is almost always about the money. Same is true in technology. Look to where organizations are putting the money. Love them or hate them, the Department of Homeland Security (DHS) is tasked with a very difficult task. To help DHS accomplish their tasks, all sides of the political spectrum have been willing to provide DHS with a budget to take on the task of securing our country.

Earlier this month, the White House issued changes to be made to the nation’s homeland security strategy. Jonah Czerwinski, from Homeland Security Watch, provides a link to the revised strategy, The National Strategy for Homeland Security. There is a section, Cyber Security: A Special Consideration, which states:

Many of the Nation’s essential and emergency services, as well as our critical infrastructure, rely on the uninterrupted use of the Internet and the communications systems, data, monitoring, and control systems that comprise our cyber infrastructure.
A cyber attack could be debilitating to our highly interdependent CI/KR and ultimately to our economy and national security.

A variety of actors threaten the security of our cyber infrastructure. Terrorists increasingly exploit the Internet to communicate, proselytize, recruit, raise funds, and conduct training and operational planning. Hostile foreign governments have the technical and financial resources to support advanced network exploitation and launch attacks on the informational and physical elements of our cyber infrastructure. Criminal hackers threaten our Nation’s economy and the personal information of our citizens, and they also could pose a threat if wittingly or unwittingly recruited by foreign intelligence or terrorist groups. Our cyber networks also remain vulnerable to natural disasters.

In order to secure our cyber infrastructure against these man-made and natural threats, our Federal, State, and local governments, along with the private sector, are working together to prevent damage to, and the unauthorized use and exploitation of, our cyber systems. We also are enhancing our ability and procedures to respond in the event of an attack or major cyber incident. The National Strategy to Secure Cyberspace and the NIPP’s Cross-Sector Cyber Security plan are guiding our efforts.

Let us follow the money, or at least a small part of the money.

There are five Regional Visualization and Analytics Centers (RVACs) led by Penn State University, Purdue University, Stanford University, the University of North Carolina at Charlotte, and the University of Washington. While Stanford was the first center, there are no links to the work. Stanford is performing “research on network traffic analysis for intrusion detection; cognitive and perceptual principles supporting reasoning with space and time; and methods to support exploratory analysis of graphs in relational databases.” These RVACs collaborate with the NVAC.

In July 2007, DHS announced a partnership between NVAC and National Science Foundation (NSF) to conduct a joint research program in data and visual analytics. A 5-year plan for collaboration, dependent on available resources, was established.

Currently, the NSF is soliciting proposals from “academia that capitalize on knowledge and expertise in the fields of mathematics, computational science and intelligent systems. The goal is to produce new data representations and transformations that will enable data stakeholders to detect the expected and discover the unexpected in massive data sets. This new program is called Foundations of Data and Visual Analytics, and FODAVA is the focus of the new NSFVAC. FODAVA is concerned only with a subset of the overall problem, namely the creation of the mathematical and computational sciences foundations required to transform data in ways that permit visual-based understanding.”

The proposals are due November 20, 2007. There will be five to seven awards consisting of one five-year FODAVA-Lead award totaling $3,000,000; four to six two to three year FODAVA-Partner awards totaling $300,000 to $500,000 each. This breaks down to $2,250,000 per year for 5 years. NSF will provide $1,500,000 per year for up to five years. DHS will provide $750,000 per year for up to five years.

Let me continue to quote the NVAC’s Novermber 2007 issue of VAC Views, “NVAC and NSF is establishing two types of research efforts: FODAVA Lead and FODAVA Partnerships. The FODAVA Lead effort will be granted to a research team where all team members belong to a single academic institution that will assume a leadership and coordination role. The FODAVA Lead will also play a key role in the development of FODAVA as a research field. In addition to forming the lead scientific research team, this institution will be responsible for assuring that results are disseminated to the FODAVA community, that effective liaison between FODAVA researchers and NVAC takes place, that testbed data sets are developed and disseminated and that the mathematics and computer science research communities become increasingly aware of the need for FODAVA-related research. FODAVA Partnership efforts will be two-to-three-year fundamental research projects. These academic partners will actively participate with the FODAVA Lead institution in developing FODAVA as a field.”

I find it interesting that both DHS and NSF are looking to security data visualization. It might just be time to read, “Security Data Visualization” by Greg Conti. Raffael Marty, author of AfterGlow, wrote the two chapters on IDS signature tuning and firewall log analysis. Raffael is also working on a book which will dive deeper into some visualization topics around security and focusing on use-cases. To quote Raffael blog posting, “How do you use visualization for compliance, insider threat, and perimeter threat? What are some of the tools out there, what are the data sources, and what are the different types of graphs you should know and understand when you are visualizing security data?” No other details are available at this time, but I look forward to what promises to be a very interesting book.

Siddhartha Gautama wrote, “These blind men, every one honest in his contentions and certain of having the truth, formed schools and sects and factions…” I am curious if sects and fractions are developing. Ian Greg in “The Failure of the Academic Contribution to Security Science” explains:

[A]cademics have presented stuff that is sometimes interesting but rarely valuable. They’ve pretty much ignored all the work that was done before hand, and they’ve consequently missed the big picture.

Why is this? One reason is above: academic work is only serious if it quotes other academic work. The papers above are reputable because they quote, only and fulsomely, other reputable work. And the work is only rewarded to the extent that it is quoted … again by academic work.

The academics are caught in a trap: work outside academia and be rejected or perhaps worse, ignored. Or, work with academic references, and work with an irrelevant rewarding base. And be ignored, at least by those who are monetarily connected to the field.

By way of thought experiment, consider how many peer-review committees on security conferences include the experts in the field?

Dr Anton Chuvakin states in “Once More on Failure of Academic Research in Security:”

Many people, myself included, have bemoaned the complete failure of academic research in information security. The main reason for this is a complete disconnect of academic security research from real-world threats and vulnerabilities (e.g. I still see people publishing papers inventing signature-based network IDS systems, reinventing MAC/RBAC, neural nets to catch hackers, etc – and if I hear about the Lincoln labs 1998 intrusion detection data set again, I will screeeeeeeeeeeam! )

Greg Conti is a Program Co-Chair while Raffael Marty on the Program Committee of the upcoming National Security Agency’s (NSA) National Information Assurance Research Laboratory (NIARL) sponsored VizSEC 2007 Workshop on Visualization for Computer Security. The workshop will be held in conjunction with IEEE Vis 2007 and IEEE InfoVis 2007. John Gerth, manager of the Computer Graphics Laboratory in the Department of Computer Science at Stanford University, is the only person on any of the boards for the workshop from the RVACs. The Standford RVAC supported John’s paper, “Enhancing Visual Analysis of Network Traffic Using a Knowledge Representation.”

While preparing the “Presentations” post, I came across the below visualization presentations. Again I am left questioning why the researchers from the RVACs sites are not presenting.

Security data visualization is a new field. This would explain why, while interest exist, work is spread in research pockets throughout industry, government, and academia. Visualization is not the solution for every security problem. Still, in combination with existing tools, it promises to help explore data, discover insights, and provide a way to effectively communicate results. It is a most interesting field offering intriguing possibilities.