Security Advancements at the Monastery » perl

Google Visualization: An Example Graphing NVD CVE Data

John Gerber — Fri, 16 Apr 2010 15:54:42 +0000

Google visualization offers graphing abilities to any number of projects. Why should security professionals care? If you are going to have to collect and present security metrics, it is best to showcase them in the very best manner possible. Andrew Jaquith in his article, “Creating meaningful information security metrics” states, “For 2010, Forrester Research expects that overall security budgets will rise less than 5 percent over 2009 –higher than in the previous year, but not by much.” Andrew goes on to point out, “smart security managers, sensing sudden vulnerability in their budgets, seek better ways to measure and prove the value of what they do every day.”

In today’s work environment there is a need to show changes, potential risks, improved performance, etc. in all areas of the company’s operations. Security professionals need to be prepared to answer the basic question, “why should the CIO or CEO care about security?” CSO Online has a great quote from the post, “From the CIO: Why You Didn’t Get the CISO Job” that challenges us to consider our views when it comes to security. The post states, “laser focus on your speciality is great in middle management. It’s what we want. One of the really hard things about jumping from management to executive is a focus on the whole of the business. It’s a rare person who manages it quickly or easily.” That is basically the problem with metrics. It is a battle between generalization to the point of uselessness and details to the point of not being understandable or collectible. At the end of the day, something needs to be done because the security industry is currently leaving upper management in the position of not understanding what is going on within their business. That is a risk that not acceptable.

Andrew’s article discusses what kind of security metrics should be used. Additional sources of information on security metrics can be found in a previous post entitled “Security Metrics.” The post provides links to wonderful sources on security metric information. You might also want to take a look at the CIS Consensus Security Metrics v1.0.0 guide, NIST Special Publication (SP) 800-55 Rev 1 “Security Metrics Guide for Information Technology Systems”, NIST IR-7564 “Directions in Security Metrics Research”, “Twenty Most Important Controls and Metrics for Effective Cyber Defense and Continuous FISMA Compliance,” and “Metrics, measures & Myths.” Once you have start gathering metrics, you will want to present them in an easy to understand format. This is where Google Visualization can help.

Today’s post walks through an example using the data from the National Institute of Standards and Technology (NIST) National Vulnerability Database (NVD ) Common Vulnerabilities and Exposures (CVE) database. The purpose is to provide a working example from which you can learn and apply to the various metrics gathered at your organization.

Data Source

A previous post, “Standardization and Interoperability in Security,” discussed how the Security Content Automation Protocol (SCAP) is an attempt to help defenders by providing a collection of XML schemas/standards that allow technical security information to be exchanged between tools. SCAP components consists of:

Common Configuration Enumeration (CCE): provide unique identifiers to system configuration issues in order to facilitate fast and accurate correlation of configuration data across multiple information sources and tools.
Common Platform Enumeration (CPE): a structured naming scheme for information technology systems, platforms, and packages.
Common Vulnerability Enumeration (CVE): a dictionary of publicly known information security vulnerabilities and exposures.
Common Vulnerability Scoring System (CVSS): a vulnerability scoring system designed to provide an open and standardized method of rating IT vulnerabilities. NIST has even provided a calculator for creating CVSS vulnerability severity scores.
eXtensible Checklist Configuration Description Format (XCCDF): a specification language for writing security checklists, benchmarks, and related kinds of documents. NIST has released the NIST Interagency Report 7275 Revision 3 “Specification for Extensible Configuration Checklist Description Format (XCCDF) Version 1.1.4.”
Open Vulnerability Assessment Language (OVAL): an information security community standard to promote open and publicly available security content, and to standardize the transfer of this information across security tools and services.

We are going to make use of the data from NVD/CVE XML feed with the Common Vulnerability Scoring System (CVSS) mappings (version 2.0). NIST documentation states:

CVSS provides an open framework for communicating the characteristics and impacts of IT vulnerabilities. Its quantitative model ensures repeatable accurate measurement while enabling users to see the underlying vulnerability characteristics that were used to generate the scores. Thus, CVSS is well suited as a standard measurement system for industries, organizations, and governments that need accurate and consistent vulnerability impact scores. Two common uses of CVSS are prioritization of vulnerability remediation activities and in calculating the severity of vulnerabilities discovered on one’s systems.

NVD provides CVSS ‘base scores‘ representing the innate characteristics of each vulnerability. ‘Temporal scores,’ which change over time due to events external to the vulnerability, are not provided though NVD does provide a CVSS score calculator. This allows an organization to add temporal data and even factor in ‘environmental scores‘ customized to reflect the impact of the vulnerability on the organization. Please refer to the CVSS standards guide and the OWASP Risk Rating Methodology concerning factors involved in estimating the severity of risks to your business.

NVD CVE XML Schema

For our example, we will be using the data feeds nvdcve-2.0-2010.xml and nvdcve-2.0-2009.xml. Examining the CVE XML 2.0 Schema, we are particularly interested in certain vulnerability and CVSS scoring information. For example, for CVE-2010-1228, we will parse and pull the following kind of information:

 id="CVE-2010-1228">
  CVE-2010-1228
  2010-04-01T18:30:00.453-04:00
  
  2010-04-05T00:00:00.000-04:00
  
  
    
      10.0
      NETWORK
      LOW
      NONE
      COMPLETE
      COMPLETE
      COMPLETE
      http://nvd.nist.gov

Using Perl to Retrieve the CVE File

Initially we will read the nvdcve-2.0-2010.xml and nvdcve-2.0-2009.xml files. If we start retrieving the file regularly, we would want to change this to nvdcve-2.0-recent.xml. Of course, previous years can also be read in to provide a longer perspective on vulnerabilities. A simple example of a Perl subroutine to read the NVD CVE file and save it locally would be:

sub readpage {
   my($url,$nvd_file) = @_;
   my($proxy) = "http://your-proxy-server:proxy-port";
   my $ua = new LWP::UserAgent;
   $ua->proxy(http  => $proxy);
   $ua->proxy(ftp => $proxy);
   $ua->proxy(https => $proxy);
   # Go out and retrieve page
   my $req = new HTTP::Request('GET', $url);
   my $res = $ua->request($req);
   my $pjstatus = 1;
   # Check if the requested webpage is there and return results
   if ($res->is_success) { # Request successful
       open(OUTFILE,">$nvd_file") || ($pjstatus = 0);
       if ($pjstatus) {
          print OUTFILE $res->content;
       }
       close(OUTFILE);
   }
   else {
      $pjstatus = 0;
   }
   return($pjstatus);
}

Please substitute “http://your-proxy-server:proxy-port” with your site’s proxy server and port, if applicable.

Creating a MYSQL Table to Hold the Data

There is a great deal of information in the NVD CVE file. You will need to determine what information your organization will be interested in storing and graphing. For better or worse, folks have come to expect vulnerabilities to have a “Low,” “Medium,” or “High” score. NIST has stated concerning the NVD Vulnerability Severity Ratings:

NVD provides severity rankings of “Low,” “Medium,” and “High” in addition to the numeric CVSS scores but these qualitative rankings are simply mapped from the numeric CVSS scores:
1. Vulnerabilities are labeled “Low” severity if they have a CVSS base score of 0.0-3.9.
2. Vulnerabilities will be labeled “Medium” severity if they have a base CVSS score of 4.0-6.9.
3. Vulnerabilities will be labeled “High” severity if they have a CVSS base score of 7.0-10.0.

While preferring quantitative over qualitative values, for this example I would like to create a stacked column chart. We will add a severity column which is based on the CVSS score. An example table follows:

CREATE DATABASE vulnerabilities;
USE vulnerabilities;
DROP TABLE IF EXISTS `nvdcve`;
CREATE TABLE `nvdcve` (
  `cve_id` varchar(13) NOT NULL,
  `published` datetime default NULL,
  `modified` datetime default NULL,
  `score` DECIMAL(5,2) default '0.0',
  `severity` varchar(6) default 'LOW',
  `vector` varchar(25) default NULL,
  `complexity` varchar(25) default NULL,
  `authentication` varchar(25) default NULL,
  `confidentiality` varchar(25) default 'NONE',
  `integrity` varchar(25) default 'NONE',
  `availability` varchar(25) default 'NONE',
  `summary` varchar(512) default NULL,
  PRIMARY KEY  (`cve_id`),
  INDEX (score),
  INDEX (vector)
)

Using Perl Populating the Database

Populating the database table is simply a matter of reading the file and adding the entries to the table. An example Perl subroutine follows:

sub readxml {
   my($nvd_file, $dbh) = @_;
   my $parser = XML::LibXML-> new();
   my $doc    = $parser-> parse_file($nvd_file);
   my $xc     = XML::LibXML::XPathContext-> new( $doc->documentElement() );
   $xc-> registerNs(
      def  => 'http://scap.nist.gov/schema/feed/vulnerability/2.0' );
   $xc-> registerNs(
     vuln => 'http://scap.nist.gov/schema/vulnerability/0.4' );
   $xc-> registerNs( cvss => 'http://scap.nist.gov/schema/cvss-v2/0.2' );
   for my $entry ($xc-> findnodes("/def:nvd/def:entry")) {
      my $cve = $xc-> find('vuln:cve-id',$entry);
      my $published = $xc-> find('vuln:published-datetime', $entry);
      my $modified = $xc-> find('vuln:last-modified-datetime', $entry);
      my $summary = $xc-> find('vuln:summary', $entry);
      my $skip = 0;
      my ($metrics) = $xc-> findnodes('vuln:cvss/cvss:base_metrics', $entry) or ($skip = 1);
      if (! $skip) {
         my $score = $xc-> find('cvss:score', $metrics);
         my $vector = $xc-> find('cvss:access-vector', $metrics);
         my $complexity = $xc-> find('cvss:access-complexity', $metrics);
         my $authentication = $xc-> find('cvss:authentication', $metrics);
         my $confidentiality =
            $xc-> find('cvss:confidentiality-impact', $metrics);
         my $integrity = $xc-> find('cvss:integrity-impact', $metrics);
         my $availability = $xc-> find('cvss:availability-impact', $metrics);
         my $severity = "LOW";
         if (int($score) >= 7) {
            $severity = "HIGH";
         }
         elsif (int($score) >= 4) {
            $severity = "MEDIUM";
         }
         my $sql = qq{ SELECT count(*) FROM nvdcve WHERE cve_id=? };
         my $sth = $dbh->prepare( $sql );
         my $rc = $sth->execute($cve);
         if ( $rc) {
            my($exist) = $sth->fetchrow_array();
            if (! $exist) {
                $sql = qq{ INSERT INTO nvdcve SET cve_id=?,
published=?, modified=?, score=?, severity=?, vector=?, complexity=?,
authentication=?, confidentiality=?, integrity=?,availability=?, summary=? };
               $sth = $dbh->prepare( $sql );
               $rc = $sth->execute($cve,$published,$modified,$score,
$severity,$vector,$complexity,$authentication,
$confidentiality,$integrity,$availability,$summary);
            }
         }
      }
   }
}

The Perl Program to Pull It All Together

The above subroutines use the Perl modules LWP::UserAgent, XML::LibXML, XML::LibXML::XPathContext, and DBI. A sample Perl program that calls the above subroutines to pull down the NVD CVE data and load it into a MySQL table would be:

#!/usr/local/bin/perl -w
use LWP::UserAgent;
use XML::LibXML;
use XML::LibXML::XPathContext;
use DBI;
BEGIN{push @INC, "/home/jgerber/projects/nvd/perl"}
use nvdsubs qw($db_host $db $mysql_user $mysql_passwd $mysql.sock
readpage readxml );
# Main
my $datadir = "/home/johngerber/projects/nvd/data";
my @timeData = localtime(time);
my $year = 1900 + $timeData[5];
my $prev_year = 1900 + $timeData[5] - 1;
my $url = "http://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-" .
    $year . ".xml";
my $prev_url = "http://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-" .
    $prev_year . ".xml";
my $nvd_file = $datadir  . "/nvdcve-". $year . ".xml";
my $prev_nvd_file = $datadir  . "/nvdcve-". $prev_year . ".xml";
$db = "vulnerabilities";
local($dbh) = DBI->connect("DBI:mysql:mysql_socket=$mysql.sock;$db:$db_host",
$mysql_user, $mysql_passwd) || die "ERROR: Connecting: $DBI::errstr\n";
my ($pjstatus) = &readpage($prev_url,$prev_nvd_file);
if ($pjstatus) {
   &readxml($prev_nvd_file,$dbh);
}
$pjstatus = &readpage($url,$nvd_file);
if ($pjstatus) {
   &readxml($nvd_file,$dbh);
}
exit;

The nvdsubs.pm file will not be included in this post. The subroutines are defined and the only pieces missing are the MySQL database username and password. You don’t need mine. Add your own. At this point, we have everything we need to finally use Google Visualization to create a graph.

Google Visualization

We are going to create a Perl program that will read our MySQL nvdcve table and generate the JavaScript that will render our charts on the client’s browser. First, we want to define the JavaScript we want to produce. Just to alleviate some concerns, with Google Visualization your data is only shared between your server and the client connecting. This is unlike Google Charts where your data is sent to Google where it is made into a chart and the result is sent back. Google states concerning the logging of chart data (via Google Charts), “The chart data included in the HTTP request is saved in temporary logs for no longer than two weeks for internal testing and debugging purposes.” Every example in the Google Visualization Gallery will state the data policy. For Google Charts, stated at the bottom of the page for each gadget description the data policy:

While Google Visualization gadgets will have the following stated data policy:

Loading Google Libraries

The first thing the JavaScript needs to do is load the required libraries. This is accomplished with the lines:

Area Chart and Table

In this example we are going to create an column chart. In a later section, “Other Charting Options” (see below) we define different Google Visualization charting options.

JavaScript code for a sample column chart would be:

    <script type='text/javascript'>
      google.load('visualization', '1', {packages:['columnchart']});
      google.setOnLoadCallback(drawChart);
      function drawChart() {
        var data = new google.visualization.DataTable();
        data.addColumn('date', 'Date');
        data.addColumn('number', 'High');
        data.addColumn('number', 'Medium');
        data.addColumn('number', 'Low');
        data.addRows([
           [new Date(2009, 0, 30),92,97,3],
           [new Date(2009, 1, 27),168,142,25],
           [new Date(2009, 2, 31),141,165,9],
           [new Date(2009, 3, 30),132,203,12],
           [new Date(2009, 4, 29),158,153,8],
           [new Date(2009, 5, 30),200,199,22],
           [new Date(2009, 6, 31),190,195,11],
           [new Date(2009, 7, 31),127,139,14],
           [new Date(2009, 8, 30),233,208,14],
           [new Date(2009, 9, 30),163,167,18],
           [new Date(2009, 10, 30),129,172,8],
           [new Date(2009, 11, 31),200,211,19],
           [new Date(2010, 0, 29),157,139,14],
           [new Date(2010, 1, 26),137,143,12],
           [new Date(2010, 2, 31),252,242,18],
           [new Date(2010, 3, 13),92,118,17]
        ]);
        var chart = new google.visualization.ColumnChart(document.getElementById('s4graph'));
        chart.draw(data, {displayAnnotations:true, is3D: true, isStacked: true, min: 0,
          allowHtml: true, colors:[{color:'#E41B17', darker:'#C11B17'}, {color:'#FFA500', darker:'#E56717'}, {color:'#FFE87C', darker:'#C8B560'}]});
      }
    script>

The resulting image would be the following column chart:

Rendering the Table

When providing qualitative results, I like to back them up with more accurate numeric values. Let us include a table with links to the CVSS scores for each vulnerability.

    <script type='text/javascript'>
      google.load('visualization', '1', {packages:['table']});
      google.setOnLoadCallback(drawChart);
      function drawChart() {
        var data2 = new google.visualization.DataTable();
        data2.addColumn('date', 'Date');
        data2.addColumn('number', 'High');
        data2.addColumn('number', 'Medium');
        data2.addColumn('number', 'Low');
        data2.addRows([
           [{v:new Date(2009, 0, 30),
              f:'2009-01-30'}, 92,97,3],
           [{v:new Date(2009, 1, 27),
              f:'2009-02-27'}, 168,142,25],
           [{v:new Date(2009, 2, 31),
              f:'2009-03-31'}, 141,165,9],
           [{v:new Date(2009, 3, 30),
              f:'2009-04-30'}, 132,203,12],
           [{v:new Date(2009, 4, 29),
              f:'2009-05-29'}, 158,153,8],
           [{v:new Date(2009, 5, 30),
              f:'2009-06-30'}, 200,199,22],
           [{v:new Date(2009, 6, 31),
              f:'2009-07-31'}, 190,195,11],
           [{v:new Date(2009, 7, 31),
              f:'2009-08-31'}, 127,139,14],
           [{v:new Date(2009, 8, 30),
              f:'2009-09-30'}, 233,208,14],
           [{v:new Date(2009, 9, 30),
              f:'2009-10-30'}, 163,167,18],
           [{v:new Date(2009, 10, 30),
              f:'2009-11-30'}, 129,172,8],
           [{v:new Date(2009, 11, 31),
              f:'2009-12-31'}, 200,211,19],
           [{v:new Date(2010, 0, 29),
              f:'2010-01-29'}, 157,139,14],
           [{v:new Date(2010, 1, 26),
              f:'2010-02-26'}, 137,143,12],
           [{v:new Date(2010, 2, 31),
              f:'2010-03-31'}, 252,242,18],
           [{v:new Date(2010, 3, 13),
              f:'2010-04-13'}, 92,118,17],
        ]);
        var table = new google.visualization.Table(document.getElementById('s4graph_tab'));
        table.draw(data2, {showRowNumber: true, sortAscending: false, sortColumn: 0, allowHtml: true});
      }
    script>

The JavaScript code assumes there is a PHP program called cvealerts.php under the /nvd directory on your web server. Adjust to your environment. A sample PHP program that could be used for cvealerts.php is provided below. The resulting table chart would look like:

Handling Events: Interactions Between Graphs

We now have two different types of graphs representing the same data. We want to add interaction between the graphs so the viewer can see the relationship. With tables rows are selected when the user clicks, which correspond to the whole column of the stacked column chart. It is not a perfect fit, but it does demonstrate nicely use of adding interactions.

        // Set a 'select' event listener for the table.
        // When the table is selected,
        // we set the selection on the line graph.
        google.visualization.events.addListener(table, 'select', function() {
          chart.setSelection([{row: table.getSelection()[0].row, column: 1}]);
         });
        // Set a 'select' event listener for the graph.
        // When the graph is selected,
        // we set the selection on the table.
        google.visualization.events.addListener(chart, 'select', function() {
           table.setSelection([{row: chart.getSelection()[0].row}]);
        });

Providing Detailed Information

When the table chart link is clicked, we would like to provide some detailed information about the vulnerability. For this example, we will do this with a simple PHP program placed in the /nvd directory on the web server. The program is called cvealerts.php.


session_start();
function db_connect($table) {
   $result = mysql_pconnect(":", "", "");
   if (!$result) return false;
   if (!mysql_select_db($table)) return false;
   return $result;
}
function do_html_header($title,$checkuser,$logpage) {
?>
  <html> <head> <title>$title?>title>head>
  <body bgcolor="#FFFFFF">

}
function do_html_footer() {
?>
<table>
<tr><td ALIGN=CENTER NOWRAP WIDTH="590">font>
<font face="Verdana, Arial, Helvetica" size=-2>Notice to Users: Use
of this system constitutes consent to security monitoring and testing.
<br>All activity is logged with your host name and IP address.font>
td>tr>
table>
body>
 html>

}
// Main
$dates= array();
$stringlist = "";
if (isset($_GET['date'])) {
    $passdates = explode(",",$_GET['date']);
    for ($index=0; $index<count($passdates); $index++) {
       array_push($dates, $passdates[$index]);
       $stringlist .= $passdates[$index] . " ";
    }
}
else {
  print("Confusion over how you arrived at this page.\n");
  exit;
}
$stringlist = preg_replace("/ $/", "",$stringlist);
do_html_header("Review NVD CVE Announcements for Month Ending $stringlist",1,1);
$nvd_host = "http://web.nvd.nist.gov/view/vuln/detail?vulnId=";
$conn = db_connect("vulnerabilities");
if (!$conn)
   logit("Could not connect to database vulnerabilities - please try later.\n",1);
for ($index=0; $index<count($dates); $index++) {
   $rule = $dates[$index];
   $sql = "SELECT cve_id,score,published,vector,severity,complexity,left(summary,50)
    FROM vulnerabilities.nvdcve
      WHERE date_format(published,'%Y-%m')='$rule'
       ORDER BY (score+0)";
   $result = mysql_query($sql,$conn);
   if (!$result)
       logit("Problem with $sql\n",1);
   print("
\n");for($count=1;list($cve_id,$score,$date,$vector,$severity,$complexity,$shortsum)=mysql_fetch_array($result, MYSQL_NUM);++$count){?><tr><td CLASS="plfieldhdrleft" WIDTH="20%" BGCOLOR='#F0F5FF'>  print("$cve_id"); ?>
      td>
      <td CLASS="plfieldhdrleft" BGCOLOR='#F9FCFF'>
        print($score); ?>
      td>
      <td CLASS="plfieldhdrleft" BGCOLOR='#F0F5FF'>
        print($date); ?>
      td>
      <td CLASS="plfieldhdrleft" BGCOLOR='#F9FCFF'>
        print($vector); ?>
      td>
      <td CLASS="plfieldhdrleft" BGCOLOR='#F0F5FF'>
        print($severity); ?>
      td>
      <td CLASS="plfieldhdrleft" BGCOLOR='#F9FCFF'>
        print($complexity); ?>
      td>
      <td CLASS="plfieldhdrleft" BGCOLOR='#F0F5FF'>
        print($shortsum); ?>
      td>
      tr>

   }
}
print("
Bulletin 
Impact 
Date 
Vector 
Severity 
Complexity 
Short Summary
            
        

      
      
");
do_html_footer();

The PHP program would generate a HTML table displaying the NVD CVE alerts for that month. The table would look like:

When the CVE link is clicked on, the user is taken to the NIST NVD site where additional information is available.

Using Perl to Create the JavaScript

The Perl code is rather simple now that we have the MySQL tables defined and the JavaScript we want to generate. Much of the code consists of the JavaScript listed above.

#!/usr/local/bin/perl -w
use DBI;
use Time::Local;
use POSIX qw(strftime);
use LWP::UserAgent;
BEGIN{push @INC, "/home/jgerber/projects/nvd/perl"}
use ornl_feds qw($db_host $db $mysql_user $mysql_passwd );
sub slide_nvd_alerts {
  my($min_date,$graph_name,$web_link,$dbh) = @_;
  my $slide = "";
  my $slide_head = qq!
    
!;
   if ($min_date eq "") {
      my $sql2 = qq{ SELECT min(published) FROM vulnerabilities.nvdcve };
      my $sth2 = $dbh->prepare( $sql2 );
      my $rc2 = $sth2->execute();
      if ($rc2) {
         $min_date = $sth2->fetchrow_array();
      }
   }
   my $table_data = "";
   my $graph_data = "";
   my $sql2 = qq{ select date_format(published,'%Y-%m'),severity,count(severity)
      FROM vulnerabilities.nvdcve where published >= ? group by date_format(published,'%Y-%m'),severity };
   my $sth2 = $dbh->prepare( $sql2 );
   my $rc2 = $sth2->execute($min_date);
   if ($rc2) {
      my ($change,$virgin,$ht,$mt,$lt,$mmax_date) = ("",1,0,0,0,"");
      while (my($snapshot_date, $severity, $pcount) = $sth2->fetchrow_array()) {
         my $sql3 = qq{ SELECT max(published) FROM vulnerabilities.nvdcve where
date_format(published,'%Y-%m')=? };
         my $sth3 = $dbh->prepare( $sql3 );
         my $rc3 = $sth3->execute($snapshot_date);
         $max_date =  $sth3->fetchrow_array();
         $max_date =~ s/ \S+$//;
         if ($change ne $snapshot_date) {
            if (! $virgin) {
                my($year,$month,$day) = split("-",$mmax_date);
                my $mmonth = $month;
                $month--;
                $graph_data .= qq!           [new Date($year, $month, $day),$ht,$mt,$lt],
!;
                $table_data .= qq!           [{v:new Date($year, $month, $day),
              f:'$mmax_date'}, $ht,$mt,$lt],
!;
                ($ht,$mt,$lt) = (0,0,0);
             }
             $change = $snapshot_date;
          }
          if ($severity eq "HIGH") { $ht = $pcount; }
          elsif ($severity eq "MEDIUM") { $mt = $pcount; }
          elsif ($severity eq "LOW") { $lt = $pcount; }
          if ($mmax_date eq "") { $mmax_date = $max_date; }
          if ($mmax_date lt $max_date) { $mmax_date = $max_date; }
          $virgin = 0;
      }
      my($year,$month,$day) = split("-",$mmax_date);
      my $mmonth = $month;
      $month--;
      $graph_data .= qq!           [new Date($year, $month, $day),$ht,$mt,$lt]
!;
     $table_data .= qq!           [{v:new Date($year, $month, $day),
              f:'$mmax_date'}, $ht,$mt,$lt],
!;
   }
   $table_data .= "        ]);\n";
   $graph_data .= "        ]);\n";
   $slide = $slide_head .  $graph_data . $slide_head_table . $table_data . $slide_tail;
   return($slide);
}
sub slide_body {
  my($graph_name,$title,$style) = @_;
  my $table_name = $graph_name . "_tab";
  my $table_text = "div id=\"$table_name\"";
  if ($style ne "") {
     $table_text .= " style=\'$style\'";
  }
  my $slide2 = "$title
\n";
  my $itext = "div id=\"$graph_name\"";
  if ($style ne "") {
     $itext .= " style=\'$style\'";
  }
  $slide2 .= qq{
    
    <$itext>
    <$table_text>
       
    
    
  };
  return($slide2);
}
# Main
my $web_link = "/nvd";
my $results_dir = "/data/html" . $web_link;
my $result_file = $results_dir . "/nvdcve_stats.html";
my $debug = 1;
my $db = "vulnerabilities";
local($dbh) = DBI->connect("DBI:mysql:$db:$db_host", $mysql_user, $mysql_passwd) ||
   die "ERROR: Connecting: $DBI::errstr\n";
$slides_data .= &slide_body("s4graph","NVD CVE Alerts","width:700px; height:400px;");
$slides_head .= &slide_nvd_alerts("","s4graph",$web_link,$dbh);
open(OUTFILE,">$result_file");
print OUTFILE "\nNVD CVE Statistics\n";
print OUTFILE "\n";
print OUTFILE $slides_head;
print OUTFILE "\n\n";
print OUTFILE $slides_data;
print OUTFILE "\n";
close(OUTFILE);
exit;

Other Charting Options

Google, Google users, and other companies have shared some JavaScript visualizations built on the Google Visualization API to help you get started. Below are some example:

	Annotated Time Line (GWT Integrated) An animated time series chart. By: Google
	Area Chart (GWT Integrated) Interactive area chart. By: Google
	Bar Chart (GWT Integrated) Interactive bar chart. By: Google
	Bars of Stuff Fun bar charts using images of trains, chocolate, worms, and more. By: The visapi project
	Bio Heat Map Heatmaps are a useful way to visualize matricies of data. Scientists often use green-black-red heatmaps to visualize gene expression data from microarrays. This visualization supports both three color heatmaps (ex: green to black to red) and two color heatmaps (ex: white to yellow). By: Institute for Systems Biology
	Column Chart (GWT Integrated) Interactive column chart. By: Google
	Drastic Treemap A dynamic treemap in Flash. By: DrasticData
	Dygraphs The dygraphs JavaScript library produces interactive, zoomable charts of time series. By: Dan Vanderkam
	Filters A Visualization that acts as a control over other visualizations. It is rendered within the browser using HTML. This visualization offers the ability to select some criteria to filter the DataTable used by the controlled visualizations. By: Institute for Systems Biology
	Gauge (GWT Integrated) Each numeric value is displayed as a gauge. By: Google
	Geo Map (GWT Integrated) A map of a country, continent, or region map, with colors and values assigned to specific regions. Values are displayed as a color scale, and you can specify optional hovertext for regions. By: Google
	Intensity Map (GWT Integrated) An intensity map that highlights regions or countries based on relative values. By: Google
	Line Chart (GWT Integrated) Interactive line chart. By: Google
	Magic-Table The Magic Table is a JavaScript library that allows you to see more in your data by applying some simple visual techniques to transform a table. The table is displayed in the browser by the canvas element. Internet Explorer is not supported. By: Greg Ross
	Map (GWT Integrated) An interactive map that uses the Google Maps API. By: Google
	Motion Chart (GWT Integrated) Motion Chart: A dynamic flash based chart to explore several indicators over time. Required columns: bubble name, time and 2 columns of numeric values. Optional columns: Numeric values or categories. By: Google
	Organizational Chart A GWT Integrated simple organizational chart. By: Google
	Parallel Coordinates Chart Parallel Coordinates is a method of visualizing multivariate data. An n-dimensional space is represented as n parallel lines. Works for browsers based on Gecko or Presto (does not work in IE). This is written in Javascript, no Flash required. By: Sri Harsha Allamraju
	Pie Chart (GWT Integrated) Interactive pie chart. By: Google
	Piles of Money Column chart made of of money bills. By: The visapi project
	Scatter Chart (GWT Integrated) Interactive scatter chart. By: Google
	Table (GWT Integrated) A highly customizable table with sorting, paging and selection capabilities. By: Google
	TermCloud A list of terms, where the size and color of each word is determined by a specified frequency value (typically the number of times it appears in some text). By: The visapi project
	Thematic Mapping API Enables visualization of data in Google Earth or other geobrowsers through the use of the Google Visualization API and KML. By: Thematicmapping.org
	WordCloud Displays all words in text with size and color based on the number of time each word appears. By: The visapi project

Additional Information

Below is the talk that Itai Raz, the lead engineer for the Visualization API product at Google, gave at Google I/O 2009 titled “Using the Visualization API with GWT:”

Additional Possibilities

The work above is meant only to serve as a starting point. There is a great deal more information to expand upon. For example, we began this post pulling some information from the XML schema for CVE-2010-1228. One field we did not pull out from the XML file is:

The Common Weakness Enumeration (CWE) represents vulnerability types and NIST provides a CWE Cross Section Mapped into by NVD table. In the above example, we see an entry:

Name	CWE-ID	Description
Race Conditions	CWE-362	The state of a resource can change between the time the resource is checked to when it is accessed.

Clicking on the link will take us to the MITRE site that provides a great deal more information on CWE entries. It is easy enough to expand on the above program to harvest this information for a richer information database.

Another possibility is to expand the above program to pull additional information on the CVE entry. In additional to the data in the NVD CVE XML file, we could pull information from the NVD site. Using CVE-2010-1228 as an example, we could have the program pull down the page:

http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2010-1228

Notice the line:

CVSS v2 Base Score:10.0 (HIGH) (AV:N/AC:L/Au:N/C:C/I:C/A:C) (legend)

The (AV:N/AC:L/Au:N/C:C/I:C/A:C) provides values that were used in determining the base score. If you follow the link, you will see the values used in the calculations:

CVSS Base Score: 10
- Impact Subscore: 10
- Exploitability Subscore: 10
CVSS Temporal Score: Undefined
CVSS Environmental Score: Undefined
Overall CVSS Score: 10

NVD has made available the equations used in calculating the CVSS base score, temporal score, and environmental score.

Three other pieces of information that might provide interesting groupings are:

Access Complexity: Low **NOTE: Access Complexity scored Low due to insufficient information
Authentication: Not required to exploit
Impact Type: Allows unauthorized disclosure of information; Allows unauthorized modification; Allows disruption of service

What information is of interest and how it is used will be dependent on your organization. There is a great deal of information available and many directions you start examining.

Final Thoughts

I am often reminded of the old phrase, “Trust us, we are from the government.” No one really trusts anyone, especially when it comes to matters they do not understand. Just because you are from the security group at your organization, is that reason enough for the CEO to give you unlimited money and authority to do what you see fit? Of course not. While management might trust you, they may not believe that you are capable of seeing the big picture. That is after all their job.

Another great old saying is that “the devil is in the details.” Those details will likely fall in the security domain. In organization across the planet there is a tug of war between the details and the big picture with multiple groups adding in their opinions and views. You need to make the details understandable to your higher management to effectively argue your view. Finding effective metrics and finding clear representation is essential in today’s business. Google Visualization can be a useful tool in accomplishing this task.

Interfacing with Request Tracker

John Gerber — Fri, 10 Apr 2009 23:52:22 +0000

Building on my previous posts, “Request Tracker Installation (Part 1 of 2)” and “Request Tracker Installation (Part 2 of 2),” today we are going to discuss how to get programs interacting with Best Practical’s Request tracker (RT). In later posts, we will build upon this to start having our security processes log information to RT. Keep in mind, tickets do not need to only be done as part of a manual process. Tickets can be generated by processes running on the system. The tickets can also be updated by other processes.

The Database

A few diagrams of the RT’s database schema are available:

If you are unfamiliar with some of the keys and conventions used in Visio graphs, the below table provides some helpful information. Mandatory (not null) columns are displayed in bold.

PK	Primary key	–––0+	0 or 1
FK	Foreign key	–––0<–	0 or more
U	Unique column	–––++	Exactly 1
I	Indexed column	–––+<–	1 or more
O	Optimal columns

Further clarification can be found in chapter 8 titled “Architect” of the “RT Essentials” book from O’Reilly. Look for the section “Logical and Object Model” which takes a tour of RT’s logical and object models.

Perl Module

Referring once more the the “RT Essentials” book’s chapter 8 on architecture, the below diagram maps the layers involved with RT.

A quick overview of what the provide:

Allowing a database independent interface to Perl is the DBI module.
DBIx::SearchBuilder encapsulate SQL queries and rows in simple perl objects allowing object-oriented applications like RT to talk to a table-oriented relational database.
The RT application platform libraries provide database connectivity, logginng infrastructures, users, groups, access control, links, etc. Basically it is the guts of RT.
The RT ticketing system libraries uses the RT application platform.
The Mason handler run on top of the RT core libraries and provides a wrapper around the Mason templating system. The Mason templates consists of the user interface templates, which designed for end users to interact with their browsers, and the REST templates, which are designed to be easy for other software to interact with RT.

Creating, querying, and editing tickets in an RT instance could be done by using RT Command Line Interface (CLI) calls embedded in programs. Or, one could directly plug into the RT libraries. To maintain compatibility with future releases of RT, we will be using RT’s built in REST interface.

Fortunately, Dmitri Tikhonov has created the RT::Client::REST. If Ruby is your preferred language, Tom Lahti has cared a Ruby library to interface with RT.

Jesse Vincent posted recently on the Best Practical blog, “RT 4 – status report.” While it will be awhile before RT4 is out, Jesse has written that “RT4 is based on Jifty and serves up both the legacy /REST/1.0 interface and Jifty’s much more modern REST interface.” End result is that compatibility will be maintained.

Perl Modules Installation

Make sure to follow the instructions from “Request Tracker Installation (Part 2 of 2).” The Perl foundation defines kwalitee as “a set of formalities that tend to coincide with quality, according to consensus. It is of course much less useful than quality, but at least it can be measured.” Install the Perl module Module::CPANTS::Analyse and Test::Kwalitee, along with supporting modules, for quality testing.

 /usr/local/src root# perl -MCPAN -e 'install Module::CPANTS::Analyse'
 /usr/local/src root# perl -MCPAN -e 'install Test::Kwalitee'

If you have read any of Terry Goodkind’s Sword of Truth series of books, you maybe familiar with his character Zedd saying, “If the road is easy, you’re likely going the wrong way.” No where is this more true than in IT. Life is made a bit easier if you check active bugs when setting up software. Fortunately, RT-Client-REST does have an active bug listing.

There is a bug involving CustomFields change in RT 3.8 and how RT matched on the # symbol. RT incorrectly matched when using the REST interface because RT::Client::REST had a CustomField with a # at the end. Jerrad Pierce has just posted that the necessary changes were mote extensive and the code should be pulled down from SVN. We will pull the code from there.

/usr/local/src root# svn checkout \
     http://rt-client-rest.googlecode.com/svn/trunk/ rt-client-rest-read-only
/usr/local/src root# cd  rt-client-rest-read-only/rt-client-rest
/usr/local/src/rt-client-rest-read-only/rt-client-rest root# perl Makefile.PL
/usr/local/src/rt-client-rest-read-only/rt-client-rest root# make
/usr/local/src/rt-client-rest-read-only/rt-client-rest root# make test
/usr/local/src/rt-client-rest-read-only/rt-client-rest root# make install

Connecting up through SSL requires a few additional steps. make sure to install the Perl module Crypt::SSLeay.

 /usr/local/src root# perl -MCPAN -e 'install Crypt::SSLeay'

Ruby

If you need to install Ruby please see my earlier post “Implementing Puppet: Act One.” Ruby 1.9.x is a fairly significant change. See Josh Haberman post “Ruby 1.9.1 released,” Markus Prinz post “Ruby 1.9 – What’s new? What’s changed?“, and Peter Cooper’s post “23 Useful Ruby 1.9 Links and Resources.” If you are working with Ruby, you need Dave Thomas‘ book “Programming Ruby 1.9: The Pragmatic Programmers’ Guide,” which is about to be released and is available in electronic format.

When you issue the “gem install rt-client,” errors involving the TMail file tmailscanner.c will occur. First, it was looking for header files in directory /usr/local/include/ruby-1.9.1 instead of /usr/local/include/ruby-1.9.1/ruby and complaining about “re.h: No such file or directory“. As Zedd would say, “Nothing is ever easy.” If you fix that problem, TMail will complain about “struct RString.” This is a show stopper if you want to use Ruby 1.9.x. When this gets fixed, I will try and come back and update this post.

Connecting Securely

Modify the RT_SiteConfig.pm to use port 443.

Before (Without SSL): Set($WebBaseURL , " http://rt.yourdomain.com");
After (With SSL): Set($WebBaseURL , " https://rt.yourdomain.com:443");

Please change rt.yourdomain.com to the appropriate host value for your organization.

The REST Interface does not support HTTP-Authentication. If your web server requires users to log in, you will end up with authentication problems. As of this writing, there are problems when when both authentication mechanisms are used together. A work around, if your program is running on the same machine as the web server, is to setup a virtual host for 127.0.0.1 that does not use HTTP-Authentication. Make sure to connections are allowed from client 127.0.0.1 only. The outside world interface can continue to be forced to use HTTPS and HTTP-Authentication.

Sample Program

With the supporting software in place, we can now write a program. Below is a simple program that connects up to OpenSSL’s RT site, pulls out all new and open tickets belonging to the OpenSSL-Bugs queue, and prints out the id, subject, owner, status, and when the ticket was created.

#!/usr/local/bin/perl -w

  use strict;
  use Error qw(:try);
  use RT::Client::REST;
  use Data::Dumper;

  my %Config = (
      server      => 'http://rt.openssl.org/',
      username    => 'guest',
      password    => 'guest',
      queue       => 'OpenSSL-Bugs'
  );
  my $rt = RT::Client::REST->new(
    server => $Config{server},
    timeout => 30,
  );

  try {
    $rt->login(username => $Config{username}, password => $Config{password} );
  }
  catch Exception::Class::Base with {
    die "problem logging in: ", shift->message;
  };

  my @ids;
  try {
    @ids = $rt->search(
        type    => 'ticket',
        query   => qq[
            (Status = 'new' or Status = 'open')
            and
            Queue = '$Config{queue}'
        ],
    );
  }
  catch Exception::Class::Base with {
    die "search failed", shift->message;
  };
  for my $id (@ids) {
    my $ticket = $rt->show(type => 'ticket', id => $id);
    print "ID: $id\n";
    print Dumper($ticket);
   }

Final Thoughts

One of the more difficult aspect of connecting several different open source projects is what to do with various versions of the software. While it is unfortunate that we will have to wait for software to get updated on the Ruby side, we are now ready to start working with Perl. In the next post, we will go through the steps to take a program that monitors activity and informs administrators via email, to a system that uses RT to perform this function. By doing so, we gain operational tracking capability. We are about to start having some fun.

Introduction to SQLite

John Gerber — Fri, 04 Apr 2008 00:11:52 +0000

“May you do good and not evil. May you find forgiveness for yourself and forgive others. May you share freely, never taking more than you give.”
— SQLite blessing (in place of legal notice)

I spent the past weekend traveling. When I travel, I listen to podcasts. Traveling is my time to catch up on some great content. I’ll post more on that later. I wanted to draw attention to FLOSS Weekly Episode #26 interview with D. Richard Hipp, creator and lead developer of SQLite. Randal Schwartz and Leo Laporte always do a great job with these interviews. What is so interesting about SQLite? Take a look at the features:

Transactions are atomic, consistent, isolated, and durable (ACID) even after system crashes and power failures.
Zero-configuration – no setup or administration needed.
Implements most of SQL92. (Features not supported)
A complete database is stored in a single cross-platform disk file.
Supports terabyte-sized databases and gigabyte-sized strings and blobs. (See limits.html.)
Small code footprint: less than 250KiB fully configured or less than 150KiB with optional features omitted.
Faster than popular client/server database engines for most common operations.
Simple, easy to use API.
Written in ANSI-C. TCL bindings included. Bindings for dozens of other languages available separately.
Well-commented source code with over 99% statement test coverage.
Available as a single ANSI-C source-code file that you can easily drop into another project.
Self-contained: no external dependencies.
Cross-platform: Linux (unix), MacOSX, OS/2, Win32 and WinCE are supported out of the box. Easy to port to other systems.
Sources are in the public domain. Use for any purpose.
Comes with a standalone command-line interface (CLI) client that can be used to administer SQLite databases.

Those features should be enough to make one take notice. SQLite is also small, compact, portable, efficient, and serverless. It is designed so it can be plugged directly into programs, scripts, or web applications. This provides programs with a lightweight relational database engine that has no external dependencies.

SQLite is very different from MySQL and PostgreSQL. Yet, frequently developers can end up using a full fledge database when something much smaller and effecient could be used. Unfortunately, there are no current comparisons in performance to MySQL and PostgreSQL. The page off the SQLite site, “Database Speed Comparison” does state that the document “describes a speed comparison between an older version of SQLite against archaic versions of MySQL and PostgreSQL.” Still, at least in the past for some operations, SQLite demonstrated impressive speeds compared to PostgreSQL and MySQL:

SQLite 2.7.6 is significantly faster (sometimes as much as 10 or 20 times faster) than the default PostgreSQL 7.1.3 installation on RedHat 7.2 for most common operations.
SQLite 2.7.6 is often faster (sometimes more than twice as fast) than MySQL 3.23.41 for most common operations.

The list of folks using SQLite is impressive: Google Gears, Firefox’s mozStorage, Apple (Safari, Mail, Core Data, Aperture), smf framework in Solaris 10 is using SQLite as its data store, PHP, yum, monotone, AOL email client, Skype, McAfee, along with many additional companies. There are extensions allowing SQLite to be used with languages such as Perl, Python, Ruby, PHP, Java, TCL, .NET, Smalltalk, and many other languages. SQLite compiles and runs on Windows, Linux, Mac OS X, BSD, Solaris, AIX, HP-UX, Symbian, WinCE, VX Works, OS/2, and the NetBSD toaster. SQLite databases are binary compatible, which means they work natively on all systems without any need for conversion. At this point, you are probably beginning to understand why SQLite is so interesting.

Richard has done a talk over at Google TechTalks that provides a good overview of SQLite.

SQLite is made to be easy to setup and use. If the above information has made you somewhat interested, the below instruction on how to setup SQLite should help get you started.

Installation

Each operating system will be somewhat different when it comes to the binary installation. While there will be different filenames, the idea and ease of installation is the same across OSs. For example, there are two files for use under Mac OS X (see the SQLite site for the most recent files):

	sqlite3-3.5.7-osx-x86.bin.gz (177.81 KiB)		A command-line program for accessing and modifying SQLite version 3.* databases. For x86 Macs only.
	sqlite3_analyzer-3.5.4-osx-x86.bin.gz (354.12 KiB)		An analysis program for database files compatible with SQLite version 3.5.4 and later.

Installation can be done by using the binaries supplied from the SQLite site. Generally, you do not need to install SQLite on its own. It will either comes installed with the OS or extensions to programming languages will come with SQLite. This is the advantage of being so small. It is easy to include SQLite.

Below is an example of how to install SQLite binaries under Mac OS X. Mac OS X does come with SQLite installed (sqlite3) by default.

 root# cd /usr/local/src
 /usr/local/src root# mkdir SQLite
 /usr/local/src root# cd SQLite
 /usr/local/src/SQLite root# wget http://www.sqlite.org/sqlite3-3.5.7-osx-x86.bin.gz
 /usr/local/src/SQLite root# wget http://www.sqlite.org/sqlite3_analyzer-3.5.4-osx-x86.bin.gz
 /usr/local/src/SQLite root# gunzip sqlite3-3.5.7-osx-x86.bin.gz
 /usr/local/src/SQLite root# gunzip sqlite3_analyzer-3.5.4-osx-x86.bin.gz
 /usr/local/src/SQLite root# chmod u+x sqlite3-3.5.7-osx-x86.bin
 /usr/local/src/SQLite root# chmod u+x sqlite3_analyzer-3.5.4-osx-x86.bin
 /usr/local/src/SQLite root# ./sqlite3-3.5.7-osx-x86.bin
SQLite version 3.5.7
Enter ".help" for instructions
sqlite>

Below are instructions for installation via source code, which would be applicable for non-windows OSs:

 root# cd /usr/local/src
 /usr/local/src root# mkdir SQLite
 /usr/local/src root# cd SQLite
 /usr/local/src/SQLite root# wget http://www.sqlite.org/sqlite-amalgamation-3.5.7.tar.gz
 /usr/local/src/SQLite root# tar xzf sqlite-amalgamation-3.5.7.tar.gz
 /usr/local/src/SQLite root# cd sqlite-3.5.7
 /usr/local/src/SQLite/sqlite-3.5.7 root# ./configure
 /usr/local/src/SQLite/sqlite-3.5.7 root# make
 /usr/local/src/SQLite/sqlite-3.5.7 root# make install

SQLite and Perl

To provide an example of how to use SQLite below are instructions on installing and using SQLite with the Perl language. As previously discussed, SQLite can be used with many languages. Perl was chosen in honor of Randal Schwartz. While Randal can probably program in all the languages listed above, many first became aware of Randal through Perl. You will find SQLite is just as easy to install and use with your favorite language.

The SQLite extension for Perl contains its own version of SQLite. There really is no need to compile and install SQLite beforehand. While SQLite is binary compatible, different version of the database may not be compatible. The Perl module DBD::SQLite uses an old SQLite database format. DBD::SQLite::Amalgamation uses the most most recent SQLite database format. If you get the error message “SQL error: file is encrypted or is not a database,” this might be caused by different database versions.

To install DBI and DBD::SQLite::Amalgamation using CPAN:

 root#  perl -MCPAN -e shell
   cpan> install DBI
   cpan> install DBD::SQLite::Amalgamation

To install DBI and DBD::SQLite::Amalgamation using source code.

 root# cd /usr/local/src
 /usr/local/src root# mkdir perl
 /usr/local/src root# cd perl
 /usr/local/src/perl root# wget http://search.cpan.org/CPAN/authors/id/T/TI/TIMB/DBI-1.604.tar.gz
 /usr/local/src/perl root# wget \

http://search.cpan.org/CPAN/authors/id/A/AU/AUDREYT/DBD-SQLite-Amalgamation-3.5.6.tar.gz

 /usr/local/src/perl root# tar xzf DBI-1.604.tar.gz
 /usr/local/src/perl root# tar xzf DBD-SQLite-Amalgamation-3.5.7.tar.gz
 /usr/local/src/perl root# cd DBI-1.604
 /usr/local/src/perl/DBI-1.604 root# perl Makefile.PL
 /usr/local/src/perl/DBI-1.604 root# make
 /usr/local/src/perl/DBI-1.604 root# make test
 /usr/local/src/perl/DBI-1.604 root# make install
 /usr/local/src/perl/DBI-1.604 root# cd ../DBD-SQLite-Amalgamation-3.5.7
 /usr/local/src/perl/DBD-SQLite-Amalgamation-3.5.7 root# perl Makefile.PL
 /usr/local/src/perl/DBD-SQLite-Amalgamation-3.5.7 root# make
 /usr/local/src/perl/DBD-SQLite-Amalgamation-3.5.7 root# make test
 /usr/local/src/perl/DBD-SQLite-Amalgamation-3.5.7 root# make install

Create Database

To create a database sample.db, issue the commands:

 root# cd /usr/local/code
 /usr/local/code root# /usr/local/bin/sqlite3 sample.db
SQLite version 3.5.7
Enter ".help" for instructions
sqlite> .quit

Create Table

Creating the table “event” can be done via the command line:

 /usr/local/code root# /usr/local/bin/sqlite3 sample.db "create table event (id INTEGER
                  PRIMARY KEY,odate DATE, description TEXT);"

Inserting Data Into the Table

To insert data into the table via command line:

 /usr/local/code root# /usr/local/bin/sqlite3 sample.db "insert into event (id, odate, description)
       values (1,'2008-04-03 17:59:26','Created entry into SQLite event table.');"

Retrieve the Data From the Table

To retrieve the information via command line:

 /usr/local/code root# /usr/local/bin/sqlite3 sample.db "select id, odate, description from event;"

Creating Database, Insert Data, Retrieve Records via Perl

Below is a Perl program that will create and enter data using DBI DBD::SQLite:

#!/usr/bin/perl

   use DBI;

   # Connect and create database if it does not already exist
   $dbh = DBI->connect( "dbi:SQLite:data.dbl" ) || die "Cannot connect: $DBI::errstr";

   # Create table
   $dbh->do( "CREATE TABLE authors ( lastname, firstname )" );
   $dbh->do( "CREATE TABLE books ( title, author )" );

   # Insert into tables
   $dbh->do( "INSERT INTO authors VALUES ( 'Conway', 'Damian' ) " );
   $dbh->do( "INSERT INTO authors VALUES ( 'Booch', 'Grady' ) " );
   $dbh->do( "INSERT INTO books VALUES ( 'Object Oriented Perl', 'Conway' ) " );
   $dbh->do( "INSERT INTO books VALUES ( 'Object-Oriented Analysis and Design',
                                             'Booch' ) ");
   $dbh->do( "INSERT INTO books VALUES ( 'Object Solutions', 'Booch' ) " );

   # Display data from tables
   $sth = $dbh->prepare( q( SELECT a.lastname, a.firstname, b.title
                                           FROM books b, authors a
                                           WHERE b.title like '%Orient%'
                                       AND a.lastname = b.author ) );
   $rc = $sth->execute();
   if ($rc) {
     while (my($lastname,$firstname,$title) = $sth->fetchrow_array()) {
        print "Name: $lastname, $firstname\nTitle: $title\n";
     }
   }
   else {
     print "Problem with SELECT statement: SELECT a.lastname, a.firstname, b.title
                FROM books b, authors a WHERE b.title like '%Orient%' AND a.lastname = b.author\n";
   }

   # Disconnect from database.
   $dbh->disconnect;

Please note that there is a know issue between DBI and SQLite where a warning message “closing dbh with active statement handles” might be generated. For now, there is no resolution. The code does work. It is only a warning message involving closing the database. You may want to keep an eye open for future resolution.

Additional Information

Mike Chirico has done a nice tutorial on using SQLite, titled “SQLite Tutorial.” While this posting has used a few examples to demonstrate how to create a database, create a table, insert values into the table, and read values from the table, please view Mike’s tutorial for additional commands and more in-depth explanations.

Mike Owens has written a really good book on SQLite, titled “The Definitive Guide to SQLite.” Mike has also made available his presentation for OSCON titled “Programming with SQLite.” The presentation covers “SQLite’s design, operation, capabilities, and limitations, providing developers with a better idea of how, when, and where to best put it to use in their applications.”

Conclusions

There are many more tools and much information available on SQLite. Firefox even has a add-on, SQLite Manager, that allows you to manage SQLite database on your computer. The truth is, I am not sure where I am going to use SQLite. There are plenty of places where I am now thinking I should be using SQLite. I know that I am glad to have it as a tool that I can use. I hope this introduction has captured your interest. Thanks to Randal Schwartz and Leo Laporte for doing the FLOSS Weekly podcast and making me aware of this valuable tool. A special thanks to D. Richard Hipp and Dan Kennedy for developing such a powerful tool.

Ruby

John Gerber — Sat, 23 Jun 2007 19:53:51 +0000

“Penguins mate for life. Which doesn’t really surprise me, ’cause they all look exactly alike. Its not like they’re gonna meet a better-looking penguin someday.”
– Ellen DeGeneres

Normally, I am pleased with myself if I can complete one post a week. Still, I had to put in that extra effort this week in order to get James Turner comic up. The cartoon is bound to become a classic. Well, maybe not among the masses. Okay, maybe just among an elite group of people who can identify languages by the animals on their O’Reilly book covers. Still, that is a pretty special group of people.

Since I am posting a comic strip involving Ruby and Perl, I figured I would add a few pointers of interest. If you are an old time Perl programmer, you will want to check out Jonathan Scott Duff’s posting on, “Everyday Perl 6.” Perlcast, a podcast focus primarily on the Perl programming language, has posted a podcast on “Learning Perl 6.” The presentation was done by Brian D Foy at the Nordic Perl Workshop 2007. Slides along with the audio podcast of the presentation are available.

Not to show favoritism, on the Google Code Blog, they did their fourth podcast where Mark Limber talks on Google SketchUp. To quote the Google SketchUp site:

Developed for the conceptual stages of design, Google SketchUp is a powerful yet easy-to-learn 3D software tool that combines a simple, yet robust tool-set with an intelligent drawing system that streamlines and simplifies 3D design. From simple to complex, conceptual to realistic, Google SketchUp enables you to build and modify 3D models quickly and easily. If you use Google Earth, Google SketchUp allows you to place your models using real-world coordinates and share them with the world using the Google 3D Warehouse.

Ruby is the scripting language that is used in SketchUp. Sorry Perl.